Metadata Ingest and Management (CPP-016)

Definition and scope

Metadata Ingest and Management is about ingesting, storing and maintaining Metadata to ensure the long-term preservation of the digital content. This Metadata (i.e. “data about the data”) provides contextual information about the Object and describes what the Object is; how it was created or acquired; how it should be managed and preserved; and how it can be accessed and used in the future. There are different categories of Metadata relevant for preservation:

• Descriptive metadata: identifies the content, provides contextual information and serves as finding aid;
• Technical metadata: contains details on the file formats, features and hard-/software used, compression, etc.;
• Fixity metadata: stores information regarding the bitwise state of a File. It is used to help to detect any changes made to the File’s data;
• Provenance metadata: records the origin of the data and keeps track of any processes performed on the data;
• Structural metadata: stores the relationships between Files and logical parts.
• Rights metadata: contains all relevant information about the rights to retain, manipulate and reproduce the data.

The Metadata is to be stored in a format that complies with an open and commonly used standard. Most categories of Metadata have their own specialised metadata formats and it is therefore not required to store all Metadata in a single format. Some common metadata standards are:

• PREMIS for Provenance metadata
• METS for Structural metadata
• Dublin Core, MODS for Descriptive metadata

The Metadata can be assigned to Objects or Files and it must at all times be clear to what Object or File it is assigned. Some parts of the Metadata will be static (e.g. checksums and File size) while others will be dynamic and continue to be enhanced or updated during the life cycle of the Object in the TDA. It is important for the TDA to keep the Metadata safe at all times. No matter how well the data is protected by redundant copies and off-site backups, if Metadata is missing the data can no longer be identified and loses its context and thus most of its value.

New Metadata is usually created and added during the ingest phase, alongside the Objects it relates to. Several processes can and will update Metadata during the preservation. If the Metadata changes in a way that alters the understanding or interpretation of the Objects it preserves, the TDA creates new AIP versions that include the updated Metadata.

In the Provenance metadata, event Metadata serves as witness of the execution of the processes performed on the data. A TDA typically documents the preservation actions it performs as Provenance metadata, thus serving as an audit trail of the Objects. Such event Metadata contains:

• An identifier and description of the process;
• A timestamp indicating when the process execution happened;
• Optionally extra Metadata associated to the event:
  • Event outcome: Single value data that documents the result of the process. (e.g. ‘Success’, ‘OK’, ‘Virus free”). The value is typically limited by a controlled vocabulary;
  • Process information: Metadata set that documents the tool and environment that was used to perform the process. It will include the tool’s name, its version and any configuration parameters that can influence its outcome like virus database version, plugin version, execution parameters, etc.
  • More detailed event result information like ignored warnings, multiple results outcomes, etc.

The Metadata can be stored in many ways [1]. For example, alongside the data Files in a METS File, in a relational database as fields, records and tables, in a noSQL database as documents, in a RDF store as a graph of triples or any combination of the above. The storage technology is not relevant as long as the Metadata 1) can be searched, retrieved and updated through open and well-documented common standards, and 2) is clearly and consistently linked to the Object or File. Usage of identifiers and URIs are key

[1] See also PREMIS v3 - pg 25 “Storing metadata”