Ingest (CPP-029)

Definition and scope

Ingest is a high-level CPP, that is composed of or utilised by many other CPPs, and refers to the process of acquiring and incorporating data into a TDA. In its most abstract form, Ingest describes the process in which a TDA receives a SIP and transforms it into one or several AIP(s) through a process that includes Data Quality Assessment (CPP-019). At the end of a successful ingest, the data is preserved in the TDA for future use.

The Ingest process begins with the transfer of digital Objects and Metadata from their source environment to the TDA in the form of a SIP. The digital preservation workflow triggered by the submission ensures that the data is properly prepared for long-term storage, discovery and access. The workflow ensures that essential Metadata about the creation, structure, and context (CPP-016 Metadata Ingest and Management) of the data exists. Fixity metadata that will be essential for future preservation actions must also exist before the data can be preserved. In addition, Technical metadata that could not be easily extracted (e.g. column delimiters for CSV, quality assessment for OCR, or EPUB Files etc.) from the Files by the TDA may be required to be supplied by the producer.

Data Quality Assessment (CPP-019) measures are integrated throughout the ingest workflow to catch potential issues early. This includes Virus Scanning (CPP-007), File Format Identification (CPP-008), Metadata Extraction (CPP-009) and File Format Validation (CPP-010), and completeness checks to ensure that the digital Objects are suitable for preservation. The TDA verifies that transferred Objects are complete and uncorrupted through Checksum Validation (CPP-002). The process also involves assessing whether the digital Objects conform to the repository's technical requirements and collection development policies. Depending on the TDA’s file format policy - preferred formats, Objects may also need to be normalised (CPP-026 File Normalisation) to preferred formats before or during the ingest process. During the ingest process, the TDA can generate additional Metadata and assign identifiers (CPP-005 Identifier Management) to support Enabling Discovery (CPP-024); perform Enabling Access (CPP-025) for cataloging purposes; and generate Provenance metadata that documents the transfer and processing history of the Objects. In order to evaluate the SIP and its contents, the TDA ensures that the SIP structure is valid and that the SIP is not incomplete (i.e. all Objects and Metadata are present).

SIPs that conform to the TDAs requirements and policies are transformed into AIPs which are sent to preservation in the archival storage of a TDA. SIPs that do not conform to the TDAs requirements will be handled according to its policies (in particular, file format policy and validation policy). The TDA can either reject the submitted data, ask the producer to address the issues before proceeding, flag the data as problematic and ingest it as it is, or perform an operation to address the identified issues.