AIP Versioning (CPP-021)

Definition and scope

Updating AIPs is a common operation in a TDA that involves keeping track of changes to data or Metadata, and preserving previous versions (entirely or partially) in order to go back to different stages of an AIP. Either the producer or the TDA may initiate an update, usually because of technical or contextual changes on the level of File, Representation or Metadata.

Reasons for Versioning include, but are not limited to:

For contextual quality reasons:

• Removal of data
  • Fault in the initial deposit: An AIP or Representation contains File(s) that were not supposed to be part of the deposit (e.g. system Files not needed for preservation);
  • Content being retracted after the initial deposit: An AIP or Representation contains File(s) that were withdrawn (e.g. due to security reasons, legal restrictions).
• Update of data
  • Update of File(s): An AIP or Representation contains File(s) that have been replaced (e.g. due to faults in the initial deposit);
  • Enrichment or correction of Descriptive Metadata;
  • Update of administrative Metadata (e.g. due to changes in the access rights or legal information for an AIP).
• Addition of data
  • Incomplete initial deposit: The initial Information Package was missing Files and some need to be added after its ingest;
  • Enhancement of the initial deposit: After ingest of the Information Package (IP), additional material was issued and needs to be preserved in the IP.

For technical quality reasons:

• Replacement or Update due to preservation action: A File or Representation has been migrated into a format deemed more suitable for preservation;
• Replacement or Update due to new packaging: A File or Representation has been re- or unpackaged (e.g. for ZIP, TAR, WARC or disk image containers).

For Metadata quality reasons:

• Addition or replacement of Technical metadata as a consequence of re-running a characterisation process (i.e. Format Identification, Metadata Extraction or Format Validation) on already ingested AIPs (e.g. due to a new tool being released).

Updates might involve the submission of a new SIP by the producer (e.g. in the case of changes to the initial deposit), but can also just take place on the storage level (e.g. new data being added as a result of a preservation action) without a SIP being involved.

In comparison to other IT domains, updating and creating new versions in digital preservation exhibits peculiarities for the following reasons:

• On the File level: As data and Metadata are generally stored in physical containers (such as ZIP, TAR, WARC, etc.), unpacking and merging content from the incoming SIP is a complex and risky process;
• On the storage level: As physical containers corresponding to IPs are usually stored on tape or other cold-storage technologies, updates are generally asynchronous;
• On the Metadata level: Authenticity and audit trail requirements imply that TDAs record precisely all details about the versioning impacts (e.g. requester, date, reasons, impacted data and Metadata, etc.). This is of particular importance when the versioning is requested by TDA and when the new version replaces the original one.

A TDA must have a versioning policy in place for handling and documenting retracted or updated data. It may decide to partially or totally retain the older versions, based on several factors such as reversibility (e.g. addition of new Files versus replacement of Files), purpose of the update (e.g. correction, enrichment, risk mitigation, etc.), issuer of the updating request (i.e. Producer or TDA), etc.

The implementation of updates to AIP (i.e. data and/or Metadata) requires consideration of a range of factors including a) what is being updated and with what frequency, b) whether that should result in a new AIP version or simply an update to the current version, c) how often the different versions of the AIP need to be accessed, d) how the AIP is stored internally (e.g. using databases, file systems, tape libraries, cloud storage), and e) how versions are serialised and exported from the TDA when they need to be exchanged with third-parties.

Specific implementation approaches are outside of the scope of this CPP.. However, some considerations may include:

• Are there regular and incremental updates in dynamic collections of content? If so, consider storing the changes (deltas) between AIP versions, rather than complete AIPs, which can reduce both storage requirements and processing overhead.
• Are there any changes to the AIP content that do not require versioning (e.g. recording the results of fixity checks). If so, consider mechanisms that allow efficient tracking of preservation actions, such as preservation Metadata updates, without duplicating or accessing unchanged content.
• Is there a need for enhanced traceability and provenance so that it is easy to follow the evolution of AIPs over time, without needing to access or analyse internal Metadata? If so, consider additional indexing and logging mechanisms alongside AIP storage.
• The ability to access the deltas between AIPs instead of whole versions can support easier integrity checks and rollback capabilities (e.g. enabling recovery from data corruption without full AIP restores).
• Is there a need to externalise and exchange AIPs with third parties (e.g. repository migration, synchronisation, and replication)? Exchanging AIP versions as deltas can enable more efficient data transfer across distributed or replicated preservation systems.
• If some or all of AIPs are held using offline media, then minimising data handling can be desirable to reduce the risk of corruption or damage when media is retrieved or read. Isolating and preserving only changed components of AIPs can reduce unnecessary manipulation of stable data, which lowers the risk of inadvertent corruption, loss, or operational error.