Data Corruption Management (CPP-004)
| CPP-Identifier | CPP-004 |
| CPP-Label | Data Corruption Management |
| Author | Johan Kylander |
| Contributors | Juha Lehtonen, Bertrand Caron |
| Evaluators | Felix Burger, Maria Benauer |
| Change history | Comments |
|---|---|
| Version 1.0.0 - 2025-08-29 | Milestone version |
| Version 1.1.0 - 2026-03-26 | Migration to XML, addition of Copy Management Policy as input for step 2. |
| Version 1.2.0 - 2026-05-18 | Reassigned step numbers based on new numbering rules. |
1. Description of the CPP
The TDA replaces damaged Files from replicated copies and reports on actions taken.
Inputs and outputs
| Input(s) | |||||||
|---|---|---|---|---|---|---|---|
| Data |
| ||||||
| Metadata |
| ||||||
| Documentation/guidance |
| ||||||
| Output(s) | |||||||
| Data |
| ||||||
| Metadata |
| ||||||
Definition and scope
Data Corruption Management is a process, where a TDA restores corrupted AIPs from parallel copies. Corrupted AIPs are identified and flagged by the Integrity Checking (CPP-003) process, which periodically scans the fixity of AIPs .
When Integrity Checking (CPP-003) has flagged that an AIP is corrupted (i.e. that it has been unintentionally altered), the TDA recovers an intact copy from another storage medium in order to replace the corrupted AIPs. As part of the retrieval process, the checksums of the copied data are validated to verify a) the integrity of the data that is used as a source and, b) that the new target has been copied successfully. This is similar to the process of Replication (CPP-011). Subsequently, Provenance metadata is updated to maintain a record of the replacement procedure.
The replacing copy can be written to another storage medium than the corrupted one (e.g. in the case of magnetic tapes, where read-write operations to a single tape are kept to a minimum). In these cases, Storage management information must also be updated to reflect the new location of the copy.
The TDA may choose to replace the whole storage medium's content, triggering a process similar to Refreshment (CPP-030). This can be done when media-wide corruption or read/write errors are detected.
TData Corruption Management relies on the TDA having several parallel copies, preferably on different storage media and in different storage locations. The number of parallel copies, and their storage conditions are defined in the TDA’s policies as maintained by Risk Mitigation (CPP-012) and a copy management policy in particular. In accordance with agreed on best-practices, at least three copies are recommended, as there should exist at least two other valid copies in case one copy is corrupted or destroyed.
Process description
Trigger event(s)
| Trigger Event | CPP-identifier |
|---|---|
| An AIP or File that has been flagged as corrupt | CPP-003 (Integrity Checking) |
| Media-wide corruption or read/write errors are detected |
Step-by-step description
| No | Supplier | Input | Steps | Output | Customer | |
|---|---|---|---|---|---|---|
| sequence | ||||||
| 1 | alternative - Elaboration of a media- or AIP-specific inventory | |||||
| 1.a | CPP-003 (Integrity Checking) | Integrity checking report | Identify and locate corrupted AIPs | Inventory of corrupted AIPs | ||
| Storage medium with broken AIPs | ||||||
| 1.b | Report of media-wide errors | Identify and locate the AIPs on the corrupted medium. | Inventory of corrupted AIPs | |||
| Storage medium with media-wide errors packages | ||||||
| 2 | CPP-012 (Risk Mitigation) | Copy Management Policy | Select source medium to copy the AIPs from | Authoritative storage medium for replicating/copying the AIPs | ||
| 3 | Select target storage medium (can, and often is, be same as the original storage medium identified in step 1a) | Target storage medium | ||||
| 4 | Optional: In case of media-wide errors: Provision of a fresh storage medium that will replace the old one. | The fresh storage medium that will replace the old one | ||||
| 5 | Source storage medium of AIPs | Gather AIP inventory and storage medium information to start the copy process (step 6) | ||||
| Target storage medium | ||||||
| Inventory of AIPs involved in the process | ||||||
| 6 | sequence - For each AIP individually, copy AIP to storage medium | |||||
| 6.1 | Retrieve the AIP from the source storage medium | |||||
| 6.2 | Copy the AIP to the target storage medium | New copy of AIP | ||||
| 6.3 | Existing Fixity metadata | Validate the fixity of the AIP on the fresh storage medium | Fixity Metadata | |||
| Optional: Valid status (step 6.4) | ||||||
| Optional: Invalid status (go back to step 6.1) | ||||||
| 6.4 | Update the fixity for the new AIP copy | Fixity metadata | ||||
| 6.5 | Optional: If the target storage medium is different from the original storage medium with the broken AIP:Update the storage location for the new AIP copy | Storage management information | ||||
| 7 | Inventory of AIPs involved in the process | Check that all AIPs in the inventory have been successfully copied | Optional: Confirm completeness of the copy process (go to step 8) | |||
| Optional: Error (go back to copy process loop, step 6) | ||||||
| 8 | Optional: In case of media-wide errors:Update information about the fresh storage medium (e.g. File locations, media identifiers) and mark the old medium and its contents as ready for deletion/decommissioning. | Storage management information | ||||
| 9 | Document the event and its timestamp | Provenance Metadata | ||||
| 10 | Original storage medium that has been refreshed | Optional: In case of media-wide errorsEnsure data security and that confidentiality is not compromised by making sure that data on the original storage medium is properly deleted. | ||||
| 11 | Decommission the old storage medium | Record of decommissioning | ||||
Rationale(s) and worst case(s)
| Rationale | Impact of inaction or failure of the process |
|---|---|
| Parallel copies of the data. | Corrupted data cannot be restored unless parallel copies exist. |
| Fixity metadata | Data corruption cannot be detected, and replaced copies cannot be verified, without Fixity metadata. |
2. Dependencies and relationships with other CPPs
Dependencies
| CPP-ID | CPP-Title | Relationship description |
|---|---|---|
| CPP-003 | Integrity Checking | Corrupted data is detected by periodic integrity checking. |
| CPP-012 | Risk Mitigation | The number of parallel copies and how they are stored (media, locations) are defined in a TDA’s policy that arises out of mitigating risks to preserved data.. |
Other relations
| Relation | CPP-ID | CPP-Title | Relationship description |
|---|---|---|---|
| May require | CPP-005 | Identifier Management | If a File is corrupted, it may need to be repaired or replaced. During this process, a new PID may be created. |
| Required by | CPP-013 | Object Management Reporting | Fixing corrupted AIPs produces Provenance metadata and data for quality reporting to the stakeholders.. |
| Affinity with | CPP-002 | Checksum Validation | All new AIP copies must have their checksum validated to verify that the process was successful. The checksum validation is more mechanical in its nature in Data Corruption Management, only aiming at verification of the copy process. In contrast to CPP-002, it does not have to negotiate with producers or examine the results. |
| Affinity with | CPP-011 | Replication | Corrupted copies are replaced by intact copies, effectively replicating the intact copy, but not creating a new parallel copy. |
| Not to be confused with | CPP-007 | Virus Scanning | If a File is detected as infected and cannot be cleaned, it might be considered "damaged." However, CPP-004 typically applies to technical corruption or loss, rather than deliberately human-made damage such as malware-infected Files (CPP-007). In practice, infected File are more likely to be replaced (by the producer) or rejected. |
| Alternative to | CPP-027 | File Repair | File Repair is an alternative (fallback) to Data Corruption Management in cases where no intact copy of corrupted or broken data is available, since repairing the structure of an altered copy is the only option. |
3. Links to frameworks
Certification
| Certification framework | Term used in framework to refer to the CPP | Section |
|---|---|---|
| CTS Link | “For each storage location, measures should be in place to ensure that unintentional or unauthorised changes can be detected and correct versions of data and metadata recovered” | R14 Storage and Integrity |
| Nestor Seal Link | “restoration of the archival information packages”“recovering archival information packages in the event of damage” | C15 Integrity: Functions of the archival storage |
| ISO 16363 Link | “Recovery actions” | 5.1.1.3.1 The repository shall record and report to its administration all incidents of data corruption or loss, and steps shall be taken to repair/replace corrupt or lost data. |
4. Reference implementations
Use cases
Summary between 2021-2024 at CSC
| Institutional background | |
|---|---|
| Institution | CSC – IT Center for Science Ltd., FI |
| Hyperlink | https://digitalpreservation.fi/en/services/quality_reports |
| Description | |
| Trigger event | 2024a: Software error2024b: Scheduled Integrity Checking2022a: Human error2022b: Scheduled Integrity Checking2021: Scheduled Integrity Checking |
| Problem statement | 2024a: Contents of one tape were lost2024b: One corrupt AIP copy on a tape2022a: One corrupt AIP copy on a disk2022b: One corrupt AIP copy on a disk2021: Ten corrupt AIP copies on a tape |
| Proposed solution | 2024a: The tape was restored from other copies2024b: A new copy was produced2022a: A new copy was produced from the tape2022b: The corrupted copy of the package had been unsuccessfully copied to storage during ingest earlier but had been later successfully copied to disk storage automatically.2021: New copies were produced |
Publicly available documentation
| Institution | Organisation type | Language | Hyperlink |
|---|---|---|---|
| TIB – Leibniz Information Centre for Science and Technology and University Library, DE | National library Non-commercial digital preservation service Research infrastructure Research performing organisation | English | https://wiki.tib.eu/confluence/spaces/lza/pages/93608373/Archival+Storage#ArchivalStorage-Recovery |
| CSC – IT Center for Science Ltd., FI | Non-commercial digital preservation service | English | https://digitalpreservation.fi/en/services/quality_reports/2024 (section Quality Deviations Relating to Preserved Content in 2024) |
| English | https://digitalpreservation.fi/en/services/quality_reports/2022 (Quality Deviations Related to the Data in Preservation in 2022) | ||
| Archivematica, CA | Digital preservation system | English | https://www.archivematica.org/en/docs/storage-service-0.23/recovery/#recovery |