File Repair (CPP-027)

Definition and scope

File Repair is an operation performed to correct erroneous format structures that could prevent the expected use or rendering of the File by the repository or the user. For additional considerations on the topic, see this conference paper. In addition, erroneous format structures can potentially hinder preservation actions, such as file format migration, as tools developed to process the Files expect the Files to have a defined structure.

Erroneous format structures have many different causes. Some examples are listed below:

• Improper behaviours of creation software (e.g. InDesign, in some previous versions, was known to export corrupted PDFs);
• Partial or failed transfers (e.g. digital photographs transferred through an unstable bluetooth connection);
• Corruption or loss when data is transferred to a TDA or is processed within a TDA where there are no ‘good’ copies of the data that can be recovered from and hence file repair is necessary (e.g. a File is corrupted during transfer to a TDA which goes undetected or a File is corrupted during ingest and is then replicated across all copies held by the TDA).
• Edition with a different software than the creation tool (e.g. PDFs produced with pdfTeX, then edited with Quark XPress);
• etc.

The impact of the erroneous format structures should be assessed as “low”, “medium” or, “high” according to the following criticality / severity criteria:

• Impact is considered “High” when important parts of the File’s or Representation’s content are not accessible;
• Impact is considered “Medium” when a regular feature of the File or Representation is not supported, either for the consumer or the TDA (e.g. if metadata extraction cannot be performed);
• Impact is considered “Low” when no effect can be detected on the File’s or Representation’s content or features.

A File or Representation might be affected by several deviations from the standard. In that case, the impact should be assessed for each identified deviation. The process should then try to address issues from the most to the least impactful. One should keep in mind that errors might be related, so that one repairing operations might affect several errors returned by a validation tool. In addition, some tools would stop the validation process at the first error, so that repairing the initial error returned by the tool would lead to a still invalid File for another reason.

This CPP covers the following steps:

• The initial analysis step that first determines precisely which erroneous format structure is responsible for the failure using validation tools and metadata extractors, and then evaluates the risk that these structures pose for current and future users;
• The intervention on the File itself;
• The control step that measures the impact of repair, and controls that no significant properties were affected in the process;
• The documentation step that records the details of the process;
• The handling of false positives and identification of gaps in the tools producing the false positive warnings / errors.

Many reasons could cause the process not to be effectively performed. In the description below, three are mentioned:

• The impact of erroneous structures is deemed low compared to the Object value and the operation’s complexity;
• The deviation from the standard could not be assessed;
• A repair method could not be provided by CPP-012 (Risk Mitigation).

In any of these cases, the file repair process would end and another error-handling option among the ones suggested in CPP-010 (File Format Validation) should be considered.

File repair is a manual and investigative process that requires in-depth knowledge of format structures. Therefore, its performance requires a high level of institutional maturity and organisational resources in digital preservation and is recommended only for mature TDAs.