File Format Identification (CPP-008)

Definition and scope

According to PREMIS, a format is defined as “a specific, pre-established structure for the organization of a digital file or bitstream” and “has to correspond to some formal or informal specification” [ https://www.loc.gov/standards/premis/v3/premis-3-0-final.pdf - p. 249 “Format information”]. In other words, a format defines how information is stored and structured within a digital File. The format of a File is essential evidence to assess its risks of obsolescence or incompatibility with future systems.

Moreover, format identification is relevant to many CPPs because several digital preservation policies and actions depend on file format information, for the following reasons:

• Format migration is driven by the source format information and requires format identification to validate the migration process outcome;
• Metadata extraction needs the format identifier to select the tools that extract the Metadata;
• Risk identification relies on the format information and Technical metadata of the File.
• The choice between rendering or emulation can be driven by a format identifier.

There exists a vast number of file formats and format versions and many of them are registered in File Format Registries. These registries collect knowledge about known formats such as name, version, reference documentation, signature, etc. The format identification process depends on one or more registries for detecting possible file formats and gathering detailed information about a format.

The maintenance of format registries is generally a community effort and external to the TDA. Some organisations have opted to maintain an in-house format registry, which requires specialised resources and a significant amount of digital preservation experience. In any case, recording the file format identifiers from the format registries is recommended. Among external registries, the following are particularly notable:

• IANA (Media types); [ https://www.iana.org/assignments/media-types/media-types.xhtml]
• The National Archives (PRONOM); [ https://www.nationalarchives.gov.uk/PRONOM/Default.aspx]
• Wikidata; [ https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/Structures/File_formats ]
• Library of Congress (Format Descriptions); [ https://www.loc.gov/preservation/digital/formats/fdd/descriptions.shtml]

Format identification is typically performed by the detection of format signatures (i.e. one or more byte sequences at relative positions at the beginning or the end of a File). This method prevents scanning the entire File and ensures reasonable performance even for large Files.

The format identification may result in 0, 1 or multiple candidate file formats. Hence, the outcome of the format identification process may not be singular, or decisive (e.g. as in the case of generic file formats like text files, XML documents or ZIP archives). In such cases, additional tools (e.g. other format identification tools, schema validators, content scanners, etc.) may be employed to enrich the results and facilitate the selection of the most probable format. The application of identification tools can therefore be a simple procedure as well as a more complex workflow involving a cascade or parallel execution of multiple tools and a decision tree. It is up to the TDA to decide the level of complexity that is appropriate in the implementation of this step (driven by their context and requirements) in order to achieve the most accurate and detailed format identification result.

All details about the File Format Identification process as well as its output(s) must be recorded (e.g. tools and tool versions used, identification method used, results, etc.). This audit trail supplements the file format information, which captures information about the format of a File (e.g. format identifier, format name, preferred file extensions, etc.) and will be stored in the Provenance metadata.

The TDA should have a procedure in place to deal with inconclusive format identification results (e.g. multiple possible format identifications possible; no format could be identified [ Results can also be unsatisfying (e.g. unrecognised Files being categorised as “application/octet-stream”) or not precise enough (e.g. EPUB Files being categorised as ZIPs).]; format identified, but file extension does not match the format, etc.). Measures for these ‘format identification exceptions’ could include instructions, such as:

• Keep the results as-is, but add annotations on the issue in the Technical and Provenance metadata so that they can be revised at a later stage.
• Queue problematic File(s) for review. The review process can be a manual inspection and/or running a set of format validators. If the review reveals a gap in the format registry, the TDA can also opt to contribute a new format definition or an update to it.
• Reject the File(s) or entire Information package(s).

It is important to note that certain formats can serve as a container for Files. Examples of such container formats are archives (zip, tar, rar, etc.) or emails with attachments. A policy should define the conditions and requirements for the extraction of such Files and whether they should be exported as separate Files. If extracted, these Files should be considered as new Files, which means that they need to undergo the entire ingest stack of processes (e.g. virus scanning, checksum creation, format identification, format validation, etc.). The process of File Format Identification can therefore become a recursive action.

This CPP does not include identification of the embedded Bitstream(s) of Files. These Bitstreams cannot be extracted as standalone Files (e.g. multimedia Files with embedded audio, video and subtitle streams; multiple images in a TIFF File). However, their characterisation is important and typically performed by CPP-009 Metadata extraction.