1、 Confidential | Copyright 2016 IHS Markit Ltd Access to Additional Content For: SMPTE ST 2034-1, Dated: February 27, 2017 (Click here to view the publication) This Page is not part of the original publication This page has been added by IHS Markit as a convenience to the user in order to provide acc
2、ess to additional content as authorized by the Copyright holder of this document Click the link(s) below to access the content and use normal procedures for downloading or opening the files. Files associated with ST 2034-1 Information contained in the above is the property of the Copyright holder an
3、d all Notice of Disclaimer AXF also carries key preservation information, such as provenance, fixity, and the like all key to ensuring long-term robustness and recoverability. Historically, digital archive systems have used media data storage formats that are proprietary to their manufacturers, eith
4、er intentionally or due to the lack of established standards. There have been neither interchange of media nor interoperability of archive systems between manufacturers and in some cases between different archive systems from the same manufacturer. Archives could be orphaned due to support ending fo
5、r the systems used to create data archives. End users and manufacturers recognized that the proprietary nature of archive systems and the data stores that they create result in significant costs of operation that are unnecessary. These costs could be avoided if there were standardization of the form
6、at used for storage of the data on media and for transfer of the data between systems and locations. AXF permits separating the stored content from the systems that create and recover sets of data, thereby enabling refreshing of storage technology, recovering sets of data that otherwise would have b
7、een orphaned, and transferring sets of data between systems and locations. This standard specifies a structure for data that can be written to any current or future data storage subsystem, regardless of the type of media on which it is stored. The data can include any types of files and associated m
8、etadata that are stored and transferred together in a structure called an “AXF Object.” A single AXF Object can be spanned across multiple physical media, can be copied from one set of physical media to SMPTE ST 2034-1:2017 Page 4 of 101 pages another, and is agnostic to the Storage Media Type on wh
9、ich it is stored, e.g., spinning disc or linear tape. Regardless of the Storage Media Types on which they are stored, AXF Objects are identically structured and formatted for any given set and relationship of contained files and metadata. AXF initially arose from the storage needs of the audiovisual
10、 production and archiving communities but quickly encompassed any type of file-based data. The transition to file-based workflows led to a new set of requirements throughout pre-production, production, distribution, storage, and preservation processes. Those requirements included long-term archiving
11、 of finished and unfinished materials, writing data to any type of storage subsystem using a standard scheme, transporting formatted archives between systems and locations using either media or networks, and allowing extensibility sufficient to accommodate any type of file, of any size, from any sou
12、rce, as well as adoption of any future storage technologies. AXF was created to address these requirements. Audiovisual content archiving spans a wide range of content and data archiving systems and practices. At the time this standard was written, many different methods and media were commonly used
13、 to store file-based audiovisual content and its supporting information. Examples range from individual hard drives, solid state drives, and linear magnetic tape drives in small organizations to large spinning disc arrays in combination with very large robotic systems with multiple robots, each havi
14、ng multiple drives, in very large cultural, scientific, and legal archives. Applications in other industries that could benefit from the methods defined herein include medical imaging, geophysical exploration, scientific research, and similar high-volume producers of data. The cultural, scientific,
15、and business value of assets stored on these data systems is significant. Methods for storage, interchange, transport, and preservation of such assets, both locally and remotely, over both short and very long retention periods, demands a standardized, well-documented, non-manufacturer-specific metho
16、d of writing data to any data storage system, from which the data then can be recovered and its contents used, updated, or transferred to another data storage system. All that would be necessary to achieve these objectives is a mechanism for recovering data from the media on which it is stored, plus
17、 utilities or applications that implement AXF. The AXF standard creates a common method of writing individual files or related sets of files, and relevant metadata, onto data storage subsystems so that the structure of an AXF Object will remain the same no matter what vendor equipment or Storage Med
18、ia Type is used. As long as the media remains viable and data can be read from that media, it will be possible to recover an AXF Object and unwrap its contents with a suitable utility or application running on whatever platform is current at the time. The AXF Object also has to be able to be recover
19、ed and stored on future data storage systems without requiring any changes to its contents simply to accomplish the act of medium migration, but it also needs to allow changes to its contents, in case updating is needed to data that already has been archived. AXF addresses these needs through a comb
20、ination of predefined eXtensible Markup Language (XML) schema fields, defined binary data structures that enable an AXF Object to carry any type of file within its File Payload, internal file system functionality, and key metadata enabling the spanning of AXF Objects across multiple physical media.
21、The XML schema also enables essential information about an AXF Object and its contents to be read without having to process all the information within the AXF Object. In addition to media interchange, AXF enables the interoperability of disparate systems through networks because it is structured as
22、a streaming data set. Such interconnections enable seamless movement of AXF Objects from systems that create them, to systems that do not recognize the AXF protocol but store the AXF Object files nonetheless (perhaps in “cloud” storage), then to systems that are designed to recover data from AXF Obj
23、ects. Functionally, AXF acts like a file wrapper or a repository for all types of data without constraint. Unlike media-centric file formats such as MXF, which are similar in that they wrap essences, AXF can contain any number or types of files of any size encapsulated in an AXF Object. It is applic
24、able across a much broader variety of file storage user groups than any media-specific file wrapper. Types of data can include media essence files, related metadata files, production files (such as word processing documents, hypertext documents, associated essence, applications, spreadsheets, and da
25、tabase copies), or any other type of data that users wish to store together. Unlike other file wrapper definitions, it is payload agnostic and does not require any special mappings or adaptations to accept the data an AXF Object carries. SMPTE ST 2034-1:2017 Page 5 of 101 pages AXF accommodates very
26、 large file sizes and quantities within AXF Objects. In the current version of this standard, 64-bit numbers are used to define the sizes of various parameters applicable to elements of AXF Objects. 64-bit numbers can express values up to 18.44674 x 1018 (e.g., 18.44674 petabytes). Use of 64-bit num
27、bers thus can define file sizes in bytes, numbers of files, numbers of media in a spanned set, and similar characteristics up to 18.44674 x 1018 of any particular element. If future requirements exceed the number spaces provided in this document, there is nothing fundamental that limits any particul
28、ar parameter to expression using a 64-bit number. Future revisions of this standard could adopt larger number spaces (e.g., 96-bit, 128-bit, etc.) for those parameters requiring them. The net result is effectively unlimited storage capability within AXF Objects, in terms of file sizes, numbers of fi
29、les in an AXF Object, number of AXF Objects on a medium, number of media in a spanned set, and the like. AXF enables updating AXF Objects when additions of, modifications to, or deletions of files or information that they contain are needed. The functionality to modify AXF Objects is provided by lin
30、king “Supplemental” AXF Objects, written into an archive system at a later time to an original (“Anchor”) AXF Object. A Supplemental AXF Object updates contents of previous AXF Objects without requiring the original AXF Object itself to be modified. Since the original content of the AXF Object is re
31、tained in its original form, it is possible to restore either the original or the modified version whenever necessary. Additional Supplemental AXF Objects can be added in a chain, with restoration of the current or any earlier version possible at any time. When AXF Objects are refreshed by copying t
32、hem to new media, it is possible to consolidate an Anchor Object and its Supplemental Object(s) into a single, new AXF Object. In doing so, it is possible to retain all of the constituent Objects of the Collected Set to which they belong, so that all earlier versions still can be reconstituted in th
33、e future. AXF abstracts the storage of data from the applications that create AXF Objects and from the operating systems, file systems, drivers, and drives that store data on media. By this mechanism, any of the surrounding hardware and software components of systems can be replaced without affectin
34、g the data and its formatting within AXF Objects. A simplified view of where AXF fits into a basic stack is shown in Figure 1. AXF -A w a re App licationSer ver/Storage Stack with AXF sup portArc h ive e Xchang e Format (A XF) , Inclu ding Internal F ile SystemBlock Level Add r essing Fil e Sys temM
35、edium (with out F ile System)Medium (wi th Fi le Syst em )Op e rating Sys te m Har d war e Abstraction Lay erDrive rPhysical DriveFigure 1 Hardware/Software Stack Incorporating AXF Writing to and Reading from Media AXF is designed so that each AXF Object comprises four main components, regardless of
36、 the technology that is used to store it. These components are: SMPTE ST 2034-1:2017 Page 6 of 101 pages Object Header Each AXF Object begins with an Object Header, which contains descriptive XML metadata such as a unique identifier (UUID) for the AXF Object, information regarding its origin, its cr
37、eation date, and a full index of all the files and folders contained in the Object, including file permissions and the like. Generic Metadata Containers Following an Object Header can be any number of optional Generic Metadata Containers. Such containers are selfcontained, open metadata containers i
38、n which applications can place AXF-Object-specific metadata that is not part of the AXF Object File Payload. The metadata can be structured or unstructured, open or vendor-specific, binary, XML, plain text, or any other format. AXF Object File Payload Following any Generic Metadata Containers is the
39、 AXF Object File Payload. It contains the files encapsulated in the AXF Object. The File Payload consists of any number of triplets: File Data + File Padding + File Footer. File Padding ensures alignment of all AXF Object elements on the boundaries of Chunks into which each AXF Object is divided, th
40、ereby enabling addressing, by location within the AXF Object, by its internal file system. File Footer structures contain full information about the preceding file, along with a file-level checksum designed to be processed on-the-fly, with little or no overhead, during restore operations by an appli
41、cation. The information in File Footers enhances the resilience of AXF, as it can be used to recover File Payload data even if Object Header and Footer structures are missing or corrupt. Object Footer Completing an AXF Object is an Object Footer. It repeats the information contained in the Object He
42、ader and adds information captured during creation of the AXF Object, including per-file checksums, precise file sizes, and file positions within the AXF Object. The Object Footer is important to the interchange of an AXF Object because it allows efficient indexing by foreign systems when the media
43、content is not previously known, thereby enabling media transport between systems that follow the AXF standard. It is one of the key structures that support the self-describing nature of AXF. Other significant structures in the AXF protocol are AXF Medium Identifiers and AXF Object Indices. AXF Medi
44、um Identifiers are used on media to indicate formatting of the media according to the AXF protocol and to provide unique identification of the media. AXF Object Indices are optional compilations of the information in all Object Footers preceding each AXF Object Index on a medium, providing a single
45、structure from which it is possible to obtain complete information on the contents of the preceding portion of a medium. When an AXF Object Index is the last structure on a medium, complete information about all AXF Objects stored on the medium can be obtained efficiently in one place. AXF does not
46、require a system to be fully compliant with this standard for it to be able to use and store AXF-generated AXF Objects. The initial adoption of AXF is anticipated to be in applications that create AXF Objects that then are stored on non-AXF-aware storage systems. Because the AXF Objects do not requi
47、re a storage system to know that the AXF Objects are AXF-formatted, the AXF Objects will be viewed simply as files to be stored and retrieved. All that will be necessary to read AXF Objects will be the software and hardware needed to read the physical storage medium. As adoption grows, files can be
48、moved into and out of AXF-aware systems as necessary, with the full range of features becoming available on systems that are AXF compliant. AXF-compliant applications will be able to read stored AXF Objects from any current operating system without unpacking entire AXF Objects to see critical metada
49、ta. Moreover, Archives, or AXF Objects within archives, also can span different types of media, allowing for flexibility within mixed-media archives and for AXF Objects to be identical, regardless of the media on which they are stored. AXF offers resilience to data corruption and loss. AXF Object Indices, repeated identifier instances, and cryptographic hash checksums on both contents and AXF Objects allow for data corruption to be identified and mitigated. Even in catastrophic events, such as the loss of an external database containing records of the contents of an archive
