1、 Copyright 2014 by THE SOCIETY OF MOTION PICTURE AND TELEVISION ENGINEERS 3 Barker Avenue, White Plains, NY 10601 (914) 761-1100 Approved August 15, 2014 Table of Contents Page Foreword . 3 Intellectual Property 3 Introduction 3 1 Scope . 7 2 Conformance Notation . 7 3 Normative References . 7 4 Def
2、initions 8 5 Storage Media Types . 14 5.1 Media with File Systems 14 5.2 Media without File Systems . 15 5.3 File Marks . 15 5.4 Relationships Between AXF Structures and Storage Media Types 15 6 Archive eXchange Format (AXF) Structure . 16 6.1 Form of Data Expression . 16 6.2 Byte Order 17 6.3 Gener
3、al AXF Concepts 17 6.4 AXF Data Structures 18 7 General Usage Considerations 36 7.1 File Naming 36 7.2 Media Preparation 36 7.3 AXF Object Index Structures . 37 7.4 Creating, Reading, Writing, Copying, and Transferring AXF Objects 38 7.5 Nesting AXF Objects 39 8 Spanning 39 8.1 Spanning Linkages . 3
4、9 8.2 Encountering a Spanning Situation 43 8.3 Recovery of Spanned AXF Objects . 43 8.4 Spanning Rules 43 9 Collected Sets 44 9.1 Collected Set Linkages 44 9.2 Collected Set Structure 45 9.3 Add/Replace/Delete Processes . 45 9.4 Tracking Versions 46 Page 1 of 96 pages SMPTE ST 2034-1:2014 SMPTE STAN
5、DARD Archive eXchange Format (AXF) Part 1: Structure AXF also carries key preservation information, such as provenance, fixity, and the like all key to ensuring long-term robustness and recoverability. Historically, digital archive systems have used media data storage formats that are proprietary to
6、 their manufacturers, either intentionally or due to the lack of established standards. There have been neither interchange of media nor interoperability of archive systems between manufacturers and potentially not even between different archive systems from the same manufacturer. End of support of
7、the systems used to create data archives could lead to the archives being orphaned. The Archive eXchange Format has grown from a recognition by end users and manufacturers that the proprietary nature of archive systems and the data stores that they create result in significant costs of operation tha
8、t are unnecessary. These costs could be avoided if there were standardization of the format used for storage of the data on media and for transfer of the data between systems and locations. Use of AXF permits separating the stored content from the systems that create and recover sets of data, thereb
9、y enabling refreshing of storage technology, recovering sets of data that otherwise would have been orphaned, and transferring sets of data between systems and locations. This standard specifies a structure for data that can be written to any current or future data storage subsystem, regardless of t
10、he type of media on which it is stored. The data can include any types of files and associated metadata that are stored and transferred together in a structure called an “AXF Object.” A single AXF Object can be spanned across multiple physical media, can be copied from one set of physical media to S
11、MPTE ST 2034-1:2014 Page 4 of 96 pages another, and is agnostic to the Storage Media Type on which it is stored, e.g., spinning disc or linear tape. Regardless of the Storage Media Types on which they are stored, AXF Objects are identically structured and formatted for any given set and relationship
12、 of contained files and metadata. AXF initially arose from the storage needs of the audiovisual production and archiving communities but quickly encompassed any type of file-based data. The transition to file-based workflows led to a new set of requirements throughout pre-production, production, dis
13、tribution, storage, and preservation processes. Among those requirements were long-term archiving of finished and unfinished materials, a new archiving method based on a standard scheme for writing data to any type of storage subsystem, the ability to transport formatted archives between systems and
14、 locations using either media or networks, and extensibility sufficient to accommodate any type of file, of any size, from any source, as well as adoption of any future storage technologies. AXF was created to serve those needs. Audiovisual content archiving spans a wide range of content and data ar
15、chiving systems and practices. At the time this standard was written, hard drives, linear magnetic data tape, and solid state drives were commonly used to store file-based audiovisual content and its supporting information. Utilization ranged from individual hard drives and tape drives in small orga
16、nizations to large spinning disc arrays in combination with very large robotic systems with multiple robots, each having multiple drives, in very large cultural, scientific, and legal archives. Applications in other industries that could benefit from the methods defined herein include medical imagin
17、g, geophysical exploration, scientific research, and similar high-volume producers of data. The cultural, scientific, and business value of assets stored on these data systems is significant. Methods for storage, interchange, transport, and preservation of such assets, both locally and remotely, ove
18、r both short and very long retention periods, demands a standardized, well-documented, non-manufacturer-specific method of writing data to any data storage system, from which the data then can be recovered and its contents used, updated, or transferred to a another data storage system. All that woul
19、d be necessary to achieve these objectives is a mechanism for recovering data from the media on which it is stored, plus utilities or applications that implement AXF. The AXF standard creates a common method of writing individual files or related sets of files, and relevant metadata, onto data stora
20、ge subsystems so that the structure of an AXF Object will remain the same no matter what vendor equipment or Storage Media Type is used. So long as the media remains viable and data can be read from that media, it will be possible to recover an AXF Object and unwrap its contents with a suitable util
21、ity or application running on whatever platform is current at the time. The AXF Object also has to be able to be recovered and stored on future data storage systems without requiring any changes to its contents simply to accomplish the act of medium migration, but it also needs to allow changes to i
22、ts contents, in case updating is needed to data that already has been archived. AXF addresses these needs through a combination of predefined eXtensible Markup Language (XML) schema fields, defined binary data structures that enable an AXF Object to carry any type of file within its File Payload, in
23、ternal file system functionality, and key metadata enabling the spanning of AXF Objects across multiple physical media. The XML schema also enables essential information about an AXF Object and its contents to be read without having to process all the information within the AXF Object. In addition t
24、o media interchange, because it is structured as a streaming data set, AXF enables the interoperability of disparate systems through networks. Such interconnections enable seamless movement of AXF Objects from systems that create them, to systems that do not recognize the AXF protocol but store the
25、AXF Object files nonetheless (perhaps in “cloud” storage), then to systems that are designed to recover data from AXF Objects. Functionally, AXF acts like a file wrapper or a “bit bucket.” Unlike media-centric file formats such as MXF, which are similar in that they wrap essences, AXF can contain an
26、y number or types of files of any size encapsulated in an AXF Object. It is applicable across a much broader variety of file storage user groups than any media-specific file wrapper. Types of data can include media essence files, related metadata files, production files (such as word processing docu
27、ments, hypertext documents, associated essence, applications, spreadsheets, and database copies), or any other type of data that users wish to store together. Unlike other file wrapper definitions, it is payload agnostic and does not require any special mappings or adaptations to accept the data an
28、AXF Object carries. SMPTE ST 2034-1:2014 Page 5 of 96 pages AXF accommodates very large file sizes and quantities within AXF Objects, now and into the future. In the current standard, 64-bit numbers are used to define the sizes of various parameters applicable to elements of AXF Objects. 64-bit numb
29、ers can express values up to 18.44674 x 1018 (e.g., 18.44674 petabytes). Use of 64-bit numbers thus can define file sizes in bytes, numbers of files, numbers of media in a spanned set, and similar characteristics up to 18.44674 x 1018 of any particular element. If future requirements exceed the numb
30、er spaces provided in this document, there is nothing fundamental that limits any particular parameter to expression using a 64-bit number. Future revisions of this standard could adopt larger number spaces (e.g., 96-bit, 128-bit, etc.) for those parameters requiring them. The net result is effectiv
31、ely unlimited storage capability within AXF Objects, in terms of file sizes, numbers of files in an AXF Object, number of AXF Objects on a medium, number of media in a spanned set, and the like. AXF enables updating AXF Objects when additions of, modifications to, or deletions of files or informatio
32、n that they contain are needed. The functionality to modify AXF Objects is provided by linking “Supplemental” AXF Objects, written into an archive system at a later time to an original (“Anchor”) AXF Object. A Supplemental AXF Object updates contents of previous AXF Objects without requiring the ori
33、ginal AXF Object itself to be modified. Since the original content of the AXF Object is retained in its original form, it is possible to restore either the original or the modified version whenever necessary. Additional Supplemental AXF Objects can be added in a chain, with restoration of the curren
34、t or any earlier version possible at any time. When AXF Objects are refreshed by copying them to new media, it is possible to consolidate an Anchor Object and its Supplemental Object(s) into a single, new AXF Object. In doing so, it is possible to retain all of the constituent Objects of the Collect
35、ed Set to which they belong, so that all earlier versions still can be reconstituted in the future. AXF abstracts the storage of data from the applications that create AXF Objects and from the operating systems, file systems, drivers, and drives that store data on media. By this mechanism, any of th
36、e surrounding hardware and software components of systems can be replaced without affecting the data and its formatting within AXF Objects. A simplified view of where AXF fits into a basic stack is shown in Figure 1. A X F - A w a r e A p p l i c a t i o nS e r v e r / S t o r a g e S t a c k w i t
37、h A X F s u p p o r tA r c h i v e e X c h a n g e F o r m a t ( A X F ) , I n c l u d i n g I n t e r n a l F i l e S y s t e mB l o c k L e v e l A d d r e s s i n g F i l e S y s t e mM e d i u m ( w i t h o u t F i l e S y s t e m )M e d i u m ( w i t h F i l e S y s t e m )O p e r a t i n g S y
38、 s t e m H a r d w a r e A b s t r a c t i o n L a y e rD r i v e rP h y s i c a l D r i v eFigure 1 Hardware/Software Stack Incorporating AXF Writing to and Reading from Media AXF is designed so that each AXF Object comprises four main components, regardless of the technology that is used to store
39、it. These components are: SMPTE ST 2034-1:2014 Page 6 of 96 pages Object Header Each AXF Object begins with an Object Header, which contains descriptive XML metadata such as a unique identifier (UUID) for the AXF Object, information regarding its origin, its creation date, and a full index of all th
40、e files and folders contained in the Object, including file permissions and the like. Generic Metadata Containers Following an Object Header can be any number of optional Generic Metadata Containers. Such containers are selfcontained, open metadata containers in which applications can place AXF-Obje
41、ct-specific metadata that is not part of the AXF Object File Payload. The metadata can be structured or unstructured, open or vendor-specific, binary, XML, plain text, or any other format. AXF Object File Payload Following any Generic Metadata Containers is the AXF Object File Payload. It contains t
42、he files encapsulated in the AXF Object. The File Payload consists of any number of triplets: File Data + File Padding + File Footer. File Padding ensures alignment of all AXF Object elements on the boundaries of Chunks into which each AXF Object is divided, thereby enabling addressing, by location
43、within the AXF Object, by its internal file system. File Footer structures contain full information about the preceding file, along with a file-level checksum designed to be processed on-the-fly, with little or no overhead, during restore operations by an application. The information in File Footers
44、 enhances the resilience of AXF, as it can be used to recover File Payload data even if Object Header and Footer structures are missing or corrupt. Object Footer Completing an AXF Object is an Object Footer. It repeats the information contained in the Object Header and adds information captured duri
45、ng creation of the AXF Object, including per-file checksums, precise file sizes, and file positions within the AXF Object. The Object Footer is important to the interchange of an AXF Object because it allows efficient indexing by foreign systems when the media content is not previously known, thereb
46、y enabling media transport between systems that follow the AXF standard. It is one of the key structures that support the self-describing nature of AXF. Other significant structures in the AXF protocol are AXF Medium Identifiers and AXF Object Indices. AXF Medium Identifiers are used on media to ind
47、icate formatting of the media according to the AXF protocol and to provide unique identification of the media. AXF Object Indices are optional compilations of the information in all Object Footers preceding each AXF Object Index on a medium, providing a single structure from which it is possible to
48、obtain complete information on the contents of the preceding portion of a medium. When an AXF Object Index is the last structure on a medium, complete information about all AXF Objects stored on the medium can be obtained efficiently in one place. AXF does not require a system to be fully compliant
49、with this standard for it to be able to use and store AXF-generated AXF Objects. The initial adoption of AXF is anticipated to be in applications that create AXF Objects that then are stored on non-AXF-aware storage systems. Because the AXF Objects do not require a storage system to know that the AXF Objects are AXF-formatted, the AXF Objects will be viewed simply as files to be stored and retrieved. All that will be necessary to read AXF Objects will be the software and hardware needed to read the
