SMPTE EG 2076-2-2016 Image Identification Alignment Transport and System Guidance for Stereoscopic (S3D) or Multi-Camera Array.pdf

1、 Copyright 2016 by THE SOCIETY OF MOTION PICTURE AND TELEVISION ENGINEERS 3 Barker Avenue, White Plains, NY 10601 (914) 761-1100 Approved August 24, 2016 Table of Contents Page Foreword 2 Intellectual Property . 2 Introduction. 2 1 Scope . 3 2 Conformance Notation . 3 3 Terms and Definitions 3 4 Ove

2、rview . 4 5 Image Identification . 5 6 Image Alignment . 6 7 Transport . 6 7.1 Network . 6 7.2 SDI 6 7.3 Payload ID 8 8 System Guidance 9 8.1 Time Stamp 9 8.2 Genlock . 9 8.3 Processing 9 8.4 Disparity Maps and Depth Maps 11 8.5 Record/Playback Device Expectations . 11 Bibliography (Informative) 12

3、SMPTE EG 2076-2:2016 SMPTE ENGINEERING GUIDELINE Image Identification, Alignment, Transport and System Guidance for Stereoscopic (S3D) or Multi-Camera Array Page 1 of 12 pages SMPTE EG 2076-2:2016 Page 2 of 12 pages Foreword SMPTE (the Society of Motion Picture and Television Engineers) is an intern

4、ationally-recognized standards developing organization. Headquartered and incorporated in the United States of America, SMPTE has members in over 80 countries on six continents. SMPTEs Engineering Documents, including Standards, Recommended Practices, and Engineering Guidelines, are prepared by SMPT

5、Es Technology Committees. Participation in these Committees is open to all with a bona fide interest in their work. SMPTE cooperates closely with other standards-developing organizations, including ISO, IEC and ITU. SMPTE Engineering Documents are drafted in accordance with the rules given in its St

6、andards Operations Manual. SMPTE EG 2076-2 was prepared by Technology Committee 32NF. Intellectual Property At the time of publication no notice had been received by SMPTE claiming patent rights essential to the implementation of this Engineering Document. However, attention is drawn to the possibil

7、ity that some of the elements of this document may be the subject of patent rights. SMPTE must not be held responsible for identifying any or all such patent rights. Introduction This section is entirely informative and does not form an integral part of this Engineering Document. Stereoscopic (S3D)

8、or Multi-Camera Array imaging systems deliver two or more images to a downstream process that can be used for stereoscopic display, light field computation or a Multi-View display. To maintain the fidelity of S3D or Multi-Camera Array images it is critically important that the spatial and temporal a

9、lignment of the images be maintained at all times. Loss of alignment between the images will result in the degradation or complete destruction of the stereoscopic or Multi-Camera Array for post processing and ultimately the viewer. This document highlights the methods and specific issues to be aware

10、 of when dealing with S3D or Multi-Camera Array images within a system. This document does not address audio or control systems as they relate to S3D or Multi-Camera Array images. SMPTE EG 2076-2:2016 Page 3 of 12 pages 1 Scope This Engineering Guideline provides an overview of the identification, s

11、ynchronization and transport of Stereoscopic (S3D) or Multi-Camera Array motion picture and television images over SDI transport systems. 2 Conformance Notation This Engineering Guideline is purely informative and meant to provide tutorial information to the industry. It does not impose Conformance

12、Requirements and avoids the use of Conformance Notation. Engineering Guidelines frequently provide tutorial information about a Standard or Recommended Practice and when this is the case, the user ought to rely on the Standards and Recommended Practices referenced for interoperability information. 3

13、 Terms and Definitions For the purposes of this document, the following terms and definitions apply. 3.1 left eye Le Abbreviation for Left Eye. 3.2 right eye Re Abbreviation for Right Eye. 3.3 three dimensional 3D Acronym for Three Dimensional. Adding depth as the third visual dimension. 3.4 stereos

14、copic 3D S3D Acronym for Stereoscopic 3D. 3.5 S3D Image Content Stereoscopic Image content, which results in the viewing of a scene with the perception of depth to the observer. 3.6 Stereoscopic Relating to the use of binocular vision to create the perception of depth in an image. 3.7 Multi-Camera A

15、rray Two or more cameras aligned to capture a single scene coincident in time for downstream light field image processing. 3.8 Coincident in Time With respect to dual image signals for stereoscopic television, this means that not only are the two image signals “genlocked“, but that they represent th

16、e same moments in time for the image displayed. SMPTE EG 2076-2:2016 Page 4 of 12 pages 3.9 Camera As defined here for this document includes the lens and all associated equipment required to create digital representations of moving images. The camera is generally capable of stand-alone operation. 3

17、.10 Camera Systems As defined here for this document, a camera system consists of all cameras and their associated equipment to create S3D content or for computational light field image processing. 3.11 Genlock Abbreviation of “sync Generator Lock.” Genlock is a technique for locking a devices inter

18、nal sync structures (and thus image structures) to a common external reference (a “sync generator”). 3.12 Temporal Alignment With respect to the capture mechanisms of digital motion picture and television cameras, the capture sensor systems need to be temporally synchronized in order to maintain pro

19、per representation of motion when using more than one sensor or device to capture S3D or Multi-Camera Array images for presentation. 4 Overview S3D or Multi-Camera Array images must maintain their spatial and temporal alignment so that downstream devices can process and display the content correctly

20、 to the viewer. Stereoscopic content consists of a pair of images normally designated Left Eye (Le) and Right Eye (Re) that must remain coincident in time. Multi-Camera Array content consists of two or more images that must remain coincident in time. This document is mostly centered around Live Even

21、t systems where real-time processing and transmission are a basic requirement for the system. (i.e. Broadcast) Non real-time workflows, such as file based processes, are mentioned briefly but are not extensively covered here. If proper identification of frames is maintained for file-based systems, t

22、hen there are fewer opportunities for synchronization issues to occur. Below for reference is an example figure showing the key components of a Live Event system. This is a general example and does not show a complete system nor does it attempt to show each process that could be used. Instead it att

23、empts to illustrate the key processes where synchronization can be affected. SMPTE EG 2076-2:2016 Page 5 of 12 pages Figure 1 Example Live Event System Identification of the images becomes a critical element for maintaining alignment throughout the system. Without knowing the relationship between th

24、e images, designing systems to maintain alignment will be difficult if not impossible. In the following sections of this Engineering Guideline it steps through the key issues for Identification, Alignment and the Transport of S3D or Multi-Camera Array images. 5 Image Identification During Image acqu

25、isition or creation, an image pair (in the case of S3D) ought to be identified as Left Eye (Le) and Right Eye (Re). Images that are part of a Multi-Camera Array ought to be identified with a non-duplicate identifier depending on the intent of the image captured or created coincident in time. For exa

26、mple Camera: 1,2,3 a, b, c Top Left, Top Center, Top Right Along with identifying the image “intent”, the system ought to provide temporal identification such as Time code or frame count. File based workflows have many ways of tagging image frames for identification. We will not explore these as the

27、re are too many methods and this is not the core objective of this document. One ought to refer to the specification of the file format that is used and decide on the best method in which to tag each image frame. This document will provide methods in the sections below to tag the image frames in SDI

28、 transport streams. SMPTE EG 2076-2:2016 Page 6 of 12 pages 6 Image Alignment Each individual camera image derived from a S3D or Multi-Camera Array system must maintain temporal alignment. This alignment must the time of capture of moving images.be maintained throughout the camera system, without ch

29、ange or drift during This alignment must be maintained throughout the production system as well, without change or drift during the time of capture or generation of the images. This alignment must be maintained and preserved to a recording device, a display device and to the output of the system. Fo

30、r optimum results there ought to be no fixed offsets or temporal delays between each of the images. If this is unavoidable, a fixed offset that does not change over time can be used provided it is transmitted and recognized by the downstream systems required to process the images. 7 Transport When t

31、ransporting S3D on all interfaces it is extremely important to be able to identify the Left Eye image stream from the Right Eye stream. If these image streams are not identified, or are improperly identified, then it is difficult to reliably connect the links to the correct inputs and outputs of equ

32、ipment. In the case where there is no identification applied at the source, it is very important to insert the payload ID as early as possible in the production system. There are devices available from some equipment vendors that allow insertion of the correct payload ID as well as swapping the sign

33、als if they are incorrectly identified. 7.1 Network More and more common IT network systems are being used to transport images these days. The bulk of this is used for non-real-time transport. There are some cases where these networks are used for real-time transport. In either case this document wi

34、ll not address this as there are many different types of file formats and network protocols. Proper identification is key when using these systems and ought to be maintained at all times. 7.2 SDI For many years SMPTE ST 292-1 has been widely used to deploy HDTV systems. It defines the bit-serial dat

35、a structure and the coaxial cable interface specifications for a nominal 1.5 Gb/s Signal/Data Serial Interface to carry either 4:2:2 10 bit 1280720, 19201080 or 20481080 active pixel formats defined by SMPTE ST 296, SMPTE ST 274 and SMPTE ST 2048-2 and which can be mapped into a 1.5 Gb/s payload. SM

36、PTE ST 292-1 also maps audio and other ancillary data into the 1.5 Gb/s payload. 7.2.1 4:2:2 Sampling With the advent of S3D production, it became necessary to carry two of these image streams in such a way as to maintain the spatial and temporal relationships between the Left Eye stream and the Rig

37、ht Eye stream. SMPTE ST 292-2 was developed to standardize a Dual Link 1.5 Gb/s (nominal) interface for S3D. That interface uses a standard SMPTE ST 292-1 interface for each of the Left and Right Eye streams and also defines a payload identifier (see Section 7.3 below) that will identify the Left Ey

38、e (Le) or Right Eye (Re) images, audio and other associated ancillary data. SMPTE ST 425-1 defines the data structures for the carriage of SMPTE ST 296, SMPTE ST 274 and SMPTE ST 2048-2 images that exceed the 1.5 Gb/s capacity of SMPTE ST 292-1 and permits Level A and Level B Dual Link mappings of t

39、hese images onto these data structures. In addition, SMPTE ST 425-1 provides a Level B Dual stream mapping which maps two 1.5 Gb/s payload images onto these data structures. SMPTE ST 424 defines the bit serial data structure and coaxial cable specifications for carrying the payloads mapped by SMPTE

40、ST 425-1 serial interface. SMPTE EG 2076-2:2016 Page 7 of 12 pages Carrying two related images on a Dual Link interface has always presented challenges in keeping the images together. SMPTE ST 425-2 was developed to standardise the carriage of S3D streams carried by Dual Link 1.5 Gb/s SMPTE ST 292-2

41、 interfaces on a single 3 Gb/s interface using these Level B Dual Stream mapping rules. It essentially carries the complete payload of the SMPTE ST 292-2 interface and applies a new payload identifier to indicate the carriage of S3D images on a Single Link 3 Gb/s interface. SMPTE ST 424 still specif

42、ies the serial interface for SMPTE ST 425-2 as it utilizes the fundamental data structures defined in SMPTE ST 425-1. The 4:2:2/10-bit image formats defined by SMPTE ST 274 and SMPTE ST 2048-2 for higher than 30 frames per second frame rates do not fit into a 1.5 Gb/s payload, and can be carried on

43、a SMPTE ST 425-1 3 Gb/s coaxial interface. SMPTE ST 425-4 was developed to standardize a Dual Link nominal 3 Gb/s interface for S3D. This interface uses a standard SMPTE ST 425-1 Level A or Level B Dual Link interface for each of the left and right eye streams and also defines the payload identifier

44、 that will identify the Le and Re images, audio and other associated ancillary data. Once again, SMPTE ST 424 specifies the serial interface for SMPTE ST 425-4 as it utilizes the fundamental data structures defined in SMPTE ST 425-1 for each Eye. Table 1 4:2:2 10 Bit Image Transport Summary S3D or M

45、ono Image Format Nominal Frame Rate (Frames/sec) HDSDI Data Mapping HDSDI Serial Interface Link Data Rate (nominal Gb/s) Number of Links Interface Data Rate (nominal Gb/s) Mono 1280720 24 to 60 ST 292-1 ST 292-1 1.5 1 1.5 19201080 24 to 30 20481080 S3D 1280720 24 to 60 ST 292-2 ST 292-1 1.5 2 3 1920

46、1080 24 to 30 20481080 1280720 24 to 60 ST 425-2 ST 424 3 1 3 19201080 24 to 30 20481080 Mono 19201080 48 to 60 ST 425-1 ST 424 3 1 3 20481080 S3D 19201080 48 to 60 ST 425-4 ST 424 3 2 6 20481080 7.2.2 4:4:4 Sampling The 4:4:4/10-bit image formats defined by SMPTE ST 274 and ST 2048-2 for up to and

47、including 30 frames per second frame rates do not fit into a 1.5 Gb/s payload, and can be carried on a SMPTE ST 425-1, 3 Gb/s coaxial interface. SMPTE ST 425-4 was developed to standardize a Dual Link nominal 3 Gb/s interface for S3D. This interface uses a standard SMPTE ST 425-1 Level A or Level B

48、Dual Link interface for each of the left and right eye streams and also defines the payload identifier that will identify the Le and Re images, audio and other associated ancillary data. Once again, SMPTE ST 424 specifies the serial interface for SMPTE ST 425-4 as it utilizes the fundamental data st

49、ructures defined in SMPTE ST 425-1 for each Eye. SMPTE EG 2076-2:2016 Page 8 of 12 pages The 4:4:4/10-bit image formats defined by SMPTE ST 274 and SMPTE ST 2048-2 for higher than 30 frames per second frame rates do not fit into a 3 Gb/s payload, and can be carried on a SMPTE ST 425-5 Dual Link 3 Gb/s coaxial interface. SMPTE ST 425-6 was developed to standardize a Quad Link nominal 3 Gb/s interface for S3D. This interface uses a standard SMPTE ST 425-5 Level A or Level B Dual Link interface for each of the left and right eye s