TIA-102 BABB-1999 Project 25 Vocoder Mean Opinion Score Conformance Test (Includes Access to Additional Content).pdf

1、Acc.ess to Additional Content for TL MOS Source and Reference Material Infonnation contained in the above is the property of the Copyright holder and all Notice of Disclaimer (b) there is no assurance that the Document will be approved by any Committee of TIA or any other body in its present or any

2、other form; (c) the Document may be amended, modified or changed in the standards development or any editing process. The use or practice of contents of this Document may involve the use of intellectual property rights (“IPR”), including pending or issued patents, or copyrights, owned by one or more

3、 parties. TIA makes no search or investigation for IPR. When IPR consisting of patents and published pending patent applications are claimed and called to TIAs attention, a statement from the holder thereof is requested, all in accordance with the Manual. TIA takes no position with reference to, and

4、 disclaims any obligation to investigate or inquire into, the scope or validity of any claims of IPR. TIA will neither be a party to discussions of any licensing terms or conditions, which are instead left to the parties involved, nor will TIA opine or judge whether proposed licensing terms or condi

5、tions are reasonable or non-discriminatory. TIA does not warrant or represent that procedures or practices suggested or provided in the Manual have been complied with as respects the Document or its contents. If the Document contains one or more Normative References to a document published by anothe

6、r organization (“other SSO”) engaged in the formulation, development or publication of standards (whether designated as a standard, specification, recommendation or otherwise), whether such reference consists of mandatory, alternate or optional elements (as defined in the TIA Engineering Manual, 4th

7、edition) then (i) TIA disclaims any duty or obligation to search or investigate the records of any other SSO for IPR or letters of assurance relating to any such Normative Reference; (ii) TIAs policy of encouragement of voluntary disclosure (see Engineering Manual Section 6.5.1) of Essential Patent(

8、s) and published pending patent applications shall apply; and (iii) Information as to claims of IPR in the records or publications of the other SSO shall not constitute identification to TIA of a claim of Essential Patent(s) or published pending patent applications. TIA does not enforce or monitor c

9、ompliance with the contents of the Document. TIA does not certify, inspect, test or otherwise investigate products, designs or services or any claims of compliance with the contents of the Document. ALL WARRANTIES, EXPRESS OR IMPLIED, ARE DISCLAIMED, INCLUDING WITHOUT LIMITATION, ANY AND ALL WARRANT

10、IES CONCERNING THE ACCURACY OF THE CONTENTS, ITS FITNESS OR APPROPRIATENESS FOR A PARTICULAR PURPOSE OR USE, ITS MERCHANTABILITY AND ITS NONINFRINGEMENT OF ANY THIRD PARTYS INTELLECTUAL PROPERTY RIGHTS. TIA EXPRESSLY DISCLAIMS ANY AND ALL RESPONSIBILITIES FOR THE ACCURACY OF THE CONTENTS AND MAKES N

11、O REPRESENTATIONS OR WARRANTIES REGARDING THE CONTENTS COMPLIANCE WITH ANY APPLICABLE STATUTE, RULE OR REGULATION, OR THE SAFETY OR HEALTH EFFECTS OF THE CONTENTS OR ANY PRODUCT OR SERVICE REFERRED TO IN THE DOCUMENT OR PRODUCED OR RENDERED TO COMPLY WITH THE CONTENTS. TIA SHALL NOT BE LIABLE FOR AN

12、Y AND ALL DAMAGES, DIRECT OR INDIRECT, ARISING FROM OR RELATING TO ANY USE OF THE CONTENTS CONTAINED HEREIN, INCLUDING WITHOUT LIMITATION ANY AND ALL INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES (INCLUDING DAMAGES FOR LOSS OF BUSINESS, LOSS OF PROFITS, LITIGATION, OR THE LIKE), WHETHER BAS

13、ED UPON BREACH OF CONTRACT, BREACH OF WARRANTY, TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY OR OTHERWISE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE FOREGOING NEGATION OF DAMAGES IS A FUNDAMENTAL ELEMENT OF THE USE OF THE CONTENTS HEREOF, AND THESE CONTENTS WOULD NOT BE PUBLISHED BY

14、TIA WITHOUT SUCH LIMITATIONS. TIA/EIA-102.BABBTable of Contents1. Introduction . 11.1 Scope 11.2 Overview 22. Revision History 33. References . 44. Definitions. 55. Speech Data Bases. 65.1 Speech Data Base Requirements 75.2 Speech Data Base Reference Conditions . 75.3 Speech Data Base for MOS Test 7

15、6. Production of Digital Vocoder Recorded Tapes . 106.1 Equipment List . 106.2 Vocoder Test Platform Operation Check . 106.3 Test Setup. 116.3.1 Audio Levels . 126.3.2 Channel Error Randomizer 126.4 Vocoder Testing . 136.4.1 Vocoder Setup. 136.4.2 System Reset . 136.4.3 Channel Error Test 146.4.4 Re

16、duced Level Input Test 146.4.5 Clean Channel Test . 146.4.6 Reference Processing 147. Subjective Evaluation of Speech Quality 157.1 Stimulus Recording Preparation. 157.1.1 Experiment Randomization. 157.2 MOS Evaluation Laboratory 177.3 Transmission Quality Testing 177.3.1 Conducting the MOS Listenin

17、g Test. 187.4 Reporting of Results. 198. MOS Result Analysis 218.1 ANOVA Calculation 218.2 Newman-Keuls Analysis 279. Disclosure of MOS Test Result. 36Annex A. Tables. 39Annex B. Vocoder Test Platform Description . 51Annex C. C Source Files 58TIA/EIA/102.BABBiiThis page intentionally left blank.TIA/

18、EIA/102.BABB11. IntroductionThis standard details definitions and methods of measurement for testing conformance ofspeech codecs used in Project 25 Digital Land Mobile Radio Equipment to the referencespeech codec defined for Project 25. The Project 25 speech codec is defined in Project25 Vocoder Des

19、cription, TIA-102.BABA, reference 13. The purpose of this standard isto assure that a speech codec in any given piece of Project 25 Equipment is compliantwith TIA-102.BABA.Section 5 will discuss the speech data base that was chosen to be used in the vocodertesting and comparison. Section 6 contains

20、the details of the vocoder testing as well asthe speech data base used for the testing. Section 7 discusses the design of the listeningtests themselves. Section 8 discusses the overall evaluation procedure of the vocodercombinations. Section 9 describes the disclosure of the MOS test results. Lastly

21、, anannex is included which contains additional detail.1.1 ScopeThis document specifies the procedures to be employed to test that implementations ofTIA-102.BABA compatible speech codecs comply with the standard. This speech codecis the Improved Multi-Band Excitation (IMBETM, a trademark of Digital

22、Voice SystemsInc.) system described in TIA-102.BABA, Project 25 Vocoder Description. The IMBEspeech codec is used to digitally encode the speech signal and provide forward errorcontrol for transmission at a data rate of 7200 b/s.The need for this standard arises from the fact that implementations of

23、 speech coders donot need to have identical numerical precision in the internal arithmetic operations to becompatible. Since individual designers implement algorithms in different manners, it isnot possible to create a standard set of test vectors to assess conformance to a standard.This document de

24、scribes a series of tests that are used to test conformance to thespecification. These tests do not necessarily ensure that the speech codecs operatesatisfactorily under all possible input signals and it is still a requirement of themanufacturer to ensure that his implementation operates in a consis

25、tent manner. Thepurpose of these tests is to test for standard compliance. The manufacturer is encouragedto provide the highest possible performance but it is not the purpose of these tests todetermine that performance.The testing of the codecs is based upon subjective tests which employ listening t

26、ests tojudge overall speech quality.This standard takes the approach of defining a reference speech codec and then eachmanufacturer tests his implementation relative to the reference. All possiblecombinations of test codec to reference codec are evaluated. The reference speech codecis based upon the

27、 Digital Voice Systems Incorporated implementation of the IMBEalgorithm on the VC-20 Project 25 hardware card.TIA/EIA/102.BABB2Reference (VC-20)Reference (VC-20)Vocoder Under TestVocoder Under TestEncoders DecodersAudio SourceAudio ComparisonFour CombinationsFigure 1-1 Encoder and Decoder Test Combi

28、nations1.2 OverviewThe way that the implementation of the speech coder is tested is by comparing it with astandardized reference implementation. The test works by interoperating the testedvocoder with the reference in all combinations of encode and decode pairs, to verify thataudio is encoded and de

29、coded such that listeners perceive little difference. Thesecombinations of speech coder pairs are diagrammed in figure 1-1.The audio quality of each combination of encoder and decoder is subjectively rated.These ratings are then compared with each other to determine if they are statisticallyequivale

30、nt. The vocoder under test passes the test if the ratings show no significantstatistical variation.The speech quality performance issue is difficult to quantify since it is a subjective issue.There is no known objective measurement that can be performed to rate the acceptabilityof the speech coder p

31、erformance to a human listener. Because of this the performanceevaluation must rely upon subjective testing. The subjective testing involves the use of apanel of listeners who rate the vocoders performance on a 5 point scale. Since opinionsof listeners will vary, the result from a number of listener

32、s is obtained and averaged toobtain an overall score.To evaluate the overall speech quality it is necessary to conduct an experiment in acontrolled manner so that unintentional variation in the scoring is avoided. The purposeof the testing is to determine differences in performance among the differe

33、nt vocodercombinations. The confidence we have that any apparent differences in performance aredue to random statistical variation and not vocoder defects will depend upon how well weprevent differences from occurring in the testing of the various vocoder combinations. Inaddition, the judgment as to

34、 the effect of various impairments upon the vocodercombinations will depend upon how well the introduction of those impairments iscontrolled. The statistical controls for the experiment are given in section 7.The listening test also evaluates the vocoder combinations under a variety of testcondition

35、s. The test conditions were chosen to be representative of those expected to beexperienced in a land mobile radio environment. The test is organized so that each testcondition tests only one aspect of system performance. Hence only a limited number ofoperating conditions are tested. To test all poss

36、ible operating conditions would lead to atest that would be too unwieldy to conduct. The operation of the vocoder with variouschannel impairments, various talkers and speaking volumes is determined through the useof a Mean Opinion Score (MOS) test.The vocoder implementations are electrically interco

37、nnected through a device that isintended to provide the standardized test conditions given above. This device is calledTIA/EIA/102.BABB3the Vocoder Test Platform. A detailed description is given in Annex B. A block diagramof how the Vocoder Test Platform interconnects the vocoders is given in figure

38、 1-2. Theuse of the Test Platform for testing is given in detail in section 6.Voice EncoderVoice Decoderi0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0

39、i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0i0Source AudioOutput AudioVocoder Test PlatformEIA-422 InterfacesFigure 1-2 Vocoder Test Platform Audio FlowThe overall plan of the test for the vocoder is outlined in figure 1-3. The test

40、 begins withthe source audio material. There are several conditions that the source material mustsatisfy, and these are covered in section 5. The source audio material is then passedthrough the vocoder test platform and the different vocoder combinations, with differenttest conditions, to produce a

41、recording of output audio. This procedure is given in section6. The output audio is then randomized in order, and commingled with audio that hascalibrated noise, i.e., Modulated Noise Reference Units (MNRUs) to provide samplessuitable for a listening test. The MNRU audio is also processed by the Voc

42、oder TestPlatform. The randomization step, together with the listening test, is described in section7. The listening step then produces MOS scores, which are statistically analyzed forvariations that are beyond the random expected variations in such an experiment. Thisanalysis is described in sectio

43、n 8. The result of the test is then presented in section 9,which describes a spreadsheet for this analysis.Get Source Audio Material.(section 5)Use the Vocoder Test Platform to test the vocoderimplementation under various test conditions.Also process the MNRU material with the Test Platform.(section

44、 6)Randomize the order of the Output Audio samples.(section 7)Conduct a Listening Test of the audio samples to getMOS results.(section 7)Statistically analyze the MOS results to detect anyexcessive variations.(section 8)TIA/EIA/102.BABB4Present the results in a spreadsheet.(section 9)Figure 1-3 MOS

45、Test Plan2. Revision HistoryVersion 0.0, 22 June 1995, initial version of the document.Version 1.0, 27 October 1995, incorporates numerous comments.Version 2.0, 10 May 1996, adjusted levels, added test software in Annex C.Version 2.1, 18 June 1996, added brief description of source material recordin

46、g.Version 2.2, 24 September 1996, modified ANOVA and N-K tables from letter ballot.TIA/EIA/102.BABB53. References1 IEEE Subcommittee on Subjective Measurements, “IEEE Speech Quality MeasureRecommendation“, IEEE Trans. on Audio RADC-TR-85-46, March 1985.3 J.D. Tardelli, J. LeBlanc and P. Gatewood, “R

47、ADC/EEV Diagnostic Rhyme TestSystem Improvements“, Chapter 8. “DRT/DAM Mastering Program“, Appendix A“DRT/DAM Master Library“; RADC-TR-89-256, November 1989.4 W.D. Voiers “Diagnostic Acceptability Measure for Speech CommunicationSystems“, Proc. 1977 IEEE ICASSP, Hartford, CN. pp 204-207, May 1977.5

48、S.R. Quackenbush, T.P. Barnwell and M.A. Clements, Objective Measures ofSpeech Quality, Prentice Hall, Englewood Cliffs, NJ, 1988. ISBN 0-13-629056-6.6 J.T. Sims and J.D. Tardelli, “The Effects of Controlled Speech Level Input on theIntelligibility Testing of Speech Compression Algorithms“, Proc. of

49、 1985 IEEEICASSP, pp. 430-433.7 J.T. Sims, “A Speech-To-Noise Ratio Measurement Algorithm“, JASA, V78, Num.5, pp. 1671-1674, 1985.8 P.T. Brady, “Equivalent Peak Level: A Threshold-Independent Speech-LevelMeasure“, JASA, V44, pp. 695-699, 1968.9 G.F. Sandy and J.E. Parker, “Digital Voice Processor Consortium Final ReportAPPENDIX A“, MITRE Corp. MTR-84W00053-02, McLean, VA, March 1982.10 CCITT “Telephone Transmission Quality“, Blue Book, series P recommendations,Volume V, Rec. P.81, pages 198ff.11 CCITT “Telephone Transmission Quality“, Blue Book, series P recomme