1、o m TELECOMMUNICATIONS SYSTEMS BULLETIN APCO Project 25 Vocoder Selection Process TSBl02.BABD MAY 1996 TELECOMMUNICATIONS INDUSTRY ASSOCIATION Copyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license
2、 from IHS-,-,-EIA TSBLQ2-WAD %b 3234600 0573586 579 W 6 NOTICE EIAmA Engineering Standards and Publications are designed to serve the public interest through eliminating niisunderstandings between manufacturers and purchasers, facilitating interchangeability and improvement of products, and assistin
3、g the purchaser in selecting and obtaining with minimum . delay the proper product for his particular need. Existence of such Standards and Publications shall not in any respect preclude any member or nonmember of EMIA fiom manufacturing or selling products not conforming to such Standards and Publi
4、cations, nor shall the existence of such Standards and Publications preclude their voluntary use by those other than EIAiTIA members, whether the standard is to be used either domestically or internationally. Standards, Publications and Bulletins are adopted by EIAAIA in accordance with the American
5、 National Standards Institute (ANSI) patent policy. By such action, EINTIA does not assume any liabii to any patent owner, nor does it assume any obligation whatsoever to parties adopting the Standard, Publication or Bulletin. Technical Bulletins are distinguished fiom EIAAIA Standards or Interim St
6、andards, in that they contain a compilation of engineering data or information usefbl to the technical community, and represent approaches to good engineering practices that are suggested by the formulating committee. This Bulletin is not intended tq preclude or discourage other approaches that simi
7、larly represent good engineering practice, or that may be acceptable to, or have been accepted by, appropriate bodies. Parties who wish to bring other approaches to the attention of the formulating committee to be considered for inclusion in fiiture revisions of this Bulletin-are encouraged to do so
8、. It is the intention of the formulating committee to revise and update this Bulletin fiom time to time as may be occasioned by changes in technology, industry practice, or government regulations, or for other appropriate reasons. (Formulated under the cognizance of the TR-8.4 Working Group on Vocod
9、er of the TR-8 Committee on Land Mobile Services.) Published by QELECTRONIC INDUSTRIES ASSOCIATION 1996 Engineering Department 2500 Wilson Boulevard Arlington, VA 22201 PRICE: Please refer to the current Catalog of EIA, JEDEC, and TIA STANDARDS and ENGINEERING PUBLICATIONS or call Global Engineering
10、 Documents, USA and Canada (1-800-854-7179) International (303-397-7956) All rights reserved Printed in U.S.A. Copyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,- EIA TSBLOZ=BABD c
11、fb 3234600 0573587 405 Foreword This foreward is not a part of this bulletin. This Telecommunications System Bulletin (TSB) is being promulgated to collect a series of documents and audio source material which has been previously generated in support of the Vocoder selection process for the APCO Pro
12、ject 25 Digital Radio Standards. This TSB captures the results of the Vocoder selection process. The following individual items have been collected and comprise this TSB: Individual documents: TIA VOCODER TST.PLAN FOR APCOINASTDIFED, 4th Draft December 2,1991 ADDENDUM TO 4th DRAFT OF TIA VOCODER TES
13、T PLAN FOR APCOINASTDffED, February 14,1992 RESULTS OF SUBJECTIVE VOICE QUALITY TEST FOR THE TIA, July 10,1992. Copies of the audio source material used for testing Copies of the processed audio used for scoring. Since this TSB is historical in nature, no maintenance of this document is required. Co
14、pyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-EIA TSBLO2-BABD 9b 3234b00 057L588 341 TSB- 102.BABD Table of Contents Section 1 . TIA VOCODER TEST PLAN FOR APCOINASTDIFED. 4th Dr
15、aft December 2. 1991 I nt roductio n 1 . 2 2 . Speech Data Bases . 3 2.1. Speech Data Base Requirements . 4 2.2. Speech Data Base Calibration Conditions 4 2.4. Speech Data Base for Experiment #2 DMOS . 5 3 . Production of Digital Vocoder Recorded Tapes . 6 3.1 . Equipment List 6 3.2. TIA Project 25
16、Vocoder Test Platform Description 6 3.3. TIA Project 25 Vocoder Test Platform Operation Check . 7 3.4. Test Setup . 8 3.4.1 . Audio Levels 10 3.4.2. Channel Error Randomizer . 10 3.5. Vocoder Testing 11 3.5.1. Vocoder Setup 12 3.5.3. Verification Measurements (Sync, Delay, and Audio) 12 3.5.4. Chann
17、el Error Test . 12 3.5.5. Reduced Level Input Test 13 3.5.6. Clean Channel Test 13 14 2.3. Speech Data Base for Experiment #1 MOS 4 . 3.5.2. System Reset . 12 3.5.7. Background Noise Test 3.6. Vocoder Provider Signoffs 14 4 . Subjective Evaluation of Speech Quality . 14 4.1 . Transmission Quality Te
18、sting 14 4.1.1 . Conducting the MOS Listening Test . 15 4.2. Audio Background Noise Conditions . 16 4.2.1 . Conducting the vlOS Listening Test 17 Copyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without lice
19、nse from IHS-,-,- EIA TSBL02-BABD 96 U 3234600 0571589 288 TSB- 102.BABD 4.3. Stimulus DAT Preparation . 18 4.3.1 . Experiment #1 Randomization . 4.3.2. Experiment #2 Randomization . 20 21 4.4. MOS/DMOS Evaluation Laboratory 22 4.5. Reporting of Results 24 5 . Vocoder Ranking . 25 5.1. Overall Quali
20、ty Score 25 5.2. Overall Figure of Merit 25 5.3. Complexity Evaluation . 26 5.3.1. Memory Requirements . 26 5.3.2. Arithmetic Operations . 27 5.3.3. MIPS Requirements . 27 5.4. Delay Calculation 27 6 Operational Aspects of the Testing 29 6.1 . Third Party Observers or the candidate coder or Test Pla
21、tform stops functioning; or the candidate coder representative believes that the coder is malfunctioning, the testing will cease, all audio tapes will be rewound and testing will begin anew from the start. Stopping formal testing for whatever reason requires testing recommence at the first test conf
22、iguration (step 3 below). If formal testing is stopped by the candidate coder representative, modifications and/or replacement of coder components will be allowed. A retest will start during the scheduled session if time allows. Otherwise, a retest will be scheduled for the next available retest ses
23、sion. The sections below list the steps for running test material through the digital vocoders and producing processed tapes. For each vocoder, use a Vocoder Test Log form (section 8.7). Program markers on the recording DATs will be enscribed by the test technician at the onset of each sentence pair
24、. These IDS and the corresponding DAT box label will be the index used for assembling presentation scram blings for evaluation. 1. The calibrated reference tone will be played into the Test Platform operating in a loopback mode with Input Level set to U dB. The wavefom analyzer will be used to assur
25、e proper input level ( 1 Vp-p). Any discrepancies will be noted on the Vocoder Test hg fm. 2. Insert DAT test source tape labelled APCO Project 25 / Source Material into the Source DAT deck and blank tapes into Target DAT decks. . 3. Rewind DATs on ail DAT recorders. 4. The recording level of the ta
26、rget DAT recorder should be adjusted so that the meters on both channels read -10 dB for the test tone. This places the EPL record level at 10 dB below the DAT saturation level (Le. +9 dBm). The reference tone is now recorded and dubbed to the output DATs. All DAT decks are now paused. Note: There w
27、ill be no adjustments to playback or record levels after this point. 5. Setup Digital Vocoder and Test Platform (section 3.5.1) 6. Perform System Reset (section 3.5.2) 7. Verification Measurements (section 3.5.3) 8. Do Channel Error test (section 3.5.4) 9. Do Reduced Level test (section 3.5.5) 10. D
28、o Clean Channel test (section 3.5.6) Copyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-EIA TSBL02mBAD 96 3234600 0573602 543 TSB-102.BABD 11. Do Background Noise test (section 3.5
29、.7) 12. Remove the vocoder and place the Test Platform in loopback mode, set to its “clean“ settings. Record a final test tone. Check levels as specified in section 3.3; note any discrepancies on the Vocoder Test Log form. 13. At the end of the test, stop ail of the DAT decks, remove the recorded ta
30、pes and mark them APCO Project 25 / Vocoder - /Date set the BER toggle switch to High. Reduced Level Input Test This is a two-part test, using two attenuated audio levels. Each part, consisting of 16 spoken sentence pairs, takes about 5 minutes. To make it easy to make the switch changes, a double b
31、eep tone is provided on the source test tape. The test technician should listen with headphones for these prompt beeps as follows: 1. At the sound of the double beep tone, set the Test Platform switches as follows: a. The Errors switch to Off b. Input Level switch to -1OdB. the tone, set the Input L
32、evel switch to -2OdB. 2. After about 5 minutes, begin listening for the beep tone. Be prepared, and when you hear 3.5.6 Clean Channel Test 13 Copyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license
33、from IHS-,-,-EIA TSBLOZ.BAED 96 U 3234600 057LbO4 314 TSB-102.BABD This series of 128 sentence pairs takes about 43 minutes. The Errors switch remains in the Off position. 1. At the sound of the double beep tone, set the Test Platform switches as follows: a. Input Level switch to OdB 3.5.7 Backgroun
34、d Noise Test This is a multi-part test of 96 sentence pairs. The Errors switch is still in the Offposition and the Input Level attenuator is still at OdB. 1. At the sound of the double beep tone, the test session runs on its own for 32 minutes. 3.6 Vocoder Provider Signoffs Each vocoder provider wil
35、l be required to sign a form (see Appendix F) verifying that their vocoder received a fair test. The signature is required before the test tapes are sent to the organization performing the scrambling and listening tests. The unprocessed speech data base is for vocoder sponsor comparison use only and
36、 is not to be used for other formal testing without the approval of WRT and ARCON Corporation. Each provider should listen to the coded audio tapes during and after their test session. (A fifth DAT deck is available for offline auditioning of recorded tapes). If the tapes are deemed unacceptable, a
37、retest will be available to vocoder providers not later than Tuesday December 10th. At the time of signoff, the vocoder provider will be given two digital audio tapes; one containing the unprocessed speech, a second containing speech processed by his companys vocoder. 4. Subjective Evaluation of Spe
38、ech Quality The subjective evaluation must be based upon widely recognized and approved listening test procedures. Listening test procedures similar to those described below have been used in severai vocoder evaluations. The codec selection processes for North -American Digital Cellular, Japan Digit
39、al Cellular, European Digital Cellular (GSM) and CCITs proposed 16 kbps speech coding standard have all empioyed similar subjective tests. Subjective testing involves the use of a number of listeners to subjectively rate the quality of the processed speech. The processed speech samples are obtained
40、by testing the candidate vocoders under a variety of test conditions of interest. Because quality assessment is subject to individual preferences, reliable results are obtain when a fairly large number of listeners are used. For this assessment a total of 32 listeners will be used. To mitigate the e
41、ffects of the order of presentation of the speech samples to the listeners, eight different presentation orders will be used with four listeners hearing each of the different presentation orders. 4.1 Transmission Quality Testing The subjective evaluation will consist of two experiments. The second e
42、xperiment will test the vocoders under the background noise conditions while the first experiment will test vocoders under the remaining conditions. The first experiment is to determine effects of error rate, input level variation and talker variability upon the performance of each vocoder. The talk
43、er variability is determined from the clean conditions only. The listening test for the first experiment will follow the Absolute Category Rating Method 1 i. In this type of test, each listener rates each test condition on a scale of 1 through 5 (1 = bad, 2 = 14 Copyright Telecommunications Industry
44、 Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-EIA TSBLOZ-BAED 96 = 3234600 0573605 250 TSB-102.BABD poor, 3 = fair, 4 = good, 5 = excellent) based on the listeners assessment of overall speech quality. Each as1;e
45、ssment is based on listening to a pair of sentences for each test condition. The results averaged ;icross listeners is referred to the Mean Opinion Score (MOS). The experiment consists of the following test conditions: Number of Experimental Conditions Clean nominal input level, no background noise,
46、 no channel errors with 16 taikers (8M, 8F) Channel Errors: nominal input level, no background noise 3 speeds (5,30,60 mph) each at 2 of 3 error rates (1,3,5%) 5 mph - 1,396; 30 mph - 1,596; 60 mph - 3,5% with 4 talkers (2M, 2F) Reduced Input level 2 levels (-10, -20 dB from nominal) with 4 talkers
47、(2M, 2F) TOTAUspeech coder Conditions 16 24 - a 48 For calibration, reference conditions must also be included. The following MNRU (Modulated Noise Reference Unit) conditions as well as the original PCM should be included: 5 Q levels (5, 25,20,25, 30 d3) that is, they should not be experts in teleph
48、one design, digital voice encoding algorithms, and so on. They should not be trained listeners; that is, they should not have been trained in these or previous listening studies using feedback trials. The sample should be adults of mixed sex and age, with the constraint that persons over the age of
49、40 should have been given a recent audiogram to ensure that their hearing is still within the normal sensitivity range. Conducting the MOS Listening Test 15 Copyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-EIA TSBLO2-BABD 96 E 3234600 0571bOb 197 , TSB-102.BABD Bad Poor Fair
