ITU-T P 862 AMD 2-2005 Perceptual evaluation of speech quality (PESQ) An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codepl.pdf

1、 International Telecommunication Union ITU-T P.862TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU Amendment 2(11/2005) SERIES P: TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Methods for objective and subjective assessment of quality Perceptual evaluation of speech qua

2、lity (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs Amendment 2: Revised Annex A Reference implementations and conformance testing for ITU-T Recs P.862, P.862.1 and P.862.2 ITU-T Recommendation P.862 (2001) Amendment 2 ITU-T P

3、-SERIES RECOMMENDATIONS TELEPHONE TRANSMISSION QUALITY, TELEPHONE INSTALLATIONS, LOCAL LINE NETWORKS Vocabulary and effects of transmission parameters on customer opinion of transmission quality Series P.10 Subscribers lines and sets Series P.30 P.300 Transmission standards Series P.40 Objective mea

4、suring apparatus Series P.50 P.500 Objective electro-acoustical measurements Series P.60 Measurements related to speech loudness Series P.70 Methods for objective and subjective assessment of quality Series P.80 P.800 Audiovisual quality in multimedia services Series P.900 Transmission performance a

5、nd QoS aspects of IP end-points Series P.1000 For further details, please refer to the list of ITU-T Recommendations. ITU-T Rec. P.862 (2001)/Amd.2 (11/2005) i ITU-T Recommendation P.862 Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of n

6、arrow-band telephone networks and speech codecs Amendment 2 Revised Annex A Reference implementations and conformance testing for ITU-T Recs P.862, P.862.1 and P.862.2 Summary This revision to Annex A/P.862 describes the revised ANSI C reference implementation of PESQ, which includes modes of operat

7、ion for Recommendation P.862.1 (narrowband MOS-LQO mapping) and Recommendation P.862.2 (the wideband extension). It also describes the conformance testing procedures for P.862 and P.862.2. This revision replaces Annex A to P.862 (February 2001) and Amendment 1 to P.862 (March 2003). Source Amendment

8、 2 to ITU-T Recommendation P.862 (2001) was approved on 29 November 2005 by ITU-T Study Group 12 (2005-2008) under the ITU-T Recommendation A.8 procedure. ii ITU-T Rec. P.862 (2001)/Amd.2 (11/2005) FOREWORD The International Telecommunication Union (ITU) is the United Nations specialized agency in t

9、he field of telecommunications. The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of ITU. ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis.

10、The World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes the topics for study by the ITU-T study groups which, in turn, produce Recommendations on these topics. The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolutio

11、n 1. In some areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborative basis with ISO and IEC. NOTE In this Recommendation, the expression “Administration“ is used for conciseness to indicate both a telecommunication administration and

12、a recognized operating agency. Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain mandatory provisions (to ensure e.g. interoperability or applicability) and compliance with the Recommendation is achieved when all of these mandatory provisions are met.

13、The words “shall“ or some other obligatory language such as “must“ and the negative equivalents are used to express requirements. The use of such words does not suggest that compliance with the Recommendation is required of any party. INTELLECTUAL PROPERTY RIGHTS ITU draws attention to the possibili

14、ty that the practice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. ITU takes no position concerning the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of the Recom

15、mendation development process. As of the date of approval of this Recommendation, ITU had received notice of intellectual property, protected by patents, which may be required to implement this Recommendation. However, implementors are cautioned that this may not represent the latest information and

16、 are therefore strongly urged to consult the TSB patent database. ITU 2006 All rights reserved. No part of this publication may be reproduced, by any means whatsoever, without the prior written permission of ITU. ITU-T Rec. P.862 (2001)/Amd.2 (11/2005) iii CONTENTS Page A.1 Files provided. 1 A.2 Sam

17、pling rate 2 A.3 Conformance tests 2 ITU-T Rec. P.862 (2001)/Amd.2 (11/2005) 1 ITU-T Recommendation P.862 Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs Amendment 2 Revised Annex A Refer

18、ence implementations and conformance testing for ITU-T Recs P.862, P.862.1 and P.862.2 A.1 Files provided A.1.1 List of files provided for the ANSI-C reference implementation The ANSI-C reference implementation of ITU-T Recs P.862, P.862.1 and P.862.2 is contained in the following text files which a

19、re provided in the source sub-directory of the CD-ROM distribution: dsp.c Basic DSP routines dsp.h Header file for dsp.c pesq.h General header file pesqdsp.c PESQ DSP routines pesqio.c File input/output pesqmain.c Main program pesqmod.c PESQ high-level model pesqpar.h PESQ perceptual model definitio

20、ns The ANSI-C reference implementation is provided in separate files and forms an integral part of this Recommendation and ITU-T Recs P.862.1 and P.862.2. The ANSI-C reference implementation shall take precedence in case of conflicts between the high-level description and the ANSI-C reference implem

21、entation. A.1.2 List of files provided for conformance validation The conformance validation process described below makes reference to the following files, which are provided in the conform sub-directory of the CD-ROM distribution: supp23_16k.txt File pairs and PESQ scores for test 1(a) supp23_16k.

22、bat Batch script to assist with test 1(a) supp23_8k.txt File pairs and PESQ scores for test 1(b) supp23_8k.bat Batch script to assist with test 1(b) voipref_16k.txt File pairs and PESQ scores for test 2(a) voipref_16k.bat Batch script to assist with test 2(a) voipref_8k.txt File pairs and PESQ score

23、s for test 2(b) voipref_8k.bat Batch script to assist with test 2(b) process.bat Sample batch script to assist with preparing material for tests 1(b) and 2(a) supp23_wb.txt File pairs and P.862.2 scores for conformance validation to Supplement 23 (wideband operation) 2 ITU-T Rec. P.862 (2001)/Amd.2

24、(11/2005) supp23_wb.bat Batch script to assist with validation of P.862.2 scores Speech files provided for validation of P.862 with variable delay: or105.wav or109.wav or114.wav or129.wav or134.wav or137.wav or145.wav or149.wav or152.wav or154.wav or155.wav or161.wav or164.wav or166.wav or170.wav or

25、179.wav or221.wav or229.wav or246.wav or272.wav dg105.wav dg109.wav dg114.wav dg129.wav dg134.wav dg137.wav dg145.wav dg149.wav dg152.wav dg154.wav dg155.wav dg161.wav dg164.wav dg166.wav dg170.wav dg179.wav dg221.wav dg229.wav dg246.wav dg272.wav u_am1s01.wav u_am1s02.wav u_am1s03.wav u_am1s01b1c1.

26、wav u_am1s01b1c7.wav u_am1s01b1c15.wav u_am1s02b1c9.wav u_am1s03b1c16.wav u_am1s03b1c18.wav u_am1s01b2c1.wav u_am1s01b2c8.wav u_am1s02b2c4.wav u_am1s02b2c5.wav u_am1s02b2c14.wav u_am1s03b2c5.wav u_am1s03b2c6.wav u_am1s03b2c7.wav u_am1s03b2c11.wav u_am1s03b2c18.wav u_af1s01.wav u_af1s02.wav u_af1s03.

27、wav u_af1s01b2c16.wav u_af1s03b2c16.wav u_af1s02b2c17.wav u_af1s03b2c17.wav The variable delay speech files are in Wave format (16-bit linear PCM, Intel byte ordering, 44 byte header), at 8 kHz sample rate. These files form an integral part of this annex. A.2 Sampling rate An implementation of ITU-T

28、 Rec. P.862 may, at the implementers discretion, operate at 8 kHz sampling rate, 16 kHz sampling rate, or both. However, the implementation must pass the conformance tests set for all sampling rates that are offered by the implementation. Implementations of the P.862.2 wideband extension must operat

29、e at 16kHz sampling rate. A.3 Conformance tests A.3.1 Conformance data sets The data sets for the conformance tests are as follows. Test Number of file pairs (a) 16 kHz data set (b) 8 kHz data set Type of test 1 1736 ITU-T P-series Supplement 23 Downsampled from ITU-T P-series Supplement 23 using IT

30、U-T Software Tool Library (version 2000, release 31) and process.bat. Mandatory 2 40 Upsampled from P.862 VoIP variable delay data using Software Tool Library (version 2000, release 3) and process.bat. ITU-T Rec. P.862 Annex A VoIP variable delay data. Mandatory 3 No data set defined. This test is o

31、pen-ended, based on general, unknown data. Mandatory 4 1736 ITU-T P-series Supplement 23 Not applicable Mandatory for P.862.2 _ 1ITU-T Rec. G.191 (2005), Software tools for speech and audio coding standardization. ITU-T Rec. P.862 (2001)/Amd.2 (11/2005) 3 A.3.2 Conformance requirements The test requ

32、irements are summarized in the following table and are set out in detail below. The requirements are based on the absolute difference in PESQ score between the implementation under test and the ANSI-C reference implementation, calculated for each reference and degraded file pair. For the conformance

33、 tests defined in Annex A/P.862 (February 2001), there is no change. Test Number of file pairs Lower threshold Upper threshold Type of test 1(a) 1736 Difference may not exceed 0.05 in any situation. Not applicable Mandatory 1(b) 1736 Difference may exceed 0.05 in not more than 2 file pairs (approx.

34、0.1% of cases). Difference may not exceed 0.1 in any case. Mandatory 2(a) 40 Difference may exceed 0.05 in not more than 1 file pair (2.5% of cases). Difference may not exceed 0.5 in any case. Mandatory 2(b) 40 Difference may exceed 0.05 in not more than 1 file pair (2.5% of cases). Difference may n

35、ot exceed 0.5 in any case. Mandatory 3 No data set defined Difference may exceed 0.05 in not more than 0.5% of cases. Difference may exceed 0.05 in not more than 5% of cases. Lower threshold is advisory. Upper threshold is mandatory. 4 1736 Difference may not exceed 0.05 in any situation. Not applic

36、able Mandatory for P.862.2 operation A.3.2.1(a) Conformance test 1(a) (16 kHz sampling rate) In this test, all files from all ten experiments as released with ITU-T P-series Supplement 23 are used, on a file-by-file basis. The Supplement 23 data is all at 16 kHz sampling rate. The original and degra

37、ded file names, and the PESQ score given by the reference implementation, are provided in the files listed above. An implementation passes this test when the absolute difference in the raw PESQ score compared to the reference implementation is not greater than 0.05 in all cases. This conformance tes

38、t is mandatory for all implementations of PESQ at 16 kHz sampling rate. ITU-T P-series Suppplement 23 can be obtained separately from the ITU. A.3.2.1(b) Conformance test 1(b) (8 kHz sampling rate) In this test, 8 kHz resampled versions of the Supplement 23 files are used, on a file-by-file basis. T

39、he original and degraded files must be downsampled using the ITU-T Software Tool Library 2000 release 3, program filter, using the following command: filter -down HQ2 inputfile.raw outputfile.raw This assumes that the 16 kHz input speech file is called inputfile.raw and the 8 kHz output file is call

40、ed outputfile.raw. 4 ITU-T Rec. P.862 (2001)/Amd.2 (11/2005) A batch script to assist with this, and the original and degraded file names, and the raw PESQ score given by the reference implementation, are provided in the files listed above. An implementation passes this test when the absolute differ

41、ence in the raw PESQ score compared to the reference implementation is not greater than 0.05 in more than 2 file pairs (these may be any two of the file pairs), and not greater than 0.1 in all cases. This conformance test is mandatory for all implementations of PESQ at 8 kHz sampling rate. ITU-T P-s

42、eries Suppplement 23 can be obtained separately from the ITU. A.3.2.2(a) Conformance test 2(a) (16 kHz sampling rate) This test is based on data provided with PESQ and described in this annex. In this test, 16 kHz resampled versions of the Annex A/P.862 VoIP test files are used on a file-by-file bas

43、is. The original and degraded files must be upsampled using the ITU-T Software Tool Library 2000 release 3, program filter, using the following command: filter -up HQ2 inputfile.raw outputfile.raw This assumes that the 8 kHz input speech file is called inputfile.raw and the 16 kHz output file is cal

44、led outputfile.raw. A batch script to assist with this, and the original and degraded file names, and the raw PESQ score given by the reference implementation, are provided in the files listed above. An implementation passes this test when the absolute difference in the raw PESQ score compared to th

45、e reference implementation is not greater than 0.05 in more than 1 file pair (this may be any one of the file pairs), and not greater than 0.5 in all cases. This conformance test is mandatory for all implementations of PESQ at 16 kHz sampling rate. A.3.2.2(b) Conformance test 2(b) (8 kHz sampling ra

46、te) A composite database was constructed for Annex A/P.862 from 40 conditions (file pairs) from two subjective tests covering real and simulated VoIP connections that exhibit time-varying delay. Many of these file pairs also trigger the bad interval realignment process. This data is provided at 8 kH

47、z sampling rate as the Annex A/P.862 VoIP test files, and these are used on a file-by-file basis. The original and degraded file names, and the raw PESQ score given by the reference implementation, are provided in the files listed above. An implementation passes this test when the absolute differenc

48、e in the raw PESQ score compared to the reference implementation is not greater than 0.05 in more than 1 file pair (this may be any one of the file pairs), and not greater than 0.5 in all cases. This conformance test is mandatory for all implementations of PESQ at 8 kHz sampling rate. A.3.2.3 Confor

49、mance test 3 (8 kHz or 16 kHz sampling rate) Additional comparisons To prevent implementers from specifically tailoring an algorithm to conform to requirements for the files described above, a further test is available. An implementation of PESQ that conforms to ITU-T Rec. P.862 must, in at least 95% of cases, give an output score that is within 0.05 of the raw PESQ score given by the ANSI-C reference implementation. These cases must be based on speech files covering a representative sample of reasonable telephone