1、 I n t e r n a t i o n a l T e l e c o m m u n i c a t i o n U n i o n ITU-T P.863.1 TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (09/2014) SERIES P: TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS Methods for objective and subjective assessment of speech quality Application guide for R
2、ecommendation ITU-T P.863 Recommendation ITU-T P.863.1 ITU-T P-SERIES RECOMMENDATIONS TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS Vocabulary and effects of transmission parameters on customer opinion of transmission quality Series P.10 Voice terminal characteristics Series P.30 P.300 R
3、eference systems Series P.40 Objective measuring apparatus Series P.50 P.500 Objective electro-acoustical measurements Series P.60 Measurements related to speech loudness Series P.70 Methods for objective and subjective assessment of speech quality Series P.80 P.800 Audiovisual quality in multimedia
4、 services Series P.900 Transmission performance and QoS aspects of IP end-points Series P.1000 Communications involving vehicles Series P.1100 Models and tools for quality assessment of streamed media Series P.1200 Telemeeting assessment Series P.1300 Statistical analysis, evaluation and reporting g
5、uidelines of quality measurements Series P.1400 Methods for objective and subjective assessment of quality of services other than voice services Series P.1500 For further details, please refer to the list of ITU-T Recommendations. Rec. ITU-T P.863.1 (09/2014) i Recommendation ITU-T P.863.1 Applicati
6、on guide for Recommendation ITU-T P.863 Summary Recommendation ITU-T P.863.1 provides important remarks that should be taken into account in the objective quality evaluation of speech conforming to Recommendation ITU-T P.863. Users of ITU-T P.863 should understand and follow the guidance given in th
7、is Recommendation. This Recommendation is a supplementary guide for users of Recommendation ITU-T P.863, which describes a means of estimating listening speech quality by using reference and degraded speech samples. The scope of Recommendation ITU-T P.863 is clearly defined in itself. This Recommend
8、ation does not extend or narrow that scope; rather, it provides necessary and important information for obtaining stable, reliable and meaningful objective measurement results in practice. History Edition Recommendation Approval Study Group Unique ID* 1.0 ITU-T P.863.1 2013-05-14 12 11.1002/1000/119
9、35 2.0 ITU-T P.863.1 2014-09-11 12 11.1002/1000/12175 _ * To access the Recommendation, type the URL http:/handle.itu.int/ in the address field of your web browser, followed by the Recommendations unique ID. For example, http:/handle.itu.int/11.1002/1000/11830-en. ii Rec. ITU-T P.863.1 (09/2014) FOR
10、EWORD The International Telecommunication Union (ITU) is the United Nations specialized agency in the field of telecommunications, information and communication technologies (ICTs). The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of ITU. ITU-T is responsible for studyin
11、g technical, operating and tariff questions and issuing Recommendations on them with a view to standardizing telecommunications on a worldwide basis. The World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes the topics for study by the ITU-T study groups
12、which, in turn, produce Recommendations on these topics. The approval of ITU-T Recommendations is covered by the procedure laid down in WTSA Resolution 1. In some areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborative basis with ISO
13、and IEC. NOTE In this Recommendation, the expression “Administration“ is used for conciseness to indicate both a telecommunication administration and a recognized operating agency. Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain mandatory provisions
14、(to ensure, e.g., interoperability or applicability) and compliance with the Recommendation is achieved when all of these mandatory provisions are met. The words “shall“ or some other obligatory language such as “must“ and the negative equivalents are used to express requirements. The use of such wo
15、rds does not suggest that compliance with the Recommendation is required of any party. INTELLECTUAL PROPERTY RIGHTSITU draws attention to the possibility that the practice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. ITU takes no position con
16、cerning the evidence, validity or applicability of claimed Intellectual Property Rights, whether asserted by ITU members or others outside of the Recommendation development process. As of the date of approval of this Recommendation, ITU had not received notice of intellectual property, protected by
17、patents, which may be required to implement this Recommendation. However, implementers are cautioned that this may not represent the latest information and are therefore strongly urged to consult the TSB patent database at http:/www.itu.int/ITU-T/ipr/. ITU 2015 All rights reserved. No part of this p
18、ublication may be reproduced, by any means whatsoever, without the prior written permission of ITU. Rec. ITU-T P.863.1 (09/2014) iii Table of Contents Page 1 Scope . 1 2 References . 1 3 Definitions 1 3.1 Terms defined elsewhere 1 3.2 Terms defined in this Recommendation . 1 4 Abbreviations and acro
19、nyms 1 5 Conventions 2 6 Introduction to ITU-T P.863 . 2 6.1 History of objective speech quality 2 6.2 Basics of a subjective test . 3 6.3 Benefit of using Recommendation ITU-T P.863 . 4 6.4 Relationship with other Recommendations 4 6.5 Moving from ITU-T P.862 to ITU-T P.863 . 4 6.6 Challenges with
20、ITU-T P.863 . 4 6.7 MOS misconceptions 5 7 Operational modes 5 7.1 Why two operational modes? . 5 7.2 When should the narrowband mode be used? 6 7.3 When should the super-wideband mode be used? 6 7.4 Why no wideband (up to 7 kHz) mode? . 6 7.5 Are narrowband signals scored equally in narrowband and
21、super-wideband modes? . 6 7.6 Can I map a narrowband score to a super-wideband score? 6 7.7 Is it recommended to mix bandwidths in subjective testing? . 6 8 Influence of reference speech on scores . 7 8.1 What characteristics should a reference signal contain? 7 8.2 Are there constraints on the reco
22、rding environment? . 7 8.3 Can we use artificial speech signals? . 7 8.4 What filter specification should be used? . 7 8.5 Does reference material influence scores? . 8 8.6 How much reference material should I use? 8 8.7 What is the impact of silence padding? 8 8.8 How do I validate a reference sign
23、al? 9 8.9 How do I use a 10 minute reference file? . 9 9 How to inject test signals 10 9.1 Over which interfaces can signals be injected into a network? 10 9.2 Should the test signal be filtered? . 10 Page iv Rec. ITU-T P.863.1 (09/2014) 9.3 What level should be used? 10 10 Influence of recording pr
24、ocess on scores 10 10.1 What influence does file speech level have on score? . 10 10.2 Should we prealign signal levels? 11 10.3 Should we apply a filter to the recording? 11 10.4 Does silence padding affect the ITU-T P.863 scores? . 11 10.5 Can I assess a narrowband recording in super-wideband mode
25、? . 11 11 Behaviour of ITU-T P.863 12 11.1 Global level changes . 12 11.2 Local level changes 13 11.3 Bandwidth limitations 13 11.4 Stretching and compressing speech 13 11.5 Step delay changes . 14 11.6 Long-term gaps in speech . 14 11.7 Short-term gaps in speech 15 11.8 Cellular handovers 15 12 Com
26、parison of ITU-T P.862.1/ITU-T P.862.2 and ITU-T P.863. 15 12.1 Scale differences . 15 12.2 Degradation handling differences . 15 12.3 Standard codecs 16 13 Procedure for comparing subjective test results to ITU-T P.863 results 17 14 Validation scope . 19 14.1 Validated . 19 14.2 Not yet validated .
27、 20 14.3 Outside scope 21 Appendix I ITU-T P.863 Transparency checks of reference files 22 Appendix II Typical scores expected from ITU-T P.863 for a given codec . 24 Appendix III Applications for further investigation . 29 Bibliography. 30 Rec. ITU-T P.863.1 (09/2014) 1 Recommendation ITU-T P.863.1
28、 Application guide for Recommendation ITU-T P.863 1 Scope This Recommendation is a supplementary guide for users of ITU-T P.863, which describes a means of estimating listening speech quality by using reference and degraded speech samples. It assumes that an objective quality algorithm strictly conf
29、orms to ITU-T P.863. This can be confirmed by the conformance test provided in ITU-T P. 863. This Recommendation does not extend or narrow the scope of ITU-T P.863; rather, it provides necessary and important information for obtaining stable, reliable and meaningful objective measurement results in
30、practice. It also provides example results for common situations and explanations for how certain degradations impact the score. Additional characterization results will be added to this Recommendation, in the form of appendices, as this information becomes available. 2 References The following ITU-
31、T Recommendations and other references contain provisions which, through reference in this text, constitute provisions of this Recommendation. At the time of publication, the editions indicated were valid. All Recommendations and other references are subject to revision; users of this Recommendation
32、 are therefore encouraged to investigate the possibility of applying the most recent edition of the Recommendations and other references listed below. A list of the currently valid ITU-T Recommendations is regularly published. The reference to a document within this Recommendation does not give it,
33、as a stand-alone document, the status of a Recommendation. ITU-T P.10 Recommendation ITU-T P.10/G.100 (2006), Vocabulary for performance and quality of service; plus its amendments. ITU-T P.863 Recommendation ITU-T P.863 (2014), Perceptual Objective Listening Quality Assessment. 3 Definitions 3.1 Te
34、rms defined elsewhere This Recommendation uses the terms and definitions defined in ITU-T P.10. 3.2 Terms defined in this Recommendation None. 4 Abbreviations and acronyms This Recommendation uses the following abbreviations and acronyms: ACR Absolute Category Rating AGC Automatic Gain Control ASL A
35、ctive Speech Level IRS Intermediate Reference System MIRS Modified Intermediate Reference System 2 Rec. ITU-T P.863.1 (09/2014) MOS Mean Opinion Score MOS-LQO Mean Opinion Score Listening Quality Objective NB Narrowband PCM Pulse Code Modulation PESQ Perceptual Evaluation of Speech Quality PSQM Perc
36、eptual Speech Quality Measure PSTN Public Switched Telephone Network SPL Sound Pressure Level SWB Super-Wideband VoIP Voice over IP 5 Conventions None. 6 Introduction to ITU-T P.863 ITU-T P.863 provides an objective speech quality measurement algorithm for measuring the voice quality of narrowband (
37、NB), wideband and super-wideband (SWB) networks. 6.1 History of objective speech quality Speech quality has no physical definition. Inherently, people just have an opinion about when something sounds good or bad. However, stakeholders need to be able to quantify the quality delivered by a telephone
38、system in order to maximize investment and ensure adequate service is provided to customers. For many years, the only effective manner by which to determine the quality of a telephone network was to perform a subjective test. Subject testing, explained more in the next clause, involves asking a pane
39、l of users what they think of a recording or connection. The panel typically vote on a five point scale, and the average of the votes is deemed to be the quality of the connection. This number is called the mean opinion score (MOS). The running of subjective tests is time consuming and costly. Durin
40、g the late 1980s, compression technologies were introduced in digital networks to increase capacity while reducing costs. Before their introduction, it was generally possible to determine the performance of a network using simple tone-based measurements. With the introduction of new speech processin
41、g technologies, it was found that results from tone-based techniques could contradict users experiences. A new measurement methodology was required. The increased availability of general purpose computing allowed the development of computer programs capable of modelling the results of subjective tes
42、ts. In 1996, Recommendation ITU-T P.861 (perceptual speech quality measure (PSQM) b-ITU-T P.861 was published. The core concept introduced in this first generation algorithm was that human hearing could be modelled to extract a representation of audible differences between a reference and a degraded
43、 pair of signals, and that these differences could be mapped to the scores of subjective tests. Shortly after b-ITU-T P.861 was published, work was started to address practical limitations of the first generation model in terms of its applicability for testing networks. This work led to the publishi
44、ng of a significantly improved model called perceptual evaluation of speech quality (PESQ), which was published as b-ITU-T P.862 in 2001, together with the withdrawal of b-ITU-T P.861. Work continued on b-ITU-T P.862 for a number of years, for example, with the introduction of a wideband Rec. ITU-T
45、P.863.1 (09/2014) 3 extension in 2005. However, as more complex signal processing was added to the telephone network, it became clear that a new model was required. In 2006 ITU-T initiated a new activity for the development of the third generation model. The intention was to provide a backward-compa
46、tible model that could also assess new speech signal processing technologies as well as the anticipated move to super-wideband networks. The result of this work was published as ITU-T P.863 at the beginning of 2011. It should be noted that the introduction of ITU-T P.863 does not deprecate b-ITU-T P
47、.862. 6.2 Basics of a subjective test It is important to consider subjective testing when discussing objective speech quality assessment. Essentially, subjective testing underlies all three generations of objective models described above. The intention for this type of objective model is to predict
48、the result of a listening quality subjective test for a given reference and degraded file pair. ITU provides various Recommendations for performing subjective tests; the most relevant for ITU-T P.863 are b-ITU-T P.800 and b-ITU-T P.830. A subjective test aims to find the average user opinion of a sy
49、stems speech quality by asking a panel of users a directed question and providing a limited-response choice. For example, to determine listening quality, users are asked to rate the quality of the speech by selecting from the scale shown in Table 1. Table 1 MOS Opinion Score Excellent 5 Good 4 Fair 3 Poor 2 Bad 1 A MOS is calculated for a particular condition by averaging the votes of all subjects. This type of test is described as an absolute category rating (ACR) experiment. The
