1、 TECHNICAL REPORT ATIS-0500010 Maintenance Testing The Alliance for Telecommunication Industry Solutions (ATIS) is a technical planning and standards development organization that is committed to rapidly developing and promoting technical and operations standards for the communications and related i
2、nformation technologies industry worldwide using a pragmatic, flexible and open approach. Over 1,100 participants from more than 350 communications companies are active in ATIS 23 industry committees and its Incubator Solutions Program. NOTE - The users attention is called to the possibility that co
3、mpliance with this standard may require use of an invention covered by patent rights. By publication of this standard, no position is taken with respect to the validity of this claim or any patent rights in connection therewith. The patent holder has, however, filed a statement of willingness to gra
4、nt license under these rights on reasonable and nondiscriminatory terms and conditions to applicants desiring to obtain such a license. Details may be obtained from the publisher. ATIS-05000010, Maintenance Testing Is an ATIS standard developed by the Emergency Services Methodologies-(ESM)(G) Subcom
5、mittee under the ATIS Emergency Services Interconnection Forum Committee (ESIF). Published by Alliance for Telecommunications Industry Solutions 1200 G Street, NW, Suite 500 Washington, DC 20005 Copyright 2007 by Alliance for Telecommunications Industry Solutions All rights reserved. No part of this
6、 publication may be reproduced in any form, in an electronic retrieval system or otherwise, without the prior written permission of the publisher. For information contact ATIS at 202.628.6380. ATIS is online at . Printed in the United States of America. ATIS-0500010 iTABLE OF CONTENTS 1 Introduction
7、 . 1 1.1 Scope. 1 1.2 Acknowledgements 1 1.3 References 1 2 Definitions . 2 2.1 Acronyms. 4 Part 1 Maintenance Testing for Accuracy. 6 3 GENERAL Test Requirements. 6 3.1 Introduction. 6 3.2 Useful Life of Data 6 3.3 System Under Test 6 3.4 Accuracy Maintenance Test Trigger Mechanisms 7 3.5 Accuracy
8、Maintenance Test Trigger Examples. 8 4 Equipment Requirements . 8 4.1 Test Phones 8 4.2 Equipment to Establish Ground Truth. 8 4.3 Data Recording Device. 9 5 Software Requirements . 9 5.1 Test Point Ground Truth GPS Reference Database 9 5.2 Mapping Software 9 5.3 Data Recording Software . 9 5.4 Data
9、 Processing Software 9 6 Test Area . 10 6.1 Definition . 10 7 Empirical Test Methods. 10 7.1 General . 10 7.2 Incremental Testing 10 7.3 Sample Size and Distribution 11 7.4 Indoor Testing . 13 8 Predictive Testing in a Maintenance Environment 13 8.1 Predictive Modeling . 13 8.2 Objective. 14 8.3 Lim
10、itations of Modeling Tools for Maintenance Testing. 14 9 Analysis and Summary of Results. 14 9.1 Objective. 15 9.2 Data Format . 15 9.3 Data Analysis Tools and Software 15 9.4 Processing of Test Call Data 15 9.5 Accuracy Maintenance Test Evaluation Criteria 16 9.6 Test Metrics 16 9.7 Data Expiration
11、 . 16 9.8 Contents of the Test Report . 16 Part 2 Maintenance of End-To-End Functionality Testing . 18 ATIS-0500010 ii10 GENERAL TEST REQUIREMENTS. 18 10.1 Introduction. 18 11 Functionality test triggers 18 12 Test methodology . 18 13 The Simple Verification Test 21 13.1 When to use the SVT 22 13.2
12、Simple Verification Test Limits. 23 14 Random Call Testing 24 15 DOCUMENTATION OF RESULTS . 24 16 Success criteria. 24 17 Appendix A Confidence Intervals for Means 25 LIST OF TABLES Table 1. Example of Sample Sizes 12 Table 2: Examples of Network Maintenance Changes that Trigger E911 Call Testing Pe
13、r ATIS-050006 High Level Requirements for End-to-End Functional Testing 19 Table 3: Examples of Network Maintenance Changes that Trigger E911 Simple Verification Testing (SVT) or Random Call Testing . 20 Table 4: Examples of Network Maintenance Changes That Do Not Require E911 Call Testing . 21 Tabl
14、e 5: SVT Testing Procedure. 22 Table 6: When is the SVT the Appropriate Test?. 23 LIST OF FIGURES Figure 1: System Under Test, ANSI Network. 7 Figure 2: System Under Test, GSM Networks 7 ATIS-0500010 11 INTRODUCTION The United States Federal Communications Commission has established accuracy require
15、ments for network and handset based location solutions for Enhanced 9-1-1 emergency call services. These requirements can be found in the Commissions Third Report and Order, adopted September 15, 1999. The Emergency Services Interconnection Forum (ESIF) identified the need for industry accepted meth
16、odologies for testing the accuracy performance and end-to-end functionality of Wireless E9-1-1 Phase 2 systems, both for initial assessment testing as well as ongoing maintenance testing. Requirements for accuracy testing methodologies were addressed in ATIS Standard ATIS-0500001, released in May of
17、 2004, and end-to-end testing requirements were addressed in ATIS Standard ATIS-0500009, released in June 2006. This document is divided into two sections; Part 1 addresses maintenance testing for accuracy, and Part 2 addresses maintenance of end-to-end functionality testing for Phase 1 and/or Phase
18、 2 Wireless E9-1-1 systems. This document neither recommends nor imposes a specific test methodology, but rather provides a common frame of reference that individual stakeholders can use to ensure continued accuracy and functionality compliance of Phase 1 or Phase 2 E9-1-1 integrated networks throug
19、h the inevitable updates and changes that occur over time, and provides a set of minimum requirements for individual test methodologies. Every possible effort has been made to ensure that these requirements remain technology neutral. Per current ESIF Operating Guidelines, due process has been follow
20、ed in the creation of this document, and development has been open for participation within the bounds of ESIF. 1.1 Scope This document defines the requirements and testing procedures needed to perform maintenance testing for accuracy, as well as the requirements and testing procedures needed for ma
21、intenance testing of end-to-end functionality in wireless E9-1-1 Phase 1 or Phase 2 systems. 1.2 Acknowledgements Subcommittee G would like to thank its members for their active participation and contribution to the creation of this document. 1.3 References 1 Enhanced Wireless 911 Phase 2 J-STD-036-
22、B Telecommunications Industry Association and Alliance for Telecommunications Industry Solutions June 2006 2 High Level Requirements for Accuracy Testing Methodologies ATIS Standard ATIS-0500001 Alliance for Telecommunications Industry Solutions May 2004 ATIS-0500010 23 High Level Requirements for E
23、nd-to-End Functional Testing ATIS Standard ATIS-0500009 Alliance for Telecommunications Industry Solutions June 2006 4 Emergency Services Interconnection Forum (ESIF) Wireless E911 Phase II Readiness Package (ATIS/ESIF 001) Alliance for Telecommunications Industry Solutions January 29, 2003 5 Revisi
24、on of the Commissions Rules To Ensure Compatibility with Enhanced 911 Emergency Calling Systems, CC Docket No. 94-102, Order, (City of Richardson Order). See also 47 C.F.R. 20.18(j). Federal Communications Commission, USA Oct. 10, 2001 6 Revision of the Commissions Rules to Ensure Compatibility with
25、 911 Emergency Calling Systems, CC Docket No. 94-102, Order on Reconsideration, FCC 02-318 (City of Richardson Reconsideration Order). Federal Communications Commission, USA Nov. 26, 2002 7 Revision of the Commissions Rules to Ensure Compatibility with 911 Emergency Calling Systems, CC Docket No. 94
26、-102, Memorandum Opinion and Order, FCC 03-262 Federal Communications Commission, USA Nov. 3, 2003 8 Near Term Issues for Emergency / E9-1-1 Services, Final Report Network Reliability and Interoperability Council VII (NRIC VII), Focus Group 1A December 2005 9 Emergency Services Interconnection Forum
27、 (ESIF) Issue 38 - Uniform Approach for Deriving Class of Service Oct. 20, 2006 2 DEFINITIONS This section offers a few definitions found to be important to maintain a common vocabulary throughout the creation of this document Accuracy Testing Accuracy testing, whether through empirical and/or predi
28、ctive test methods, consists of generating location data to gauge the accuracy performance of the system. Location data, typically significant in volume, involves the location infrastructure of the carriers network. The primary objective is to verify location accuracy and correct any location system
29、 errors. Limiting the test to the carriers location network minimizes impact to the rest of the Phase II network and maximizes the capability of the carriers to optimize their system. Event-Driven Accuracy Maintenance Trigger: Any accuracy maintenance trigger arising from an incident within a test a
30、rea or sub-area that might significantly alter the validity of pre-recorded empirical test data for the affected area. ATIS-0500010 3Functionality Testing (End to End) Functionality testing consists of testing the delivery of the location data from the carrier to the PSAP. The objective of this test
31、ing activity is to ensure interoperability between the carrier and the Emergency Service Network. This testing activity requires tight coordination among the involved parties, which normally includes the Emergency Service Network, the carrier and the technology vendors. Initial Deployment (Phase 1 a
32、nd Phase 2) Deployment of Phase 1 or Phase 2 wireless location capability by a Wireless Service Provider (WSP) that has not previously deployed in a PSAP jurisdiction within the 911- Entitys region. International Roamer A subscriber handset that has local service outside of the United States. Jurisd
33、ictional PSAP The PSAP having primary jurisdictional responsibility for a given location. KPI Key Performance Indicator. KPIs consist of relevant location network metrics and ratios which can be helpful in detecting performance problems. Specific KPIs are typically technology and implementation depe
34、ndent. In the context of functionality end-to-end maintenance testing, examples of KPIs might include monitoring Phase II yield, monitoring the mix of Phase II location types or the count of location requests by PSAP. Maintenance Testing Maintenance testing may be conducted after a system has been t
35、urned up with the Emergency Service Network. Like all network systems, maintenance testing will be conducted as needed to ensure functionality and performance. This testing activity may include functionality and/or accuracy testing and the participation of the Emergency Service Network may or may no
36、t be required. Maintenance testing can be a condensed version of the original accuracy and functionality testing. pANI The pseudo automatic number assigned to identify and an emergency call. Includes both ESRD and ESRK. Phase 1 Response A response shall be classified as Phase 1 when the location del
37、ivered is based on the cell sector from which the call originated. The displayed Class of Service may be either WRLS (no Phase 2 location capability exists), or WPH1 (Phase 2 capability exists but the specific caller location could not be determined). Cell sites with a sufficiently limited coverage
38、radius may also be considered Phase 2. Note: Class of Service displays may vary among 9-1-1 system service providers and are not controlled directly by the wireless carrier. ATIS-0500010 4Phase 2 Response A response shall be classified as Phase 2, with a WPH2 Class of Service, when the MPC/GMLC prov
39、ides the estimated location of the caller. The Position Source value delivered to the ESME indicates a positioning method capable of determining the callers location was used, regardless of accuracy. Cell sites with a sufficiently limited coverage radius may also be considered Phase 2. Predictive Te
40、sting A predictive test method consists of utilizing a predictive model to compute the expected accuracy of a location determining technology within a wireless carriers service area. The predictive model takes into account the physical elements of the location determining system for network or hands
41、et based solutions as well as the relevant terrain and RF propagation characteristics. PSAP Authority The entity having responsibility for PSAP operations and function. This may be a local, regional or state level organization. Roamer A subscriber handset that has local service outside the WSPs loca
42、l serving area, but within the United States. Routing PSAP Wireless network coverage areas and position determining systems do not typically align exactly with PSAP jurisdictional areas. The Routing PSAP is the PSAP to which a call from a given location is routed based on wireless system coverage fa
43、ctors and position determination capabilities, and may or may not be the same as the jurisdictional PSAP. Timing and Latency Timing refers to measuring the relative time of occurrence of events in the system relative to a common time reference. Latency refers to the elapsed time between the start of
44、 an event (or a specific sequence of events) in one part of the system and its (or their) conclusion in the same or another part of the system. Uninitialized Handset A handset that is not subscribed to any wireless service. 2.1 Acronyms AFLT Advanced Forward Link Trilateration AGPS Assisted GPS ALI
45、Automatic Location Identification ANSI American National Standards Institute AOA Angle of Arrival E-911 Enhanced 911 Emergency Service ESRD Emergency Services Routing Digits ESRK Emergency Services Routing Key ESME Emergency Services Messaging Entity ESNE Emergency Services Network Entity ATIS-05000
46、10 5GMLC Gateway Mobile Location Center GPS Global Positioning System GSM Global System for Mobile Communications MPC Mobile Positioning Center MSC Mobile Switching Center pANI Pseudo Automatic Number Identification PDE Position Determining Equipment PSAP Public Safety Answering Point RF Radio Frequ
47、ency S/R Selective Router SMLC Serving Mobile Location Center TDOA Time Difference of Arrival U-TDOA Uplink TDOA WAAS GPS Wide Area Augmentation System GPS ATIS-0500010 6Part 1 Maintenance Testing for Accuracy 3 GENERAL TEST REQUIREMENTS 3.1 Introduction Once a location system has been tested and fo
48、und to be compliant1,2the wireless carrier shall ensure that its network maintains compliance through the accuracy maintenance test requirements established in this document. OET-71 encouraged the development of accuracy test methods that were efficient, reliable, simple, and cost-effective. These a
49、ttributes are especially significant given the on-going nature of maintenance testing. It has been recognized by both ESIF and NRIC that maintenance testing can be a condensed version of the original accuracy and functionality testing. ESIFs goal in developing these requirements is to define efficient, simple and cost-effective maintenance test methods, which are still reliable and technically sound. With this in mind, it is further recognized that maintenance testing efforts will typically not be full-fledged OET-71/ATIS-05000
