ATIS 0500001-2011 High Level Requirements for Accuracy Testing Methodologies.pdf

1、 ATIS-0500001 ATIS Standard on - HIGH LEVEL REQUIREMENTS FOR ACCURACY TESTING METHODOLOGIES ATIS is the leading technical planning and standards development organization committed to the rapid development of global, market-driven standards for the information, entertainment and communications indust

2、ry. More than 200 companies actively formulate standards in ATIS Committees, covering issues including: IPTV, Cloud Services, Energy Efficiency, IP-Based and Wireless Technologies, Quality of Service, Billing and Operational Support, Emergency Services, Architectural Platforms and Emerging Networks.

3、 In addition, numerous Incubators, Focus and Exploratory Groups address evolving industry priorities including Smart Grid, Machine-to-Machine, Connected Vehicle, IP Downloadable Security, Policy Management and Network Optimization. ATIS is the North American Organizational Partner for the 3rd Genera

4、tion Partnership Project (3GPP), a member and major U.S. contributor to the International Telecommunication Union (ITU) Radio and Telecommunications Sectors, and a member of the Inter-American Telecommunication Commission (CITEL). ATIS is accredited by the American National Standards Institute (ANSI

5、). For more information, please visit .Notice of Disclaimer i.e., WGS-84 or more recent. Ground truth accuracy shall be as per the requirements in Section 7, Empirical Test Methods (Paragraph 7.4, Static/Dynamic Testing). Vertical dimensions may be included, but are not required. 9.4 Data Analysis T

6、ools and Software Data analysis tools including Data Recording Software, Data Processing Software described in Section 0, 4 Equipment Requirements (Paragraph 5, Software Requirements) may be used to automate the analysis, enhance the efficiency and increase the reliability of calculations of locatio

7、n error. Data tools shall be adequately described and documented as part of any accuracy test plan. 9.5 Processing of Test Call Data During the testing process, all calls shall be documented and classified according to their results. Calls shall be categorized and results documented in accordance wi

8、th the OET 71 guidelines. 9.5.3 Failed or Dropped Calls Any failure to complete a test call or any dropped test calls shall be documented as part of the data summary. Such incidents shall be documented, but not be included as part of the accuracy statistics and their associated results. 9.5.4 System

9、atic Errors Any systematic errors that are determined as a result of data analysis shall be reported as part of the summary. The processing of outliers (an instance of large errors or locations where no Phase 2 fixes are obtained) shall be handled consistent with the OET Bulletin 71 Guidelines and i

10、ncluded in the data analysis and processing. 9.5.5 Weighting of Data Weighting of data is a method to take into consideration such factors as the likelihood that a wireless 9-1-1 call (or any wireless call) will be made from a particular location. OET 71 provides a general discussion of call weighti

11、ng. ATIS-0500001 15 As a goal any weighting should be conducted as part of test planning and test point selection so as to minimize any subjective, post data collection filtering. If used, weighting criteria shall be established during test planning, and may either be applied in the test site select

12、ion, or in post test analysis, but not both. Data shall be weighted only in accordance with a justifiable, verifiable and statistically valid methodology. No arbitrary portion of the data collected shall be removed during data analysis. Examples of Weighting of Data Examples of weighting of data inc

13、lude, but are not limited to the following (other examples may be acceptable to the FCC): 1. One weighting method is to first gather accuracy test measurements essentially uniformly and randomly over the entire test area. Next, weight each of those test measurements (i.e., the measured positioning e

14、rror) by a ratio of the number of wireless 9-1-1 calls placed via the cell site covering the location of each test call, relative to the total number of wireless 9-1-1 calls placed in the entire test area, over a period of time. The number of wireless 9-1-1 calls placed in the test area should be me

15、asured over a significantly long interval of time, for example 1-3 months. (The intent is that the period be long enough to ensure capturing adequate 9-1-1 statistics yet short enough so that substantial changes to the network will not have occurred.) The weighting ratio for cell sites which receive

16、d no wireless 9-1-1 calls during this measurement time period could be established as the average ratio of wireless 9-1-1 calls made per cell site over the entire test area during that time, or some other suitably small, yet non-zero ratio. This step would ensure that no test results collected are c

17、ompletely eliminated from the subsequent statistical computations due to 9-1-1 call weighting. This redistributed (9-1-1 call-weighted) data is then used in the subsequent statistical computations for the test area. 2. Final test area accuracy performance is determined by weighting accuracy performa

18、nce according to (1) the percentage of actual wireless 9-1-1 calls in a given sector relative to the test area, or, if wireless 9-1-1 call data is not available, (2) the percentage of actual total wireless calls originating in a given sector relative to the test area. 9.5.6 Pass Fail Criteria The pa

19、ss - fail criteria for accuracy testing of the positioning technology deployed for Wireless E9-1-1 shall be developed and documented as part of the test plan being implemented. While the criteria may vary according to the objectives and requirements of the test being performed, they shall be applied

20、 in accordance with the methodology and practices outlined in this document. 9.5.7 Resulting Statistics Sufficient amounts of data shall be collected, analyzed and reported so that the applicable error percentiles are calculated with at least 90% confidence. 9.6 Data Summaries and Reports This secti

21、on includes a list of what ESIF considers to be a reasonable set of accuracy test-related data to be collected, organized, and stored by the individual company or organization responsible for the testing. While this collection of data is deemed essential to sound engineering practice for ATIS-050000

22、1 16 accuracy testing, reporting of test data shall be based upon mutual agreement between the requesting company or organization and the company or organization performing the test. Note that reporting to the FCC is addressed elsewhere (outside the scope of this document) and as such is not the sub

23、ject of this section. Data summaries/reports including all statistics, pre-processed and post processed data shall be stored on a standard commercial media in accordance with the established guidelines of the individual company or organization responsible for the testing. Data summaries and reports

24、shall include as a minimum: 1. Description of the testing objectives. 2. Description of the location technology and air interface tested. 3. Description of the test configuration used (Including the test versions of each location network element and handsets used e.g., GMLC Version XYZ). 4. Descript

25、ion of test area(s) used, including a graphical representation. 5. Description of test point locations, test route selection method, test routes used, and test route identification. 6. Description of “ground truth” or reference locations used. 7. Description of any data recording equipment/software

26、and data analysis equipment/software used. 8. Description of any predictive modeling used to support the location testing. Description of the available baseline used as a basis for the predictive modeling and the applicability of the baseline to the test area for which this predictive modeling has b

27、een applied. 9. Description of the “Pass-Fail” criteria. 10. Description of any systematic errors that occurred during testing, if applicable. 11. Description of any failed or dropped calls, if applicable. 12. Description of any weighting used and the statistical justification. 13. Location error st

28、atistics for the applicable error percentiles of the total samples collected and processed and the level of statistical confidence with which these percentiles have been determined. Other statistics may be presented based upon the test plan objectives. 14. Description of any remaining problems and p

29、lan for resolution (e.g., re-test plan). ATIS-0500001 17 10 ANNEX A: On Confidence Intervals and Levels for Location Testing Confidence intervals and confidence levels are interrelated elements in the estimation of certain statistics of a sample. To illustrate the terminology, if we are estimating t

30、he mean positioning error, the confidence interval is the interval around the sample average in which we expect this mean to fall with a certain probability, e.g., 0.9, called the confidence level. The approach discussed here falls under the so-called distribution-free confidence intervals, i.e., in

31、tervals that do not depend on a priori knowledge of the distribution of the variable being estimated, which in our case is the location or positioning error. This is a robust approach that does not entail significant assumptions about the error, and primarily depends on a sample that is large enough

32、. At a given test location when an adequate number of independent calls is placed, then although the error for each individual call is not normally distributed, the average error for the sample quickly approaches a normal distribution. The number of calls at the test location does not have to be qui

33、te large for this Central Limit Theorem application to hold. As long as the location system is not providing totally inconsistent results from call to call resulting in a very large variance, which is a very reasonable assumption for calls placed from the same location, then a sample “N“ of 20 or mo

34、re calls (but sometimes less) would be adequate. We can then apply the simple rules of the normal distribution to determine the confidence interval and level associated with the measurement as follows. N is the sample size X is the location error (a random variable) Xav is the sample average for the

35、 error observed is the mean error at the test location is the standard deviation of the error Thus, the 90% confidence interval associated with the measurement or estimation of the mean error at the test location is: (Xav , Xav+ ) Where = 1.645 / (N). The number 1.645 corresponds to the probability

36、of 0.9 from the Normal curve. Another way of stating the above is that the probability is 0.9 (90%) that the mean location error is within the confidence interval (Xav , Xav+ ) For a 95% confidence interval, i.e., a confidence interval with a 95% confidence level, the number 1.645 would be replaced

37、by 1.96, and similarly for other confidence levels. A convenient way to choose or express is as a percentage of the observed average error. Example: If 20 calls are placed at a single test location and the observed Xavand are 50 m and 25 m, respectively. Then the 90% confidence interval in measuring

38、 the mean error is 50 m +/- 9.2 m or 50 m +/- 18.4%. ATIS-0500001 18 11 ANNEX B: Example of using the 90% confidence interval to obtain a statistically significant sample size. The following example demonstrates a technique to determine the number of samples needed to demonstrate accuracy compliance

39、 with a 90% confidence interval in a contiguous deployment test area. This technique requires a prior understanding of the underlying radial position error behavior specific to the location technology being tested. It also requires the actual test area deployment to behave predictably according to t

40、his distribution. This specific example is applicable to a Network Based U-TDOA location system. Through a significant level of field-testing, it has been determined that the statistics of the radial position error of this U-TODA system closely match that of a Rayleigh distributed random variable. A

41、ssuming a Raleigh distribution for the error statistics, a 90 percent confidence interval can be determined for both the 67th and 95th percentile accuracies as a function of the number of test calls. This technique can be used to determine the number of test calls required to achieve a given measure

42、ment accuracy. It should be noted, however, that once a Rayleigh distribution is assumed, a selection of a specific number for the 67th percentile implies the associated number for the 95th percentile. So if the 67th percentile is taken to be 100 m then this implies that the 95th percentile is 164 m

43、. Similarly, if the 67th percentile is taken to be 50 m, the 95th percentile becomes 82 m. Thus, it is important for the user of this approach to ascertain that the distribution of errors observed in the field is consistent with this relationship between those percentiles. If it is not, then a more

44、general distribution needs to be fitted to the data and applied. If this more general approach is found to be unwieldy, then a distribution free approach, such as the one discussed in ANNEX A, can be used. In the table below the Rayleigh distribution of error is assumed. The table shows the 90% conf

45、idence interval for both the 67th and 95th percentile accuracies for 2 example systems with 67th percentile accuracy performance of 50m and 100m. ATIS-0500001 19 67% = 50 95% = 82 90% CI of 67% Error 90% CI of 95% Error N Low High Low High 100 46.1 54.4 75.8 89.4 500 48.1 51.8 79.1 85.1 1000 48.6 51

46、.2 79.9 84.2 2000 49.0 50.8 80.5 83.5 67% = 100 95% = 164 90% CI of 67% Error 90% CI of 95% Error N Low High Low High 100 92.2 108.8 151.6 178.8 500 96.2 103.6 158.2 170.3 1000 97.2 102.4 159.9 168.4 2000 98.0 101.6 161.0 167.1 Note that in the case of a Rayleigh distribution of error, for just 500 calls, the confidence intervals are less than +- 5%, showing that 500 calls provides a sufficiently tight 90% confidence interval.