ATIS 0500038-2018 Recommendations for Extensions to Indoor Test Methodology.pdf

1、ATIS-0500038 ATIS Standard on Recommendations for Extensions to Indoor Test Methodology Alliance for Telecommunications Industry Solutions Approved June 25, 2018 Abstract This document provides recommendations specific to horizontal accuracy testing within the framework of the 9-1-1 Location Technol

2、ogies Test Bed. It should be viewed as an extension to ATIS-0500031.v002, Test Bed and Monitoring Regions Definition and Methodology. ATIS-0500038 ii Foreword The Alliance for Telecommunications Industry Solutions (ATIS) serves the public through improved understanding between carriers, customers, a

3、nd manufacturers. The Emergency Service Interconnection Forum (ESIF) provides a form to facilitate the identification and resolution of technical and/or operational issues related to the interconnection of wireline, wireless, cable, satellites, Internet, and emergency services networks. The mandator

4、y requirements are designated by the word shall and recommendations by the word should. Where both a mandatory requirement and a recommendation are specified for the same criterion, the recommendation represents a goal currently identifiable as having distinct compatibility or performance advantages

5、. The word may denotes an optional capability that could augment the standard. The standard is fully functional without the incorporation of this optional capability. Suggestions for improvement of this document are welcome. They should be sent to the Alliance for Telecommunications Industry Solutio

6、ns, ESIF, 1200 G Street NW, Suite 500, Washington, DC 20005. At the time of consensus on this document, ESIF, which was responsible for its development, had the following leadership: R. Hixson, ESIF Chair (NENA) R. Marshall, ESIF 1stVice Chair (Comtech) J. Green, ESIF 2nd Vice Chair, ESIF ESM Co-Cha

7、ir (Sprint) K. Springer, ESIF ESM Co-Chair (AT for example, for Wi-Fi-beacon-based positioning, time of applicability is typically the time when the Wi-Fi scan occurs. Since time of applicability can occur at any point during the execution of the test, a way to produce ground truth at any arbitrary

8、time along the route is needed. The solution is to create tooling that allows a tester to walk predefined trajectories, as shown in Figure 6.3, capturing the times when the mobile devices depart from and arrive at predetermined, fixed points along the route. These predefined trajectories consist of

9、a series of line segments, separated by fixed, known positions. Ground truth for these fixed positions are first determined using the same methods described in Clause 6.2.6. Using the captured times of arrival and departure and the known positions of the route points, the tool can linearly interpola

10、te between the points to produce a truth position for any arbitrary time during the test, including when paused at one of the route points. Such a tool would need to be implemented on a mobile device, such as a smart phone or tablet, so that the tester can hold the device in his/her hand while walki

11、ng a predefined route, clicking a button as he/she arrives at, or departs from, each point along the route. Requirements for this tooling are described in Clause 7.1.2. Figure 6.3 Test in Pedestrian Motion Along Fixed Routes 6.3.1 Route Length, Duration, however, this improved accuracy needs to be a

12、ssessed in the context of overall cost and complexity of execution. It is recommended that development of a mobile platform-based tool, specifically the “Test Point Time Capture & Route Management Tool Feature for Approach #1” described in Clause 7.1.1, be undertaken under the direction of the 911 L

13、ocation Technologies Test Bed. This evaluation exercise should be leveraged for: Providing feedback to enhance the testing process including tool functionality, and provide feedback based on real world testing experience. In-depth comparative analysis of horizontal error and cdf plot analysis, on a

14、per route and a per building basis. Temporal analysis of the results produced from a set of test points or a route versus obtained from one test point along the path. Analysis results from the phased “Approach #1 Methodology” introduction should be used to help guide future test campaigns. ATIS-0500

15、038 22 Annex A (informative) A Simple Referential Ground Truth Calculation Methods Simple flat earth referential ground truth position calculation methods can be used to populate one or more high-quality ground truth points to nearby points. The term flat earth refers to assuming that the surface of

16、 the earth approximates a flat surface over short distances, so that very simple geometric and trigonometric calculations can be applied. Figure A.1 illustrates a simple approach for determining position coordinates for any unknown point between two known positions, using linear interpolation. Note

17、that for points on the same floor, the Height Above Ellipsoid can typically be assumed to be the same. Figure A.1 Determining Unknown Position Using Two Known Positions and Relative Distances Figure A.2 illustrates a simple approach for determining position coordinates for any unknown point based on

18、 the position of a single known point and the bearing and distance from the known point to the unknown point. As before, for points on the same floor, the Height Above Ellipsoid can typically be assumed to be the same. ATIS-0500038 23 Figure A.2 Determining Unknown Position Using Known Position, Kno

19、wn Angle, and Distance ATIS-0500038 24 Annex B (informative) B Confidence Intervals for Proportions in the Context of Test Calls Placing a sample of wireless location test calls in the environment of a given building can be considered as a probabilistic experiment of repeated trials (Bernoulli Trial

20、s), where one of two outcomes of each trial is of interest, either the positioning error of the test call is within 50m, or it is not. We are interested in the proportion p of the sample of n calls for which the positioning error is within 50 m. We can estimate this proportion with a certain level o

21、f confidence and a certain level of “precision”, which is more commonly called the confidence interval. Note that if we do not know much about p, then p= 0.5 provides the most conservative analysis for sample size. This basically means that any given test call is equally likely to have a positioning

22、 error either less or more than 50 m. For a sufficiently large sample of independent test calls n (at least 20) and when np and n(1-p) are not too small, Normal approximations for the estimation error can be used and the estimate for the proportion p is given by p +/- z Sqrt p (1-p) / n where z is a

23、 number derived from the area under the unit Normal density function and corresponds to the desired level of confidence in the estimate. For 90% confidence z = 1.645 and for 95% confidence z = 1.96. If it is acceptable to estimate the proportion p to within, for example, ten percent, then we can rea

24、dily solve for n for different nominal values of p, including a most conservative case with p =0.5 and a less conservative case of p = 0.8. This latter case is still quite practical in the context of indoor wireless location testing. Whereas in a large urban or dense urban building a 50/50 chance of

25、 the positioning error being within 50 m is a good starting assumption, in a smaller, less challenging suburban building, the likely probability of the positioning error being within 50 m is much closer to 80%. This will result in a somewhat smaller minimum sample size. The most conservative sample

26、size corresponds to p = 0.5. For estimating p within 10% the minimum sample size is 68 test calls. For p = 0.8 the minimum sample is 44 calls. The Normal approximation conditions commonly break for higher values of p (closer to one) or conversely for very small values of p close to zero, since np or

27、 n(1-p) becomes a small number, e.g., less than five. Hence smaller samples would not be reliable. It is also critical that the test calls be independent. ATIS-0500038 25 Annex C (informative) C Summary of Experiments Used to Estimate Error Sources for Pedestrian Motion Ground Truth In 2014, experim

28、ents were performed to establish the viability of using simple linear interpolation between a series of points with known position to determine ground truth while in pedestrian motion between the points. As part of this assessment, the error sources inherent in this method were identified, and using

29、 simple experiments, each error source value was approximated. This Annex summarizes how these experiments were performed, defines the error sources, provides the approximate error source value calculated, and describes how each error source value was approximated. This experimentation consisted of

30、two phases: 1. Two outdoor routes in open sky conditions were constructed. These routes were then walked while generating interpolation-based truth using a prototype interpolation-based ground truth handset-based tool. A high-quality GNSS device, in some cases with an inertial measurement unit, was

31、also positioned with the tester while repeatedly walking the routes. The results from the interpolation-based ground truth tool were compared against the GNSS results to approximate error sources. 2. Routes were then constructed in numerous public indoor venues, and a tester was carefully observed w

32、hile walking routes to witness how real-world conditions affected the ground truth error produced. From these observations, error values were confirmed, and the relationships between real-world conditions and error were better understood, leading to the definition of several usage guidelines. Note t

33、hat this error assessment focused on the error sources inherent to the execution of this pedestrian test methodology, specifically Etime_capture, Enonlinearity, and Epath_deviation, as defined below. Inaccuracies caused by ground truth errors of the points used (pre-defined) to form the route were n

34、ot assessed in this experiment, as this error source is driven by entirely different processes. Likewise, the inaccuracy caused by device time of applicability problems was not assessed. Also note that this was a relatively simple series of experiments, thus it is not possible to confidently measure

35、 an error source below a resolution of about one meter. This was deemed sufficient to determine an approximate baseline. Table C.1 lists each error source assessed, defines the error source, gives the approximate error from these experiments, summarizes the method used to derive the error approximat

36、ion, and provides additional observations. ATIS-0500038 26 Table C. 1 Pedestrian Motion Error Sources Error Source Error Source Description Error Methodology / Observations Etime_capture This error source quantifies inaccuracies introduced when the tester captures the time of arrival at or departure

37、 from a point slightly before or slightly after the true time of arrival or departure. This error source focuses on the users inaccuracies in pressing the button on the mobile platform-based tool, not on timing errors within the tool itself, which are presumed to be minimal. This human-caused timing

38、 error translates in to position error of the ground truth produced, as a function of pedestrian speed, and the geometry of the points. One meter or less The tester used a tool running on a mobile device to capture these times by pressing a button. The accuracy of time within the mobile device is kn

39、own to be within milliseconds of true time, as the device was slaved to network time, and thus negligible. Accurate GNSS-based and interpolation-based positions were compared at the time of arrival at and departure from the points, and found to be quite close. Also, while walking public routes, the

40、prototype handset-based tool produced an audio beep when the tester departed from and arrived at each point. The tester was visually observed to arrive and depart at the same time as the beep easily within a half a second, which translates to less than 0.7 meters at typical walking speeds, confirmin

41、g the numerical calculations. Enonlinearity This error source quantifies axial inaccuracies introduced when the tester moves slightly faster during one portion of a line segment, and slightly slower during another portion of the same line segment. Since Approach #2 relies on linear interpolation to

42、produce ground truth, changes or non-linearity in speed along the axis of travel translates into position error, as a function of the amount of non-linearity, the geometry of the beginning and ending linear segment points, and the positions within the line segment where the non-linearity occurred. T

43、ypically no more than about 1.25 meters An error vector between accurate GNSS and interpolation-based positions were compared at GNSS time of applicability, and the axial component was measured to determine typical worst-case error. Investigation into this error source suggests that tester speed whi

44、le walking is surprisingly consistent, for route legs under 100 meters. This was in conditions where there was no significant pedestrian traffic in the route area. One caveat is that in crowded conditions along the route of travel, this error source can increase. Thus, it is desirable to pick data c

45、ollection times when the venue is not crowded to minimize error. Also, it is important to account for known obstacles by placing a point before the obstacle and using a pause capability in the tool when needed. Note that it is also desirable to avoid very large route leg distances. This is easily do

46、ne by adding additional points to breakup long, straight legs. Epath_deviation This error source quantifies transversal inaccuracies introduced when the tester deviates from the linear path. Unlike Enonlinearity, this error source focuses on the transversal error caused when the tester has to travel

47、 slightly to the left or right off the true direct path, typically to avoid an obstruction. One meter or less Error vectors between accurate GNSS and interpolation-based positions were compared at GNSS time of applicability, and the transversal component was measured to determine typical worst-case

48、error. Also, testers were observed while walking real-world line segments, and the deviation needed to avoid obstacles was visually assessed. This error source was found to be minimal under one meter as long as the venue was not crowded. Picking data collection times when the venue is not crowded is needed to minimize this error source.