By Cleophas KiioDirector, ICT.ppt

1、By Cleophas Kiio Director, ICT,15-sep-10,1,The Best Practices in Census Data Processing Operation: Case of 2009 Census:,Overview,Data processing Activities Review Planning for Data processing Setting the Data processing site Implementation Data capture Analysis Dissemination Archival,15-sep-10,2,Dat

2、a Processing Activities Review,DP follows the completion of field data collection and entails the following: Capture Cleaning/Editing Tabulation Analysis Dissemination Archival,15-sep-10,3,Planning for Data Processing (DP),15-sep-10,4,Identification of Methodology/technology: Keying From Paper (KFP)

3、 - Manual Data Entry largely used in KNBS for small Surveys Keying From Image (KFI) -scanning Optical Mark Reading (OMR)- scanning Optical/Intelligent Character Recognition (OCR/ICR) - scanning Online data capture use of pc Use of mobile devices (PDA) For the 2009 Census, KNBS chose scanning technol

4、ogy with OCR/ICR having used the same in the 1999 Census. A study tour the US Census Bureau was conducted to understudy the best practices. Major considerations were the budget and availability of technical knowhow.,Planning for Data Processing (DP) contd,Selection of Tools and Equipment: Computers

5、acquired 125 high capacity computers with duo screens. Servers- 3 high-end servers did the census (32 GB memory, multiple processors, 1 Terabyte secondary storage each) Storage 3 high capacity Storage Area Networks (SANs) were procured initially 5 Terabytes (TB) each but later upgraded to 14 TB each

6、. Software- Capture software - with the challenges faced the 1999 census where the bureau used the AFPS pro from Top Image Systems (TIS), the Bureau chose to use the iCADE system ( integrated Computer Assisted Data Entry System) developed by the US Census Bureau. Cleaning/tabulation- Cspro (Census a

7、nd Surveys Processing software) Scanners- 3 new Kodak 1860 high volume scanners were acquired in addition to the 2 existing Kodak 1900 scanners used during the 1999 Census. Capable of scanning over 200 ppm.Network infrastructure- all computers, scanners, servers and SAN were connected in a wide area

8、 network (WAN),15-sep-10,5,Planning for Data Processing (DP) contd,Design of Questionnaires As standard practice questionnaires are developed and designed with technology to be used in mind.The 2009 Census questionnaires were designed by highly trained Bureau staff.Technical support was offered by t

9、he US Census Bureau Precision in design was critical for compatibility with the iCADE system.,15-sep-10,6,Setting Up the DP Site,Planning the layout (library, KFI, OCR/Manual registration, server room, editing ) Installing the computer network Installing the power supply system and provisioning for

10、power backup system: UPS and generators Installing the furniture, lifts and Air-conditioning Procuring high bandwidth internet. A ware house for storage Recruitment of staff,15-sep-10,7,15-sep-10,8,Installation Systems and testing was completed after census enumeration Integrated Computer Aided Data

11、 Entry (iCADE) system training In 2009 we had approximately 12 million A3 questionnaires. Engaged close to 500 personnel for the processing. Processing took less than a one (1) year to complete,Implementation,Tracking of questionnaires done with a custom made tracking system with inbuilt geocode lis

12、t to ascertain completeness and flow control Guillotining- trimming/cutting off the spirals iCADE system processes Batching- registering books from each EA in the iCADE Scanning Auto and Manual registration Exception review OCR review Key From Image (KFI),15-sep-10,9,a) 2009 Data Capture Processes,1

13、5-sep-10,10,Capture Output,Captured data was output to a text file then auto-formated as input to the CSPro softwareOCR characters read: 2,485,008,272 with an accuracy rate of 99.86% (0.14% error) KFI characters keyed: 228,771,647 with a 99.94 accuracy rate (0.055%error)This means the OCR read over

14、90% of the characters with a very high accuracy rate (OCR review definitely helped get this accuracy rate but customization algorithms had to be added to the quality).22,326,373 images from the census questionnaires 273,201 books in 144,098 batches10,602 batches went to exception review and 133,496

15、batches bypassed Exception Review altogether and went straight into OCR.,15-sep-10,11,15-sep-10,12,b) 2009 Data Analysis KNBS used CSPro a freeware from the US Census Bureau. This process required: Subject matter specialists provide editing rules Programmers implement editing rules through programs

16、The team developed the editing program with which data is cleaned.,Editing/cleaning and Imputation,15-sep-10,13,Systematic inspection of invalid and inconsistent responses, and subsequent manual or automatic correction according to predetermined rules (edit specs).Imputation is the procedure of assi

17、gning values to missing, invalid, or inconsistent data using a set of predefined criteria embedded to an editing program.,Why Edit and Impute?,Clean up data to facilitate analysis Identify types and sources of error Improve quality of census dataErrors must be detected and their causes identified Ap

18、propriate corrective measures are taken to improve the overall data quality.,15-sep-10,14,15-sep-10,15,c) Data Tabulation Process of producing data outputs (tables, frequencies, cross-tabulations,) Requires subject matter specialists to prepare dummy output layouts supported programmers Data in then

19、 presented in this tabular layouts.,15-sep-10,16,15-sep-10,17,d) Data Dissemination Providing public with information through census books, fliers, CDs, DVDs, online databases (Census info, IMIS, sms service)e) Data Archival Documentation for permanent storage for further and future analysis,15-sep-

20、10,18,15-sep-10,19,Ware-house was located about 10Km from processing centre Inadequate processing space Printing was not perfect this affected the OCR Limited number and constant breakdown of the KNBS dedicated lift slowed down processing. Power outages posed a major challenges Being a new system, t

21、here was a cautious and slow acceptance of the system.,Challenges,15-sep-10,20,Comprehensive DP plan be developed with clearly defined objectives: Efficiency and effectiveness to process in the shortest time possible. Control cost of processing to avoid budget overruns. Quality data output Carry out

22、 risk analysis beforehand to identify potential pitfalls and put in place mitigation measures.,Best practices: Lessons learnt,15-sep-10,21,Best practices: Lessons learnt contd,Cartographic mapping be completed 1 year before census geographical codes and related documentation (geo-codes) to be ready

23、6 months before enumeration. Timely acquisition of census tools and equipment DP site be ready 6 months before enumeration date for test runs . Technical and maintenance support measures must be instituted and enforced.,15-sep-10,22,Questionnaires and manuals be ready 5 months before census date to

24、allow for logistics and pretesting. Total quality control at the printing press must be ensured for precision printing. Recruitment and training of staff be done before the census date. DP site be located in close proximity to the questionnaire warehouse,Best practices: Lessons learnt contd,15-sep-1

25、0,23,Conclusion,Despite the challenges, it was possible to complete DP in less than a year after census. However better planning and organization of the exercise it possible to complete the exercise within 6 months after enumeration. The lessons learnt may form the recommendations that if adopted the above can be attained.,15-sep-10,24,15-sep-10,25,Thank You!,