REG NASA-LLIS-0976--2001 Lessons Learned User Feedback (Operational) Testing.pdf

1、Lessons Learned Entry: 0976Lesson Info:a71 Lesson Number: 0976a71 Lesson Date: 2001-07-26a71 Submitting Organization: HQa71 Submitted by: Wilson HarkinsSubject: User Feedback (Operational) Testing Description of Driving Event: In the 1995-1996 timeframe the Office of Safety and Mission Assurance (OS

2、MA) began development of a distance learning capability under the umbrella of the Professional Development Initiative (PDI). This distance learning capability eventually evolved from a Safety and Mission Assurance discipline system into the Site for Online Learning and Resources (SOLAR) (http:/solar

3、.msfc.nasa.gov), currently one of NASAs primary distance learning resources. The intent of the initial development effort was to design and implement a prototype system for the Safety and Mission Assurance discipline. Since web-based distance learning was an untried technology within NASA, the devel

4、opment team determined that before initial system release a User Feedback Test was required to exercise the system and identify problems from a user perspective. (Since this test was conducted under expected operating conditions using a portion of the expected user base it could also be considered a

5、n operational test).Not surprisingly the test results identified a series of changes that should be made to the system before production release. These included:a71 Adding acronym and test buttons to the system navigation aids frame.a71 Improving the legibility of embedded forms by using Portable Do

6、cument Format filesa71 Providing clear indications of what each hyperlink identified within the course work does (glossary vs. more course material vs. reference, etc.)While identification of changes to features and other improvements of that type were expected outcomes of the User Feedback Test we

7、were not prepared for one of the findings that conflicted with one of the primary assumptions wed made as we initiated design. We had assumed that even though the Internet and the world-wide-web were relatively new, most of the NASA workforce would have enough familiarity and capability to use the w

8、eb-based distance learning capability Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-without additional assistance. The test results indicated otherwise, we had a wide spread of web user capability from experienced to novice and if we wanted the web

9、-based distance learning capability to succeed on a wide basis within NASA additional support would be required. (As a note while most new users/potential users now have enough basic internet/web skills to navigate the SOLAR web-site we still experience some difficulties when users are required to p

10、erform non-routine web activities, for example downloading and integrating plug-ins required for system use. Therefore the introduction of new capabilities that require non-routine web activities needs to be well thought out and should include arrangements for support of users.)In response to the re

11、sults of the User Feedback Test an upgrade to the system was provided that included a more extensive tutorial “Web-Workshop“ to the user help features. This “Web-Workshop“ was oriented to the novice web user and included topics such as Browser Basics, Screen Resolution, Plug-ins and Helper Applicati

12、ons as well as an Explanation of System Features. Examination of the User Feedback Test results also was a factor in including hands-on system demonstrations as a part of the system roll out and marketing campaign.A secondary outcome from the test was the identification that the pace of individual u

13、sers taking the training varied much more than we had originally anticipated. Our original plan was to identify a “suggested“ time to complete each module based upon an “average“ user; however, after noting the wide variation in the time required to complete modules we decided to play-down the “sugg

14、ested“ times so as not to stigmatize slower self-paced learners.Lesson(s) Learned: Use of a User Feedback (or Operational) Test is effective in identification of system weaknesses as well as weaknesses in assumptions applied to the development process. In addition to identifying shortfalls in the sy

15、stem the User Feedback Test can also helps in determining the capability of the users to operate the system.Recommendation(s): Conduct a User Feedback (or Operational) Test prior to system release.Evidence of Recurrence Control Effectiveness: N/ADocuments Related to Lesson: N/AProvided by IHSNot for

16、 ResaleNo reproduction or networking permitted without license from IHS-,-,-Mission Directorate(s): N/AAdditional Key Phrase(s): a71 Information Technology/Systemsa71 Policy & Planninga71 Test & Verificationa71 Training EquipmentAdditional Info: Approval Info: a71 Approval Date: 2001-07-31a71 Approval Name: Eric Raynora71 Approval Organization: QSa71 Approval Phone Number: 202-358-4738Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-