1、Public Lessons Learned Entry: 6656 Lesson Info: Lesson Number: 6656 Lesson Date: 2012-08-7 Submitting Organization: JPL Submitted by: David Oberhettinger Subject: Does Software IV spacecraft bus supplied by contractor 10 instruments with software Contractor FSW Manager preferred direct interaction a
2、nd integration of the IV spacecraft bus supplied by contractor 1 instrument PSSE handled most interactions with IV 400 hours - systems team 0.5 - 1.0 FTE (10,500 hours) Table 1. NASA IV&V staffing and JPL project support staffing for recent JPL projects as of Phase E, from Reference (2), p. 9 (MSL I
3、V&V services are continuing, so the MSL values may change.) .Key:. Communicated - Includes IV&V issues in States: Project Accepts Risk, Not To Be Verified, Closed, To Be Verified, Withdrawn Accepted - Includes IV&V issues in States: Project Accepts Risk, Not To Be Verified, Closed, To Be Verified Re
4、solved - Includes IV&V issues in States: Project Accepts Risk, Not To Be Verified, Closed Quality of Results - Ratio of Accepted to Communicated Recent JPL experience with IV&V support for the Mars Science Laboratory (MSL) project suggests that IV&V can bring about explicit FSW improvements. IV&V su
5、pport to MSL was limited to Entry, Descent, and Landing (EDL) software, fault protection (FP) software, and surface mobility software. Specific examples of IV&V contributions to the MSL project (Reference (2) include: Analysis of flow-down from the functional description documents (FDDs) to MSL FSW
6、requirements, and then to testing: o Project processed engineering change requests (ECRs) to bring documentation/code into alignment. o In most cases, the code was found to be correct, so IV&V inquired whether the wrong requirement had been tested. o IV&V found cases where a deleted requirement was
7、still supported in the code. Requirements-to-code tracing: o IV&V discovered cases where the MSL code was implemented incorrectly. o Fatal Event Verification Records (EVRs) were identified and traced to requirements per the project policy. Code (static) analysis: o Spot-checked the interim work prod
8、ucts of the MSL FSW development team using different tools. o IV&V found uninitialized variables, unused variables, buffer overruns, and violations of coding standards. o (This code analysis is duplicative of the current JPL FSW development process). The IV&V team developed a database that captured
9、the MSL FP design and facilitated checking for consistency between the FP design and the FP documentation (including cross-checks for FP requirements, FP FDD, monitors, and responses) o IV&V found many inconsistencies and errors where the same stimulus would map to a different system FP response. o
10、The project found the IV&V teams database very useful, and they may adopt it as an operations tool. Table 2 lists the 512 MSL IV&V Team findings on MSL fault protection (as of December 2011) across the range of mission phases/domains. Phase/ Domain Functional Description Document (FDD) Total Local O
11、nly System + Local System Only Other Cross- Cutting Power & Power/Analog Module (PAM) 7 2 1 4 Pyro 1 1 Sequencing 10 2 5 2 1 Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Telecom 8 3 4 1 Thermal 135 130 1 3 1 Time Definition & Checks 10 7 2 1 Uplin
12、k and Command 3 2 EDL EDL Actuators 35 35 EDL Comm 1 1 EDL Sensors 11 11 Dedicated Fault Protection SW 1553 Bus Off-Nominal 6 2 4 Intercomm 4 3 1 Rover Compute Element (RCE) 47 38 2 7 Remote Electronics Unit (REU) Off-Nominal 14 10 4 Radio Science Beacon (RSB) Off-Nominal 11 3 6 1 1 System Fault Pro
13、tection (SFP) 1 1 Wakeup & Shutdown 15 9 2 4 LCA Cruise, Attitude Control System (ACS), and Propulsion 16 15 1 Surface- General Actuators and Motor Controller (MC) 50 50 Communication Behavior (CBM) 2 1 1 Mobility, Vol. 1 29 29 Surface Attitude & Position (SAPP) 8 8 Surface- Remote Science Dynamic A
14、lbedo of Neutrons (DAN) 3 3 Malin Space Science Systems (MSSS) Imaging 10 10 Remote Sensing Mast (RSM) 3 3 Surface- Sample Science Chemistry and Mineralogy instrument (CheMin) instrument 5 5 Collection and Handling for Interior Martian Rock Analysis (CHIMRA) 8 8 Drill 26 24 1 1 Robotic Arm 512 437 4
15、0 29 6 Table 2. MSL IV&V fault protection analysis findings to date (Reference (2), p. 14) The MSL IV&V team checked over 3,000 flight software requirements duplicated in three sources, identifying several inconsistencies (Reference (3). While the inconsistencies themselves did not represent serious
16、 discrepancies, the requirements-checking aided MSLs Certification of Flight Readiness (COFR) process. The PSSEs for the GRAIL, Juno, MSL, and SMAP projects met in August 2011 to discuss their experiences with NASA IV&V. Although they could not recall any mission-critical errors averted by IV&V, the
17、y concluded that IV&V support to JPL has resulted in valuable contributions to the projects without placing excessive demands on project resources. The IV&V teams have produced significant findings, such as: FSW discrepancies (e.g., both errors like MSL fatal events that that could cause a reset, an
18、d process deficiencies such as Juno interface control diagrams (ICDs) that were not properly traced and GRAIL requirements that were not properly tested), Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Design inconsistencies (e.g., FP responses to t
19、he same stimulus should not differ), Requirements-to-code tracing (i.e., identifying where the FSW code responds to a requirement), Missing or misapplied requirements in the requirements flow-down. (In Phases C and D of the Juno project, the IV&V team created tools for visualizing the requirements f
20、low-down.), Violations of coding standards (e.g., using initialized variables, defining variables but not using them, exceeding buffer limits). The open dialog with the IV&V teams was very beneficial to the JPL projects. For example, there were cases where a technical issue memorandum (TIM) would ha
21、ve been written, except a five-minute discussion concluded that the issue did not represent a valid problem. The collaborative environment prevented a “false positive.“ JPLs recent GRAIL, Juno, MSL, and SMAP project experience suggests that IV&V support can add value when well managed. It is importa
22、nt that the PSSE assist the IV&V team in focusing their efforts in areas that will best mitigate project risk. Rather than await formal reports, the project should schedule frequent meetings with the IV&V team to discuss potential TIMs. The IV&V team should be granted early access to draft artifacts
23、, arrive at a process for handling “observations” that are not formal findings, and triage TIMs to clarify which are mandatory for corrective action, rather than merely advisory. Less favorable outcomes are likely to result from refusing or limiting IV&V services, limiting interactions with IV&V to
24、discussions about already submitted TIMs, withholding artifacts until they are fully mature, or treating all TIMs as “must fix” regardless of their severity levels. These and other lessons learned from recent IV&V activities have been documented in newly released JPL procedures (Reference (3). Refer
25、ences: 1. Margaret Smith, “Experiences with IV&V Services on JPL Projects, Rev. 0,” JPL DocID 78609, March 29, 2012 2. Dave Nichols and Margaret Smith, “Modernizing the Project/IV&V Interface: How Projects Are Getting the Most Value from their IV&V Services, ” presented December 14, 2011. 3. “Suppor
26、ting the Scoping and Execution of IV&V Services on a Project, Rev. 0,” JPL Document No. DocID 78586, June 26, 2012. 4. Steve Larson, “Analysis of Phoenix Anomalies and IV&V Findings Applied to the GRAIL Mission,” 2012 IEEE Aerospace Conference, March 3-10, 2012. Lesson(s) Learned: Recent NASA spacef
27、light projects that have undergone IV&V of their mission software perceive the process as offering concrete benefits beyond those accrued through V&V performed solely by project personnel. Frequently, it was IV&V discoveries that stimulated the PSSEs to study the flight software coding and functiona
28、lity and to verify that a defect existed. Recommendation(s): 1. The NASA IV&V Center should continue to retrospectively evaluate their project IV&V performance to identify opportunities for process improvement. 2. Future FSW development contracts should take into account FSW deficiencies identified
29、in the IV&V process. Assure that lessons learned are incorporated into future contracts. This may have a big impact because for spaceflight projects, flight software development is a major contributor among those activities that challenge project cost control. 3. Projects selected for IV&V Services
30、should follow successful models used on earlier spaceflight missions when interacting with the IV&V Center (Reference 1). For example: Line management should hold regular meetings with the IV&V team to resolve cross-project issues, but the PSSE should remain the projects main point-of-contact with t
31、he IV&V team. Full engagement and IV&V support by the project (i.e., project management, the PSSE, and the in-house or contracted FSW management team) are needed to ensure a productive relationship with the IV&V team. Resistance to IV&V team participation, or mere accommodation (e.g., delivering art
32、ifacts as requested but otherwise disengaged), will likely lead to a poor return on resource investment. Hence, the project should help structure the IV&V teams involvement to be success-oriented, and proactively identify and deliver the artifacts that will be useful to them. Establish firm rules of
33、 engagement prior to the release of draft artifacts to the IV&V team. The rules should assure that the IV&V team write TIMs only against an agreed upon set of formally released documents. Inappropriate TIMs diminish the credibility of the IV&V product. Applicable IV&V lessons learned from one projec
34、t should be applied to subsequent missions. For example, the GRAIL project analyzed the impact on post-launch anomalies of 893 TIMS from the Mars Phoenix mission, and used the results to improve the efficiency and effectiveness of the GRAIL IV&V activity (Reference (4). Evidence of Recurrence Contro
35、l Effectiveness: JPL has referenced this lesson learned as additional rationale and guidance supporting Paragraph 6.19.2 (“Software IV&V”) in the Jet Propulsion Laboratory standard “Flight Project Practices, Rev. 9,” JPL DocID 58032, August 21, 2012. Documents Related to Lesson: N/A Provided by IHSN
36、ot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Mission Directorate(s): Aeronautics Research Exploration Systems Science Space Operations Additional Key Phrase(s): Additional Categories.Independent Verification and Validation Safety and Mission Assurance.Quality Eng
37、ineering Design (Phase C/D).Software Engineering Systems Engineering and Analysis.Planning of requirements verification processes Program Management.Cross Agency coordination Program Management.Acquisition / procurement strategy and planning Additional Categories.Software Additional Info: Project: M
38、ars Science Laboratory (MSL), Juno, Gravity Recovery and Interior Laboratory (GRAIL), and Soil Moisture Active-Passive (SMAP) Approval Info: Approval Date: 2012-10-26 Approval Name: mbell Approval Organization: HQ Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-
