1、Lessons Learned Entry: 1519Lesson Info:a71 Lesson Number: 1519a71 Lesson Date: 2004-11-29a71 Submitting Organization: JPLa71 Submitted by: Neil Toy / Steve SpohnSubject: Procure an Adequate Mission Operations Server (2004) Abstract: Because the Operations Storage Server was procured and installed ve
2、ry late in Phase D and featured unreliable, non-integrated hardware that could not reliably provide the dataflow performance required for critical MER mission operations, it posed a mission risk and had to be replaced just prior to EDL. Procure an integrated, expandable suite of server hardware that
3、 fully meets mission data throughput and reliability requirements, install the system at least nine months prior to its first planned operational use, and avoid subsequent major MOS/GDS IT infrastructure changes.Description of Driving Event: An Operations Storage Server (OSS) capable of supporting b
4、oth Mars Exploration Rover missions (MER-A and MER-B) was implemented, but only after early server hardware failures that necessitated major re-planning. In 2002, during Phase C (Design & Build) of the MER project, plans were proposed to acquire a high performance, centralized, project server to rap
5、idly process the high volumes of science data expected of the Ground Data System (GDS) during operations on the Martian surface. The Mission Operations System (MOS) funding in Phase D (integration & test), however, was inadequate to support full implementation of the recommended OSS system. The less
6、 expensive OSS that was deployed experienced repeated failures during pre-launch test, post-launch test, and launch/cruise operations. This included major server hardware failures lasting for days that occurred only weeks prior to MER-A launch in June 2003, and several weeks prior to Entry, Descent,
7、 and Landing (EDL) activities in January 2004. Contributing issues included:a71 Late procurement of a server system comprised of low-end, poorly integrated hardware incapable of meeting mission requirements for speed and reliability.a71 OSS design engineering involvement in the OSS was significantly
8、 reduced (from 1.5 FTE to 0.25 FTE) prior to final OSS implementation.Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-a71 Very limited testing prior to operational use because the original OSS was not installed until late in system integration and te
9、st (ATLO).a71 Major changes to the system architecture from a single isolated server assigned to each MER mission, to three servers networked between two buildings, multiple floors and four subnets, plus backup servers.Reliability concerns prompted the MER project to scrap this OSS design late in 20
10、03, and procure a turnkey system that was installed by the vendor just prior to EDL. The project was then able to use components of the original system to provide a backup server capability. This cluster system has successfully supported both rover missions without a major failure. Additional Key Wo
11、rds: Data Acquisition System, Distributed Computing Environment, Ground & Mission Operations, Multimission Data Services, Mission Data System, science data processing, server architecture Lesson(s) Learned: Critical mission dataflow (ground) components that are procured and implemented too late, or
12、feature low performance, non-integrated hardware, are unlikely to reliably provide the speed and capability required by mission operations.Recommendation(s): Procure and provide design engineering support to implement an integrated suite of server hardware that fully meets mission data throughput re
13、quirements and can be expanded without redesign. Utilize the expertise of experienced vendors to install the system at least nine months prior to expected operational use (e.g., launch operations). Stabilize Mission Support Area (MSA) plans early to prevent infrastructure changes that could impact o
14、verall system reliability and performance. Plan for a robust backup system.Evidence of Recurrence Control Effectiveness: Preventive Action Notice No. Z87145 was opened by JPL on August 2, 2005 to initiate and document appropriate Laboratory-wide corrective action on the above recommendation.Document
15、s Related to Lesson: JPL Flight Project Practices (Doc ID 58032), Paragraph 5.20: Science Data Management, Feb 27, 2003.Mission Directorate(s): Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-a71 ScienceAdditional Key Phrase(s): a71 Computersa71 Faci
16、litiesa71 Flight Operationsa71 Ground Equipmenta71 Ground Operationsa71 Hardwarea71 Information Technology/Systemsa71 Policy & Planninga71 Procurement Small Business & Industrial Relationsa71 Program and Project ManagementAdditional Info: Approval Info: a71 Approval Date: 2005-04-14a71 Approval Name: Carol Dumaina71 Approval Organization: JPLa71 Approval Phone Number: 818-354-8242Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-
