To date, SpaceX has completed 24 tests of its upgraded Mark 3 parachute design they are working to certify for use on the Crew Dragon spacecraft that will fly NASA astronauts to the International Space Station. The system was used during the SpaceX in-flight abort test in January.
On March 24, SpaceX lost a spacecraft-like device used to test the Crew Dragon Mark 3 parachute design. The test requires a helicopter to lift the device suspended underneath it to reach the needed test parameters. However, the pilot proactively dropped the device in an abundance of caution to protect the test crew as the test device became unstable underneath the helicopter. At the time of the release, the testing device was not armed, and a test of the parachute design was not performed.
Although losing a test device is never a desired outcome, NASA and SpaceX always will prioritize the safety of our teams over hardware. We are looking at the parachute testing plan now and all the data we already have to determine the next steps ahead of flying the upcoming Demo-2 flight test in the mid-to-late May timeframe.
The Boeing CST-100 Starliner spacecraft at the company’s Commercial Crew and Cargo Processing Facility in Florida, undergoing inspection after its Orbital Flight Test. (Photo Credit: NASA/Frank Michaux)
The joint NASA and Boeing Independent Review Team formed following the anomalies during the company’s uncrewed Orbital Flight Test as a part of the agency’s Commercial Crew Program has completed its initial investigation. The team was tasked with reviewing three primary anomalies experienced during the mission: two software coding errors and unanticipated loss of space-to-ground communication capability. During the investigation, the team identified several technical and organizational issues related to Boeing’s work. Separate from the independent team, NASA reviewed its role in the flight test and identified several areas where the agency can improve its level of participation and involvement into company’s processes.
While the review team, NASA and Boeing have made significant progress during the last month, more work will be required to inform the agency’s decision of whether Boeing will need to perform another uncrewed test flight of the Starliner system. NASA will determine whether a repeat of the flight will be needed after Boeing has presented its detailed resolution and rework plan, and NASA has independently assessed the thoroughness of that plan.
NASA also will perform an evaluation of the workplace culture of Boeing ahead of crewed test flights through an Organizational Safety Assessment (OSA). The goal of the OSA is to provide a comprehensive safety assessment through individual employee interviews with a sampling from a cross-section of personnel, including senior managers, mid-level management and supervision, and engineers and technicians at various sites.
Further, NASA will designate the anomalies experienced during the mission as a high visibility close call. As there were no injuries during the flight, this close call designation is where the potential for a significant mishap could have occurred and should be investigated to understand the risk exposure and the root cause(s) that placed equipment or individuals at risk. Since 2004, the year NASA updated this procedural requirement, NASA has designated about 24 high visibility close calls. For example, in July 2013, astronaut Luca Parmitano discovered a leak in his spacesuit that could have resulted in asphyxiation; as a result, that incident also was given the same designation.
Description of the three primary anomalies:
Mission Elapsed Timer (MET): Following spacecraft separation with the Atlas V launch vehicle, Boeing’s CST-100 Starliner is programmed to execute a few maneuvers tied to the mission timer. Because of an error in the coding, the Starliner synced its clock with the rocket before the terminal count had begun, which is when the rocket sets the correct time for a designated T-0. This led to the spacecraft thinking it was at a different point in the mission following separation, and it did not conduct the correct maneuvers.
Service Module Disposal Burn: Following the MET anomaly, Boeing and NASA reviewed other phases of flight where software coding could impact mission success. This review resulted in the team discovering and correcting a software issue during Starliner’s crew and service module separation sequence. The correction ensured a successful separation and disposal of the service module.
Space-to-Ground Communication (S/G): An Intermittent S/G forward link issue impeded the flight control team’s ability to command and control Starliner during the mission and could impede reliable voice communication with crew during a flight with astronauts.
What the Review Team Found and Recommends
The review team’s analysis identified 61 corrective and preventative actions to address the two software anomalies; those actions are organized into four categories to help manage and execute the scope of the work. Below are the four categories and examples of the resulting actions that Boeing has already begun working on:
Perform code modifications: Boeing will review and correct the coding for the mission elapsed timer and service module disposal burn.
Improve focused systems engineering: Boeing will strengthen its review process including better peer and control board reviews, and improve its software process training.
Improve software testing: Boeing will increase the fidelity in the testing of its software during all phases of flight. This includes improved end-to-end testing with the simulations, or emulators, similar enough to the actual flight system to adequately uncover issues.
Ensure product integrity: Boeing will check its software coding as hardware design changes are implemented into its system design.
Boeing already has accepted the full action list as defined by the review team and is in the process of refining its implementation schedule and incorporating this work into its plans with multiple actions already underway. As work continues, NASA and Boeing have asked the joint review team to track their progress and execution of each action.
The review team also is continuing its investigation of the intermittent space-to-ground forward link issue that impeded the flight control team’s ability to command and control the spacecraft. The team has identified the technical root cause as radiofrequency interference with the communications system. While the team has recommended specific hardware improvements already in work by the company, the full assessment and resulting recommendations will continue through March.
In addition to the technical issues described above, the review team identified organizational issues that contributed to the anomalies. In response, Boeing plans to institutionalize improvements in its engineering board authority, operational testing practices for both hardware and software, and the standardization problem review and approval processes.
NASA’s Internal Review and Forward Work
Concurrent with the independent review team, NASA performed an in-depth assessment of its role and identified multiple actions the agency will take to complement the actions planned by the Boeing Starliner team.
NASA has developed a comprehensive plan to ensure the agency has full coverage of critical Boeing software improvements. This plan also includes reassessing all hazard report verifications of software controls, re-opening hazard reports as necessary, reviewing software verification plans, and reviewing the adequacy of the test environments and audits of scripts used in testing. NASA also will co-locate personnel with the Boeing software team, increase support to the Boeing Software Change Control Board and the problem resolution process. NASA also plans to perform additional flight software audits.
In addition, NASA will improve its software independent verification and validation performance and overall NASA insight into this area. NASA also plans to address areas where additional NASA “safety nets” may be beneficial for all providers.
NASA also will take several actions to improve the overall system integration of Starliner, including revisiting all hazard causes related to system interfaces to ensure hazards are fully defined, well-controlled, and properly verified; and reviewing existing Interface Control Documents to ensure NASA understands where the definitive data sources are for subsystem interfaces.
Boeing, NASA, and U.S. Army personnel work around the Boeing CST-100 Starliner spacecraft shortly after it landed in White Sands, New Mexico, Sunday, Dec. 22, 2019. Photo Credit: (NASA/Bill Ingalls)
Following the anomaly that occurred during the December Boeing Starliner Orbital Fight Test (OFT), NASA and Boeing formed a joint investigation team tasked with examining the primary issues, which occurred during that test. Those issues included three specific concerns revealed during flight:
An error with the Mission Elapsed Timer (MET), which incorrectly polled time from the Atlas V booster nearly 11 hours prior to launch.
A software issue within the Service Module (SM) Disposal Sequence, which incorrectly translated the SM disposal sequence into the SM Integrated Propulsion Controller (IPC).
An Intermittent Space-to-Ground (S/G) forward link issue, which impeded the Flight Control team’s ability to command and control the vehicle.
The joint investigation team convened in early January and has now identified the direct causes and preliminary corrective actions for the first two anomalies. The intermittent communications issues still are under investigation. NASA reviewed these results on Friday, Jan. 31 along with multiple suggested corrective actions recommended by the team. While NASA was satisfied that the team had properly identified the technical root cause of the two anomalies, they requested the team to perform a more in-depth analysis as to why the anomalies occurred, including an analysis of whether the issues were indicative of weak internal software processes or failure in applying those processes. The team is in the process of performing this additional analysis, as well as continuing the investigation of the intermittent communications issues. NASA briefed the Aerospace Safety Advisory Panel on the status of the investigation this week.
Regarding the first two anomalies, the team found the two critical software defects were not detected ahead of flight despite multiple safeguards. Ground intervention prevented loss of vehicle in both cases. Breakdowns in the design and code phase inserted the original defects. Additionally, breakdowns in the test and verification phase failed to identify the defects preflight despite their detectability. While both errors could have led to risk of spacecraft loss, the actions of the NASA-Boeing team were able to correct the issues and return the Starliner spacecraft safely to Earth.
There was no simple cause of the two software defects making it into flight. Software defects, particularly in complex spacecraft code, are not unexpected. However, there were numerous instances where the Boeing software quality processes either should have or could have uncovered the defects. Due to these breakdowns found in design, code and test of the software, they will require systemic corrective actions. The team has already identified a robust set of 11 top-priority corrective actions. More will be identified after the team completes its additional work.
The joint team made excellent progress for this stage of the investigation. However, it’s still too early for us to definitively share the root causes and full set of corrective actions needed for the Starliner system. We do expect to have those results at the end of February, as was our initial plan. We want to make sure we have a comprehensive understanding of what happened so that we can fully explain the root causes and better assess future work that will be needed. Most critically, we want to assure that these necessary steps are completely understood prior to determining the plan for future flights. Separate from the anomaly investigation, NASA also is still reviewing the data collected during the flight test to help determine that future plan. NASA expects a decision on this review to be complete in the next several weeks.
NASA and Boeing are committed to openly sharing the information related to the mission with the public. Thus, NASA will be holding a media teleconference at 3:30 p.m. EST Friday, Feb. 7.
In addition to these reviews, NASA is planning to perform an Organizational Safety Assessment of Boeing’s work related to the Commercial Crew Program. The comprehensive safety review will include individual employee interviews with a sampling from a cross section of personnel, including senior managers, mid-level management and supervision, and engineers and technicians at multiple sites. The review would be added to the company’s Commercial Crew Transportation Capability contract. NASA previously completed a more limited review of the company. The goal of the Organizational Safety Assessment will be to examine the workplace culture with the commercial crew provider ahead of a mission with astronauts.
Boeing’s Orbital Flight test launched on Friday, Dec. 20, on United Launch Alliance Atlas V rocket from Space Launch Complex 41 at Cape Canaveral Air Force Station in Florida. The mission successfully landed two days later on Sunday, Dec. 22, completing an abbreviated test that performed several mission objectives before returning to Earth as the first orbital land touchdown of a human-rated capsule in U.S. history.
The Boeing CST-100 Starliner spacecraft is back home at the company’s Commercial Crew and Cargo Processing Facility, undergoing inspection after its first flight as part of NASA’s Commercial Crew Program, known as the Orbital Flight Test.
Starliner launched atop a United Launch Alliance Atlas V rocket from Space Launch Complex 41 at Cape Canaveral Air Force Station (CCAFS) in Florida, Friday, Dec. 20, 2019. The mission successfully landed two days later on Sunday, Dec. 22, completing an abbreviated test that performed several mission objectives before returning to Earth as the first orbital land touchdown of a human-rated capsule in U.S. history.
Boeing, NASA, and U.S. Army personnel work around the Boeing CST-100 Starliner spacecraft shortly after it landed in White Sands, New Mexico, Sunday, Dec. 22, 2019. Photo Credit: (NASA/Bill Ingalls)
NASA and Boeing are in the process of establishing a joint, independent investigation team to examine the primary issues associated with the company’s uncrewed Orbital Flight Test.
The independent team will inform NASA and Boeing on the root cause of the mission elapsed timer anomaly and any other software issues and provide corrective actions needed before flying crew to the International Space Station for the agency’s Commercial Crew Program. The team will review the primary anomalies experienced during the Dec. 2019 flight test, any potential contributing factors and provide recommendations to ensure a robust design for future missions. Once underway, the investigation is targeted to last about two months before the team delivers its final assessment.
In parallel, NASA is evaluating the data received during the mission to determine if another uncrewed demonstration is required. This decision is not expected for several weeks as teams take the necessary time for this review. NASA’s approach will be to determine if NASA and Boeing received enough data to validate the system’s overall performance, including launch, on-orbit operations, guidance, navigation and control, docking/undocking to the space station, reentry and landing. Although data from the uncrewed test is important for certification, it may not be the only way that Boeing is able to demonstrate its system’s full capabilities.
The uncrewed flight test was proposed by Boeing as a way to meet NASA’s mission and safety requirements for certification and as a way to validate that the system can protect astronauts in space before flying crew. The uncrewed mission, including docking to the space station, became a part of the company’s contract with NASA. Although docking was planned, it may not have to be accomplished prior to the crew demonstration. Boeing would need NASA’s approval to proceed with a flight test with astronauts onboard.
Starliner currently is being transported from the landing location near the U.S. Army’s White Sands Missile Range to the company’s Commercial Crew and Cargo Processing Facility in Florida. Since landing, teams have safed the spacecraft for transport, downloaded data from the spacecraft’s onboard systems for analysis and completed initial inspections of the interior and exterior of Starliner. A more detailed analysis will be conducted after the spacecraft arrives at its processing facility.
Boeing’s Orbital Flight test launched on Friday, Dec. 20, on United Launch Alliance Atlas V rocket from Space Launch Complex 41 at Cape Canaveral Air Force Station in Florida. The mission successfully landed two days later on Sunday, Dec. 22, completing an abbreviated test that performed several mission objectives before returning to Earth as the first orbital land touchdown of a human-rated capsule in U.S. history.
SpaceX Crew Dragon spacecraft inside of a SpaceX processing facility at Cape Canaveral in Florida. Credit: SpaceX
NASA and SpaceX are targeting no earlier than Saturday, Jan. 18, for an In-Flight Abort Test of the Crew Dragon spacecraft from Launch Complex 39A at the Kennedy Space Center, Florida, pending U.S. Air Force Eastern Range approval. The new date allows additional time for spacecraft processing.
The demonstration of Crew Dragon’s in-flight launch escape system is part of NASA’s Commercial Crew Program and is one of the final major tests for the company before NASA astronauts will fly aboard the spacecraft.
The Boeing CST-100 Starliner spacecraft lands in White Sands, New Mexico, Sunday, Dec. 22, 2019. Photo Credit: (NASA/Aubrey Gemignani)
Boeing’s CST-100 Starliner spacecraft completed the first land touchdown of a human-rated capsule in U.S. history Sunday at White Sands Space Harbor in New Mexico, wrapping up the company’s uncrewed Orbital Flight Test as part of NASA’s Commercial Crew Program.
Starliner settled gently onto its airbags at 7:58 a.m. EST (5:58 a.m. MST) in a pre-dawn landing that helps set the stage for future crewed landings at the same site. The landing followed a deorbit burn at 7:23 a.m., separation of the spacecraft’s service module, and successful deployment of its three main parachutes and six airbags.
The Boeing CST-100 Starliner spacecraft is seen after it landed in White Sands, New Mexico, Sunday, Dec. 22, 2019. Photo Credit: (NASA/Bill Ingalls)
Boeing’s CST-100 Starliner landed safely this morning, concluding the company’s Orbital Flight Test for NASA’s Commercial Crew Program.
NASA Astronaut Suni Williams, who will command the next mission of the Starliner that landed this morning, named this Orbital Flight Test capsule “Calypso.” The name is an ode to the ship of Jacques Cousteau and invokes the vastness of the sea and space.
Tune into NASA TV for a postlanding news conference at 10:00 a.m. EST with:
NASA Administrator Jim Bridenstine
Jim Chilton, senior vice president of Boeing’s Space and Launch Division
Steve Stich, deputy manager of NASA’s Commercial Crew Program
Boeing’s CST-100 Starliner moments after landing in White Sands, New Mexico
Boeing’s CST-100 Starliner touched down safely at White Sands Space Harbor, New Mexico at 7:58 a.m. EST, concluding its Orbital Flight Test for NASA’s Commercial Crew Program. Recovery teams are beginning work to retrieve Starliner this morning.
Infrared image of CST-100 Starliner touching down in White Sands, New Mexico
NASA and Boeing will host a postlanding news conference at 10 a.m. EST with:
NASA Administrator Jim Bridenstine
Jim Chilton, senior vice president of Boeing’s Space and Launch Division
Steve Stich, deputy manager of NASA’s Commercial Crew Program
To participate in the postlanding news conference via phone bridge, media must contact the newsroom at NASA’s Johnson Space Center at 281-483-5111 no later than 9:45 a.m. The news conference will air live on NASA TV and the agency’s website.
Boeing’s CST-100 Starliner’s parachutes have deployed in preparation for landing at White Sands Space Harbor in New Mexico. Landing is estimated at 7:57 a.m. EST, approximately one hour before sunrise in the desert.
Tune in to NASA TV now for continuing live coverage.
To date, SpaceX has completed 24 tests of its upgraded Mark 3 parachute design they are working to certify for use on the Crew Dragon spacecraft that will fly NASA astronauts to the International Space Station. The system was used during the SpaceX in-flight abort test in January.
