NASA, SpaceX Team Up for Emergency Egress Exercise

On Friday, April 3, 2020, NASA and SpaceX completed an end-to-end demonstration of the teams’ ability to safely evacuate crew members from the Fixed Service Structure during an emergency situation at Launch Complex 39A at NASA’s Kennedy Space Center in Florida.
On Friday, April 3, 2020, NASA and SpaceX completed an end-to-end demonstration of the teams’ ability to safely evacuate crew members from the Fixed Service Structure during an emergency situation at Launch Complex 39A at NASA’s Kennedy Space Center in Florida. Photo credit: SpaceX

Safety is a top priority as NASA and SpaceX prepare for liftoff of the company’s second demonstration flight test (Demo-2), the first flight to carry astronauts to the International Space Station onboard the Crew Dragon spacecraft as part of NASA’s Commercial Crew Program. The teams conducted an emergency egress exercise at Launch Complex 39A at the agency’s Kennedy Space Center in Florida on April 3. The end-to-end demonstration is the latest in a series of similar exercises to ensure the crew and support teams can quickly evacuate from the launch pad in the unlikely event of an emergency prior to liftoff.

On Friday, April 3, 2020, NASA and SpaceX completed an end-to-end demonstration of the teams’ ability to safely evacuate crew members from the Fixed Service Structure during an emergency situation at Launch Complex 39A at NASA’s Kennedy Space Center in Florida.
Photo credit: SpaceX

NASA and SpaceX personnel, including the Kennedy pad rescue team, participated in the exercise. The primary objective was to demonstrate the teams’ ability to safely evacuate crew members from the launch pad during an emergency situation. Teams rehearsed locating injured personnel on the 265-foot-level of the launch tower, loading them into the pad’s slidewire baskets and safely descending the tower, then successfully loading the injured participants into Mine Resistant Ambush Protected (MRAP) vehicles staged at the pad perimeter.

Scheduled for launch no earlier than May 2020, Demo-2 will be the first launch of NASA astronauts from American soil to the International Space Station since the space shuttle era. It also is the final flight test for the SpaceX Falcon 9 rocket and Crew Dragon spacecraft system to be certified for regular flights to the station with crew onboard.

NASA Update on Orbital Flight Test Independent Review Team

The Boeing CST-100 Starliner spacecraft at the company’s Commercial Crew and Cargo Processing Facility in Florida, undergoing inspection after its Orbital Flight Test. (Photo Credit: NASA/Frank Michaux)

The joint NASA and Boeing Independent Review Team formed following the anomalies during the company’s uncrewed Orbital Flight Test as a part of the agency’s Commercial Crew Program has completed its initial investigation. The team was tasked with reviewing three primary anomalies experienced during the mission: two software coding errors and unanticipated loss of space-to-ground communication capability. During the investigation, the team identified several technical and organizational issues related to Boeing’s work. Separate from the independent team, NASA reviewed its role in the flight test and identified several areas where the agency can improve its level of participation and involvement into company’s processes.

While the review team, NASA and Boeing have made significant progress during the last month, more work will be required to inform the agency’s decision of whether Boeing will need to perform another uncrewed test flight of the Starliner system. NASA will determine whether a repeat of the flight will be needed after Boeing has presented its detailed resolution and rework plan, and NASA has independently assessed the thoroughness of that plan.

NASA also will perform an evaluation of the workplace culture of Boeing ahead of crewed test flights through an Organizational Safety Assessment (OSA). The goal of the OSA is to provide a comprehensive safety assessment through individual employee interviews with a sampling from a cross-section of personnel, including senior managers, mid-level management and supervision, and engineers and technicians at various sites.

Further, NASA will designate the anomalies experienced during the mission as a high visibility close call. As there were no injuries during the flight, this close call designation is where the potential for a significant mishap could have occurred and should be investigated to understand the risk exposure and the root cause(s) that placed equipment or individuals at risk. Since 2004, the year NASA updated this procedural requirement, NASA has designated about 24 high visibility close calls. For example, in July 2013, astronaut Luca Parmitano discovered a leak in his spacesuit that could have resulted in asphyxiation; as a result, that incident also was given the same designation.

Description of the three primary anomalies:

  • Mission Elapsed Timer (MET): Following spacecraft separation with the Atlas V launch vehicle, Boeing’s CST-100 Starliner is programmed to execute a few maneuvers tied to the mission timer. Because of an error in the coding, the Starliner synced its clock with the rocket before the terminal count had begun, which is when the rocket sets the correct time for a designated T-0. This led to the spacecraft thinking it was at a different point in the mission following separation, and it did not conduct the correct maneuvers.
  • Service Module Disposal Burn: Following the MET anomaly, Boeing and NASA reviewed other phases of flight where software coding could impact mission success. This review resulted in the team discovering and correcting a software issue during Starliner’s crew and service module separation sequence. The correction ensured a successful separation and disposal of the service module.
  • Space-to-Ground Communication (S/G): An Intermittent S/G forward link issue impeded the flight control team’s ability to command and control Starliner during the mission and could impede reliable voice communication with crew during a flight with astronauts.

What the Review Team Found and Recommends

The review team’s analysis identified 61 corrective and preventative actions to address the two software anomalies; those actions are organized into four categories to help manage and execute the scope of the work. Below are the four categories and examples of the resulting actions that Boeing has already begun working on:

  1. Perform code modifications: Boeing will review and correct the coding for the mission elapsed timer and service module disposal burn.
  2. Improve focused systems engineering: Boeing will strengthen its review process including better peer and control board reviews, and improve its software process training.
  3. Improve software testing: Boeing will increase the fidelity in the testing of its software during all phases of flight. This includes improved end-to-end testing with the simulations, or emulators, similar enough to the actual flight system to adequately uncover issues.
  4. Ensure product integrity: Boeing will check its software coding as hardware design changes are implemented into its system design.

Boeing already has accepted the full action list as defined by the review team and is in the process of refining its implementation schedule and incorporating this work into its plans with multiple actions already underway. As work continues, NASA and Boeing have asked the joint review team to track their progress and execution of each action.

The review team also is continuing its investigation of the intermittent space-to-ground forward link issue that impeded the flight control team’s ability to command and control the spacecraft. The team has identified the technical root cause as radiofrequency interference with the communications system. While the team has recommended specific hardware improvements already in work by the company, the full assessment and resulting recommendations will continue through March.

In addition to the technical issues described above, the review team identified organizational issues that contributed to the anomalies. In response, Boeing plans to institutionalize improvements in its engineering board authority, operational testing practices for both hardware and software, and the standardization problem review and approval processes.

NASA’s Internal Review and Forward Work

Concurrent with the independent review team, NASA performed an in-depth assessment of its role and identified multiple actions the agency will take to complement the actions planned by the Boeing Starliner team.

NASA has developed a comprehensive plan to ensure the agency has full coverage of critical Boeing software improvements. This plan also includes reassessing all hazard report verifications of software controls, re-opening hazard reports as necessary, reviewing software verification plans, and reviewing the adequacy of the test environments and audits of scripts used in testing. NASA also will co-locate personnel with the Boeing software team, increase support to the Boeing Software Change Control Board and the problem resolution process. NASA also plans to perform additional flight software audits.

In addition, NASA will improve its software independent verification and validation performance and overall NASA insight into this area. NASA also plans to address areas where additional NASA “safety nets” may be beneficial for all providers.

NASA also will take several actions to improve the overall system integration of Starliner, including revisiting all hazard causes related to system interfaces to ensure hazards are fully defined, well-controlled, and properly verified; and reviewing existing Interface Control Documents to ensure NASA understands where the definitive data sources are for subsystem interfaces.

NASA Shares Initial Findings from Boeing Starliner Orbital Flight Test Investigation

Boeing, NASA, and U.S. Army personnel work around the Boeing CST-100 Starliner spacecraft shortly after it landed in White Sands, New Mexico, Sunday, Dec. 22, 2019. Photo Credit: (NASA/Bill Ingalls)

Following the anomaly that occurred during the December Boeing Starliner Orbital Fight Test (OFT), NASA and Boeing formed a joint investigation team tasked with examining the primary issues, which occurred during that test. Those issues included three specific concerns revealed during flight:

  1. An error with the Mission Elapsed Timer (MET), which incorrectly polled time from the Atlas V booster nearly 11 hours prior to launch.
  2. A software issue within the Service Module (SM) Disposal Sequence, which incorrectly translated the SM disposal sequence into the SM Integrated Propulsion Controller (IPC).
  3. An Intermittent Space-to-Ground (S/G) forward link issue, which impeded the Flight Control team’s ability to command and control the vehicle.

The joint investigation team convened in early January and has now identified the direct causes and preliminary corrective actions for the first two anomalies. The intermittent communications issues still are under investigation. NASA reviewed these results on Friday, Jan. 31 along with multiple suggested corrective actions recommended by the team. While NASA was satisfied that the team had properly identified the technical root cause of the two anomalies, they requested the team to perform a more in-depth analysis as to why the anomalies occurred, including an analysis of whether the issues were indicative of weak internal software processes or failure in applying those processes. The team is in the process of performing this additional analysis, as well as continuing the investigation of the intermittent communications issues. NASA briefed the Aerospace Safety Advisory Panel on the status of the investigation this week.

Regarding the first two anomalies, the team found the two critical software defects were not detected ahead of flight despite multiple safeguards.  Ground intervention prevented loss of vehicle in both cases. Breakdowns in the design and code phase inserted the original defects. Additionally, breakdowns in the test and verification phase failed to identify the defects preflight despite their detectability. While both errors could have led to risk of spacecraft loss, the actions of the NASA-Boeing team were able to correct the issues and return the Starliner spacecraft safely to Earth.

There was no simple cause of the two software defects making it into flight. Software defects, particularly in complex spacecraft code, are not unexpected. However, there were numerous instances where the Boeing software quality processes either should have or could have uncovered the defects. Due to these breakdowns found in design, code and test of the software, they will require systemic corrective actions. The team has already identified a robust set of 11 top-priority corrective actions. More will be identified after the team completes its additional work.

The joint team made excellent progress for this stage of the investigation. However, it’s still too early for us to definitively share the root causes and full set of corrective actions needed for the Starliner system. We do expect to have those results at the end of February, as was our initial plan. We want to make sure we have a comprehensive understanding of what happened so that we can fully explain the root causes and better assess future work that will be needed. Most critically, we want to assure that these necessary steps are completely understood prior to determining the plan for future flights. Separate from the anomaly investigation, NASA also is still reviewing the data collected during the flight test to help determine that future plan. NASA expects a decision on this review to be complete in the next several weeks.

NASA and Boeing are committed to openly sharing the information related to the mission with the public. Thus, NASA will be holding a media teleconference at 3:30 p.m. EST Friday, Feb. 7.

In addition to these reviews, NASA is planning to perform an Organizational Safety Assessment of Boeing’s work related to the Commercial Crew Program. The comprehensive safety review will include individual employee interviews with a sampling from a cross section of personnel, including senior managers, mid-level management and supervision, and engineers and technicians at multiple sites. The review would be added to the company’s Commercial Crew Transportation Capability contract. NASA previously completed a more limited review of the company. The goal of the Organizational Safety Assessment will be to examine the workplace culture with the commercial crew provider ahead of a mission with astronauts.

Boeing’s Orbital Flight test launched on Friday, Dec. 20, on United Launch Alliance Atlas V rocket from Space Launch Complex 41 at Cape Canaveral Air Force Station in Florida. The mission successfully landed two days later on Sunday, Dec. 22, completing an abbreviated test that performed several mission objectives before returning to Earth as the first orbital land touchdown of a human-rated capsule in U.S. history.

NASA, SpaceX Complete Final Major Flight Test of Crew Spacecraft

SpaceX in-flight abort test
NASA and SpaceX completed a launch escape demonstration of the company’s Crew Dragon spacecraft and Falcon 9 rocket on Jan. 19, 2020. The test began at 10:30 a.m. EST with liftoff from Launch Complex 39A at NASA’s Kennedy Space Center in Florida on a mission to show the spacecraft’s capability to safely separate from the rocket in the unlikely event of an inflight emergency. Credits: NASA Television
Watch the prelaunch activities and launch

NASA and SpaceX completed a launch escape demonstration of the company’s Crew Dragon spacecraft and Falcon 9 rocket Sunday. This was the final major flight test of the spacecraft before it begins carrying astronauts to the International Space Station under NASA’s Commercial Crew Program.

The launch escape test began at 10:30 a.m. EST with liftoff from historic Launch Complex 39A at NASA’s Kennedy Space Center in Florida on a mission to show the spacecraft’s capability to safely separate from the rocket in the unlikely event of an inflight emergency.

“This critical flight test puts us on the cusp of returning the capability to launch astronauts in American spacecraft on American rockets from American soil,” said NASA Administrator Jim Bridenstine. “We are thrilled with the progress NASA’s Commercial Crew Program is making and look forward to the next milestone for Crew Dragon.”

As part of the test, SpaceX configured Crew Dragon to trigger a launch escape about 1.5 minutes after liftoff. All major functions were executed, including separation, engine firings, parachute deployment and landing. Crew Dragon splashed down at 10:38 a.m. just off the Florida coast in the Atlantic Ocean.

“As far as we can tell thus far, it’s a picture perfect mission. It went as well as one can possibly expect,” said Elon Musk, chief engineer at SpaceX. “This is a reflection of the dedication and hard work of the SpaceX and NASA teams to achieve this goal. Obviously, I’m super fired up. This is great.”

Teams of personnel from SpaceX and the U.S. Air Force 45th Operations Group’s Detachment-3 out of Patrick Air Force Base will recover the spacecraft for return to SpaceX facilities in Florida and begin the recovery effort of the Falcon 9, which broke apart as planned.

“The past few days have been an incredible experience for us,” said astronaut Doug Hurley. “We started with a full dress rehearsal of what Bob and I will do for our mission. Today, we watched the demonstration of a system that we hope to never use, but can save lives if we ever do. It took a lot of work between NASA and SpaceX to get to this point, and we can’t wait to take a ride to the space station soon.”

Prior to the flight test, teams completed launch day procedures for the first crewed flight test, from suit-up to launch pad operations. The joint teams now will begin the full data reviews that need to be completed prior to NASA astronauts flying the system during SpaceX’s Demo-2 mission.

NASA’s Commercial Crew Program is working with the American aerospace industry as companies develop and operate a new generation of spacecraft and launch systems capable of carrying crews to low-Earth orbit and the International Space Station. Commercial human space transportation to and from the station will provide expanded utility, additional research time and broader opportunities for discovery on the orbiting laboratory. The program also has the benefit of facilitating and promoting for America a vibrant economy in low-Earth orbit.

In-Flight Abort Post-Test News Conference Underway

A post-test news conference for NASA and SpaceX’s in-flight abort demonstration is taking place at NASA’s Kennedy Space Station in Florida.

An update from SpaceX and NASA officials, including NASA Commercial Crew Program astronauts Victor Glover and Mike Hopkins, is underway. Tune in to the post-test news conference, which is being broadcast live from Kennedy Space Center in Florida on NASA TV and the agency’s website. Participants include:

  • NASA Administrator Jim Bridenstine
  • Kathy Lueders, manager, NASA’s Commercial Crew Program
  • Elon Musk, chief engineering, SpaceX
  • Victor Glover, astronaut, NASA Commercial Crew Program
  • Mike Hopkins, astronaut, NASA Commercial Crew Program

Today’s in-flight abort blogging will conclude with a wrap-up post featuring comments from the post-test news conference.

Post-Test News Conference No Earlier Than 11:30 a.m. EST

SpaceX's In-Flight Abort Test
NASA and SpaceX successfully completed the in-flight abort test. The Crew Dragon spacecraft is now in the process of being recovered from the Atlantic Ocean. A post-test news conference is scheduled for today, no earlier than 11:30 EST.

The SpaceX Crew Dragon spacecraft splashed down offshore in the Atlantic Ocean at 10:39 a.m. EST after a launch on the company’s Falcon 9 rocket from Launch Complex 39A at NASA’s Kennedy Space Center in Florida.

Teams of personnel from SpaceX and the U.S. Air Force 45th Operations Groups Detachment-3 out of Patrick Air Force Base will recover the spacecraft for return to SpaceX facilities in Florida, and a dedicated team will begin the recovery effort of the Falcon 9, which broke apart as planned.

Next up on NASA television and the agency’s website:

  • 11:30 a.m. (no earlier than) – Post-test news conference at Kennedy, with the following representatives:
    • NASA Administrator Jim Bridenstine
    • Elon Musk, chief engineering, SpaceX
    • Kathy Lueders, manager, NASA Commercial Crew Program
    • Victor Glover, astronaut, NASA Commercial Crew Program
    • Mike Hopkins, astronaut, NASA Commercial Crew Program

Today’s in-flight abort blogging will conclude with a wrap-up post featuring comments from the post-test news conference.

Learn more about NASA’s Commercial Crew Program by following the commercial crew blog, @commercial_crew and commercial crew on Facebook.

Spacecraft Splashes Down Safely in the Atlantic Ocean

SpaceX’s Crew Dragon capsule has splashed down into the ocean! The test was completed in less than 10 minutes.

The next step is recovery of the spacecraft. That process is expected to take approximately two hours.

Stay tuned for a preview of the post-test news conference.

Crew Dragon Chutes Deploy

SpaceX In-Flight Abort Test
Crew Dragon’s drogue and main parachutes will enable the spacecraft to have a soft landing in the Atlantic Ocean.

Crew Dragon’s parachutes have deployed and the spacecraft is descending toward the Atlantic Ocean.

Splashdown is expected in about six minutes.

Spacecraft Maneuvers for Chute Deployment

SpaceX In-Flight Abort Test
The in-flight abort test will demonstrate the Crew Dragon spacecraft’s capability to safely separate from the Falcon 9 rocket in the event of a failure during launch.

Crew Dragon separated from the Falcon 9 rocket and the spacecraft’s trunk has deployed.

Parachute deploy is expected at about the four-minute mark.

Liftoff! SpaceX’s In-Flight Abort Test Underway

SpaceX In-Flight Abort Test
SpaceX’s Crew Dragon spacecraft and Falcon 9 rocket lifts off from Launch Complex 39A at NASA’s Kennedy Space Center in Florida on Sunday, Jan. 19, 2020.

We have liftoff! A Falcon 9 rocket has blasted off from Kennedy Space Center’s Launch Complex 39A, carrying the Crew Dragon spacecraft for NASA and SpaceX’s in-flight abort test.

Crew Dragon separation should occur at 1 minute, 24 seconds, followed by the spacecraft’s trunk deploy less than 40 seconds later.