Boeing admits it failed to test the Starliner spacecraft adequately before its maiden flight

In the most comprehensive comments to date on what went wrong during Boeing's test mission - an autonomous flight without astronauts - to the International Space Station, John Mulholland, the manager of Boeing's Starliner program, said the company had cut short a key test of the craft's software, failed to test a critical system against crucial hardware, and instead used a flawed computer system to conduct the test. 

It was a stunning admission from the world's largest aerospace company, which has been beset with questions about the software aboard its 737 Max aircraft. That software is being blamed for two fatal crashes that killed 346 people and led to the global grounding of the aircraft nearly a year ago.

Mulholland said that in addition to reviewing all 1 million lines of software code on the spacecraft, Boeing will revamp the way it tests its systems before they are put into service.

"From a hindsight standpoint, it's very easy to see what we should have done because we uncovered an error," he said. "But I really don't want anyone to have the impression that this team tried to take shortcuts. They didn't. They did an abundance of testing. And in certain areas obviously we have some gaps to fill."

With the flight of its Starliner spacecraft, Boeing had hoped to show that it was getting close to flying NASA's astronauts to the space station as part of the agency's multibillion-dollar "commercial crew program." But instead, the mission became another in a series of significant setbacks for the beleaguered company.

The Starliner spacecraft encountered problems almost immediately after reaching space when its onboard computer, with its time miscalibrated by 11 hours, failed to fire the thrusters that would send the craft on a path to the space station. While aloft, the spacecraft struggled to communicate with the ground. By the time controllers could figure out what went wrong, the spacecraft had burned too much fuel, preventing it from docking with the station, one of its primary objectives.

When Boeing officials discovered a software issue had caused the timing problem, they immediately began to hunt for other potential problems. It found a big one: another issue that would have fired the wrong thrusters during the separation of the service module and the crew module.

Boeing was able to quickly diagnose that problem and send a software patch to the spacecraft. The spacecraft then landed safely in the New Mexico desert several days ahead of schedule.

NASA is investigating the mishap and expects to release more results of its probe next week. The space agency is still mulling whether to require Boeing to fly another test mission without astronauts or to proceed to a flight with three people on board

Doug Loverro, NASA's head of human spaceflight, said earlier this month that Boeing's problems were a "fundamental" and widespread "breakdown of the software process."

"We don't know how many software errors we have - if we have just two or many hundreds," he said.

He said the company needed to make its testing procedures more robust. And in a blog post, NASA said "there were numerous instances where the Boeing software quality processes either should have or could have uncovered the defects." It added that the problems could have had serious consequences and "led to spacecraft loss."

Speaking to reporters on Friday, Mulholland said that the company has been looking "to see if we have any other coding anomalies we need to fix. We haven't found any yet."

While many things went right during the mission, from its heatshield to environmental controls to the landing, he said the company knew "we need to do our part to rebuild trust with our customer."

On the timing issue, he said the clock on the spacecraft was pulling its time from the rocket. During tests of the software in the laboratory, the crews were primarily concerned with making sure the two vehicles were communicating correctly. The testing team proved there were no communication issues, but it cut the test short so it never uncovered that the spacecraft was reading the wrong time.

"Unfortunately, the run was stopped after we separated from the launch vehicle," he said.

If the test had continued, "we would have caught it."

Boeing also had a breakdown in the testing of another key milestone in the flight - the moment when the service module was to separate from the crew module just before re-entry into the atmosphere. At the same time the test of the software was supposed to happen, Boeing had simultaneously scheduled a "hot fire" test of the module's thrusters. As a result, the actual service module hardware was at another location for that test. To test the software, then, Boeing officials used an "emulator," a computer system used to mimic the service module.

The problem was the emulator had the wrong thruster configuration programmed in.

Mulholland said the problems with the tests were "definitely not a matter of cost. Cost has never been in any way a key factor in any of our decisions on how we need to test our systems. The team thought it was more logical to break these mission phases into chunks and to do a lot of testing in those smaller chunks."

In the future, he said, Boeing would continue to test the systems in smaller chunks but then also perform longer tests to simulate the moment from launch to docking at the space station, and then from release to landing.

"This is a tough business," he said. "It's a game of inches. And so you had a highly talented, very dedicated team that made that error. And going forward we just need to make sure we have the discipline that it won't happen again."