The max issue on the Boeing planes wasn’t a software issue in that sense. The software actually worked as intended. Boeing just hid the fact that the software existed and made it purposefully hard to override, giving pilots quite the surprise when the plane didn’t operate the way they expected, especially when the angle of attack sensor wasn’t working.
There are all kinds of aspects to software development. Under-training pilots and then having software lead them to crash a plane full of passengers counts as a software development fail in my book. Also a human factors fail, marketing fail, a management fail, a quality systems fail, probably a regulatory oversight fail… it’s up there with ignoring the guy telling the Space Shuttle mission control not to launch when it has been under 32F the night before, which wasn’t a software fail at all, but was a broad cultural failure. Boeing’s MAX-10 situation just happened to have software in the crux of the matter.
I do software development. When I see something, I say something. I’m frequently ignored. Several times over the years, I’ve had the people that ignored me come back asking for help fixing the issue, after they realized that it was more important than they thought it was when they initially ignored me. Luckily, none of the people who died in those situations died (or were harmed in any significant way) because of the problem, but people dying “near the problem” brought the seriousness of it to a level where it finally got addressed.
If someone demands a piece of software that completely erases all available data when a certain button is pressed, no one is told about it, someone presses it by accident and deletes a whole bunch of data, that’s not a software development error. That’s a management failure. The software did exactly what it was supposed to do.
And LLMs wouldn’t make anything better in that regard. An LLM is built to do as the user wishes, with much, much less pushback than a human. So whereas a human getting the demand to write that software might say, “what in the actual fuck are you smoking and can I have some of it”, an LLM will typically say, “that’s a very bold decision!” In fact, in the case of MCAS, the issue of safety was raised by the engineers many times, and was flat out ignored by management. An LLM is much less likely to make a fuss and much more likely to just do it.
Very true. LLMs are not likely to perform any more admirably in the Milgram experiments than people do. Although, with a bit of appropriate pre-prompting, you might regulate LLM output to better comply with regulations. I think people are even more easily intimidated to “do what they are told” both directly and implied.
If you ask LLMs about cybersecurity implementation decisions, and keep the prompts straightforward like: “is this adequately secure for the target environments?” they are coming out on the highly conservative / secure side of things lately. Of course you can prompt and reprompt them with ever more detailed expectations until they agree with whatever you say, but out of the gate a clean context will generally bring you every “best practice” and then some and point out a big list of things your legacy system isn’t doing.
The max issue on the Boeing planes wasn’t a software issue in that sense. The software actually worked as intended. Boeing just hid the fact that the software existed and made it purposefully hard to override, giving pilots quite the surprise when the plane didn’t operate the way they expected, especially when the angle of attack sensor wasn’t working.
There are all kinds of aspects to software development. Under-training pilots and then having software lead them to crash a plane full of passengers counts as a software development fail in my book. Also a human factors fail, marketing fail, a management fail, a quality systems fail, probably a regulatory oversight fail… it’s up there with ignoring the guy telling the Space Shuttle mission control not to launch when it has been under 32F the night before, which wasn’t a software fail at all, but was a broad cultural failure. Boeing’s MAX-10 situation just happened to have software in the crux of the matter.
I do software development. When I see something, I say something. I’m frequently ignored. Several times over the years, I’ve had the people that ignored me come back asking for help fixing the issue, after they realized that it was more important than they thought it was when they initially ignored me. Luckily, none of the people who died in those situations died (or were harmed in any significant way) because of the problem, but people dying “near the problem” brought the seriousness of it to a level where it finally got addressed.
If someone demands a piece of software that completely erases all available data when a certain button is pressed, no one is told about it, someone presses it by accident and deletes a whole bunch of data, that’s not a software development error. That’s a management failure. The software did exactly what it was supposed to do.
And LLMs wouldn’t make anything better in that regard. An LLM is built to do as the user wishes, with much, much less pushback than a human. So whereas a human getting the demand to write that software might say, “what in the actual fuck are you smoking and can I have some of it”, an LLM will typically say, “that’s a very bold decision!” In fact, in the case of MCAS, the issue of safety was raised by the engineers many times, and was flat out ignored by management. An LLM is much less likely to make a fuss and much more likely to just do it.
Very true. LLMs are not likely to perform any more admirably in the Milgram experiments than people do. Although, with a bit of appropriate pre-prompting, you might regulate LLM output to better comply with regulations. I think people are even more easily intimidated to “do what they are told” both directly and implied.
If you ask LLMs about cybersecurity implementation decisions, and keep the prompts straightforward like: “is this adequately secure for the target environments?” they are coming out on the highly conservative / secure side of things lately. Of course you can prompt and reprompt them with ever more detailed expectations until they agree with whatever you say, but out of the gate a clean context will generally bring you every “best practice” and then some and point out a big list of things your legacy system isn’t doing.