Report: AI Mistake Nearly Led U.S. Military to Board a Chinese Ship
All users of AI are familiar with its tendency to produce false results from time to time, particularly when using older, lower-end chatbots. While more recent frontier models are less likely to feed out made-up results, they are still capable of error, which is why corporate adopters emphasize that accountability for the product remains with the human who delivers it - not the AI agent behind the scenes. The U.S. military also uses AI extensively for data analysis, and nearly got drawn into launching an armed boarding of a Chinese cargo ship because of an AI intel program's hallucination earlier this year, according to CNN. In the wrong circumstances, it was the kind of mistake that could have launched a broader conflict; since it was caught in time, it was merely a misuse of resources.
Four sources with knowledge of the episode told CNN that a military intel analyst's report flagged a Chinese cargo ship operating in the Mideast during the height of the Iran conflict. According to this report - formatted like any other - this cargo ship had nuclear weapons components on board. Commanders received and treated the report as though it were accurate, and began preparations to interdict and board the vessel - which, if carried out, would almost certainly spark a significant diplomatic incident with China. The effort got so far along that there were military aircraft aloft and boarding teams getting ready to go for the interdiction.
The report, however, was wrong from the start. An analyst had used an AI chatbot to ask about some intelligence data related to the suspect ship's cargo manifest. The chatbot drew on classified data and open-source intelligence, then made up a conclusion: it decided that the vessel was carrying components for a nuclear weapons program, CNN's sources said. This was false, but the analyst did not detect that; instead, they packaged it up for formal use - again using AI for formatting - and sent it up the chain of command.
that matters most
Get the latest maritime news delivered to your inbox daily.
In the run-up to the interdiction operation, officials examined the analyst's intelligence report more closely and determined that AI had been used to come up with the contents. They figured out that the chatbot had misidentified the Chinese vessel's cargo, and halted the operation before any action on the water occurred.
The Chinese vessel continued on its commercial voyage, and the AI "hallucination"-inspired intercept effort became a cautionary tale rather than a potential flashpoint in an already tense situation in the Mideast. But CNN's sources said that comparable errors had happened before, and warned that there are no standardized, joint guidelines for catching AI mistakes during one of the military's most consequential data processes - target identification and selection.
