US military had close call after using AI for false intelligence report, sources say

https://www.cnn.com/2026/09/18/politics/us-military-ai-false-intelligence-china-ship

So this is now a story how AI errors could have disastrous consequences if there are no safeguards between AI generated analyses and subsequent action.

The US military swung into action with plans to intercept the vessel, according to four sources familiar with the episode. According to two of the sources, armed members of the US military were preparing to board the ship. Military planes were in the air, one of those sources and another source familiar with the incident said.

It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence (AI) — and that a chatbot the analyst had used inaccurately identified the material the ship was carrying. CNN was not able to learn what the misidentified cargo was.

The report, according to one of the sources, was “entirely false.” But it also “almost started a war,” the source said. Any US operation against a Chinese vessel could have risked spiraling into an armed conflict between the two nations.

[...]

For some older intelligence officials — even those who broadly support the use of AI inside the military — AI has put pressure on analysts to produce and disseminate intelligence faster, opening the door for mistakes. Young analysts in particular, several sources said, are natives on these tools and more likely to trust them uncritically.

This follows a host of other pushes to use AI-based methods in e.g. law enforcement and other areas which will likely already have real-life issues. The big question here is what kind of safeguards would be feasible in the first place and would folks be willing or able to implement them?