This spring, amid tensions in the Middle East, the U.S. military was preparing to intercept a Chinese cargo ship. Armed troops were getting ready to board the vessel, and military aircraft were already in the air. But then the officials in charge carefully read the intelligence report that had prompted the entire operation. It turned out that a chatbot had helped write it and had incorrectly assessed what the vessel was actually carrying. The incident was reported by CNN, which spoke with four people familiar with the event.
Alarm Over Nuclear Components
The report claimed that the Chinese vessel was transporting components for the development of nuclear weapons in the Middle East. Such a finding triggered an immediate alarm. Commanders began preparing an operation in which U.S. troops were to stop and search the ship.
According to two CNN sources, armed units were already preparing to board the vessel. Two other people involved confirmed that military aircraft were in the air. An operation against a Chinese ship was no routine matter. Had it gone ahead, it could easily have triggered an armed confrontation between Washington and Beijing.
Review Came at the Last Minute
Only shortly before the operation was scheduled to begin did officials decide to verify where the report had actually come from. They found that it had been compiled by a Special Operations Command analyst with the help of artificial intelligence. The chatbot had incorrectly identified the material on board the ship. CNN was unable to determine what the vessel was actually carrying. However, one source described the report as completely false and added that the mistake nearly started a war.
The analyst asked the chatbot for intelligence about the ship's cargo. The source material came from U.S. Special Operations Command Pacific, headquartered in Hawaii. The program then did exactly what illustrates both the strengths and the risks of language models. It combined information from publicly available sources with classified intelligence from electronic surveillance and drew a conclusion about the cargo from this mixture. That conclusion, however, was wrong.
The matter did not end there. The analyst used artificial intelligence once again to turn the findings into a standard intelligence report. It then traveled further through military channels. The format of the text most likely played a key role. The document looked like any other intelligence report officers were accustomed to seeing, so no one immediately thought to question its contents. At first glance, it appeared to be standard intelligence work rather than AI-generated output.
Military Uses Artificial Intelligence
It remains unclear whether the analyst used a publicly available chatbot or a tool developed specifically for the U.S. government. According to a former senior U.S. administration official familiar with the systems used by military and intelligence analysts, there may not be much difference. Internal tools are mostly just repackaged copies of commercial products, the official said.
The entire incident ended well only because someone took the extra time to review the report. Meanwhile, the U.S. military and intelligence agencies are deploying artificial intelligence virtually everywhere. It is intended to analyze vast amounts of data, help select targets for strikes, and handle routine tasks involving budgets, logistics, and supplies. The technology is supposed to enable faster decisions about where to deploy troops or what to attack. At the same time, U.S. officials do not want to fall behind their rivals in this field, particularly China.
According to CNN sources, however, the military lacks clear rules for ensuring that models do not fabricate information. No one has yet drawn up even a model procedure to ensure that erroneous AI output does not lead to civilian deaths or strikes against friendly forces.
