The debate about lethal autonomy—core to the Trump administration’s fight with Anthropic—obscures a deeper danger of the Pentagon’s rapid adoption of commercial AI tools: they might weaken the U.S. military’s ability to tell fact from fiction.
New research suggests that relying on AI to do various tasks can erode one’s native ability to do them. Military commanders are taking note.
“The more you use AI, the more you will use your brain in a different way,” said French Adm. Pierre Vandier, NATO Supreme Allied Commander for Transformation, in a conversation in Defense One’s “State of Defense” series. “And so [we need to] be able to have some oversight, to be able to critique what we see from AI, and to be sure you are not fooled by a sort of false presentation of things. It’s something we need to take care of.”
But there is scant evidence that the Pentagon, in its rush to deploy AI tools, is taking steps to keep its users sharp—or even to monitor the effects of AI.
“Because of the pace and the urgency associated with deploying these models, maybe that hasn’t been put in place,” said one former senior military official who worked to deploy AI in combat settings.
In the meantime, the pressure to use AI tools is only growing.
“Especially as you get deeper into any conflict, there will be more and more pressure to find more targets. That happens in every conflict as well. After two weeks, you’re sort of going through your delivery target list. Now you’re demanding targets. Well, how fast can you generate targets?” the former senior military official said.
A cognitive trap
A growing body of research shows that wide use of large language models can undermine human thought and communication.
For example, it can homogenize thinking among users, reinforcing “dominant styles while marginalizing alternative voices and reasoning strategies,” according to an Air Force Research Laboratory paper published earlier this month in the journal Cell.
This presents at least two problems for the military, said Morteza Dehghani, a University of Southern California computer science professor who helped write the paper. In the moment, “it washes away signals about who the author is,” and therefore eliminates important context for evaluating data. And over time, it can stifle critical thinking.
“Because these models are optimized for the most likely and ‘idealized’ responses, they often enforce a linear, ‘Chain-of-Thought’ reasoning style,” Dehghani wrote in an email. “This can disincentivize experienced analysts from employing the non-linear, intuitive, or ‘gut feeling’ strategies that are essential for identifying rare exceptions or navigating complex, non-standard intelligence scenarios.”
Similarly, researchers at Wharton found in January that people using LLMs spend less and less time scrutinizing results for accuracy or employing their own judgment. They found that users rely on the AI’s judgement even when they know it’s wrong, a phenomenon the authors call “cognitive surrender.”
And a February paper from Princeton found that the way LLMs speak to users—referred to as “sycophantic AI”—instills a false sense of confidence and isolates people within preconceived biases: “Our results show that the default interactions of a popular chatbot resemble the effects of providing people with confirmatory evidence, increasing confidence but bringing them no closer to the truth.”
Who is driving?
The military is aware of at least some dimensions of the problem. Speaking on a March 6 podcast, Pentagon research-and-engineering chief Emil Michael said that his core concern about Anthropic—which President Trump has barred from use by the federal government—was that military users might become too dependent on a single untrustworthy tool.
“Maybe it’s a rogue developer who could poison the model to make it not do what you want at the time, or sort of trick you because you have to trick it. I mean, all these things that we know when we worry about models that hallucinate purposefully or do not follow instructions,” Michael said.
Anthropic had similar concerns: that its tools might be used by people untrained to properly evaluate its output. One company official said their chief worry was that they had not validated that the model could be used reliably for compiling targeting lists. Worse, they had no way of knowing how the military was putting their tools to use. For example, Anthropic officials only learned after the fact that their tools had been used to plan the Jan. 3 raid into Venezuela.
The Pentagon has since declared Anthropic products a supply-chain risk and is working to replace them, though they are still in use by military planners, including in the Middle East.
A former senior official described it as “a serious governance question.” Frontier AI companies like OpenAI, Anthropic, Google, and xAI cannot provide the in-depth support required for military units to understand the conditions under which these tools are used.
“It’s almost becoming a journey of discovery for the government,” the former senior military official said. “Are there people on site from these companies helping the day-to-day user? My guess is, if there are, there may be only one or two of them.”
Read the full article here
