The Pentagon’s Dangerous Adoption of AI – Emma Isabella Sage

Despite the immediate availability of multiple competing models with fewer restrictions, the Pentagon went straight for the nuclear option: threatening Anthropic with a supply chain risk designation. This penalty would effectively force anyone who works with the DOD to cut ties with Anthropic. It is a completely abnormal use of the designation, which is meant to identify truly dangerous (and almost exclusively foreign) companies, not those that place inconvenient restrictions on their products.

Anthropic has a reputation as one of the most ethically aware AI companies, and its model Claude is considered by many to be the best on the market. And while Anthropic was gearing up to do battle with the government on behalf of Claude’s good name, a different model with the opposite reputation went live on Pentagon servers. 

The integration of Grok into military systems should have set off alarms across Washington as one of the clearest signs that the “adults in the room” were no longer calling the shots. Elon Musk’s pet project has been embroiled in multiple serious scandals, from its brief obsession with “white genocide” in South Africa to identifying itself as “MechaHitler” to the explosion of nonconsensual pornographic content generated from images of real people, including minors. In retrospect, these scandals aren’t entirely surprising: Grok has been explicitly positioned as less constrained, more reactive, and more radical in its “opinions” than its competitors. Its training, safeguards, and system instructions are shaped by its market positioning as the maverick model, making it the worst possible choice to be involved in decisions with human lives at stake. 

But while Grok is particularly problematic, no frontier, consumer-oriented model can be safely or sanely integrated into classified military environments without violating essential principles of safety.

Commercial Large Language Models (LLMs) are trained on vast datasets of, essentially, whatever AI companies can get their hands on. That’s not the way to produce a model virtual government employee, unless we want the aggregated wisdom of Reddit comments influencing the actions of the United States Armed Forces, but AI’s need for ever more training data creates other problems.

The vulnerabilities of LLMs have been long and widely known, but it is becoming increasingly clear that these models (including, but not limited to, Grok) are being subjected to data poisoning attacks on a massive scale, the most successful of which are attributed to the Russian-linked Pravda network. We know this is happening, and we know it works, but we cannot easily stop it, nor can we predict with confidence the precise effect it will have on model behavior. This tactic already poses a huge disinformation risk in the public-facing versions of the models, but it’s impossible to overstate the degree to which the problem is magnified when these models are brought into military contexts.

There is also an inherent clash between the way government works and the very nature of generative AI. These are not accountable tools, and government must be accountable to the public. The Pentagon is deploying, in adversarial conditions, models whose internal logic cannot be meaningfully directed, explained, or trusted. AI in its current form comes with a fragility and a literacy gap that makes it a potential liability from the office to the killing fields of eastern Ukraine, and there is no way to know exactly how it is currently being put to use at the Defense Department.

Therefore, these integrations raise serious questions regarding the future of government transparency and trustworthiness. Government work is subject to FOIA, audit, litigation, and oversight. If an AI-assisted assessment informs policy decisions, and later collapses under scrutiny, who is responsible? People can go to jail, but there are no repercussions for a chatbot. 

No amount of fine-tuning or cybersecurity measures can resolve this: The core of the thing itself is incompatible with the role it has been given. In high-risk domains, relying on this type of AI is dangerously negligent. 

I’m currently undertaking research that uses deep learning in the pursuit of national security goals, so I’d be the first to make the case that there is a legitimate path for AI in defense. Here’s what it looks like: purpose-built models that are narrow in scope, trained on controlled data, rigorously tested, and auditable. Models for high-risk use cases should be built differently from day one, sacrificing speed, creativity, and “wackiness” for reliability. 

Instead of going this route, the government is choosing commercial models which have been optimized for people-pleasing, not accuracy. But the Department of Defense is not a venture capital-backed startup; it does not get to “move fast and break things.” When the Pentagon adopts a system, reversing course later is like trying to turn an aircraft carrier—or, as one senior official recently put it, “an enormous pain in the ass.” Something usually has to go massively, publicly wrong for such a lumbering institution to unwind integrations. It is unbelievably rare that corporate ethics constrain government behavior, as opposed to the other way around—and that alone should bring enormous scrutiny to bear on the current administration’s current spat with Anthropic. 

Sane policy on AI in government requires realism about what this technology can and should do. This means resisting the temptation to treat AI as a sort of universal lubricant for the grinding gears of a complex bureaucracy. It means doing an actual problem assessment by talking to Pentagon employees about what sort of integration would reduce the menial tasks that unnecessarily fill their time. It means understanding that government contracts convey legitimacy, but that the street goes both ways: The more the government associates itself with the most disreputable corners of private industry, the more trustworthy these unreliable sources will seem to some, but the less trustworthy the government will appear to others.

If Washington wants to maintain the confidence of its own national security employees, it must recognize that this community judges systems not by how cutting-edge or exciting they appear, but by how well they perform under pressure and how resilient they are under fire. The current danger is not that AI will become self-aware or go rogue and fire off the nukes. We’re not there yet. The more likely scenario is that taxpayer money is now underwriting a system that worsens military groupthink and gives adversaries a backdoor into the policy of the Pentagon.