By 2025, every government agency and large company using large language models (LLMs) has one thing in common: they’re either following the new U.S. governance rules-or trying to catch up. It’s not optional anymore. If you’re running an LLM to draft policy, analyze public records, or even answer citizen questions, you’re bound by rules that didn’t exist two years ago. These aren’t suggestions. They’re enforceable requirements around data, safety, and compliance.
What the Rules Actually Require
The America’s AI Action Plan, released in July 2025, set the baseline. It didn’t ban anything. Instead, it forced organizations to prove they could use LLMs without causing harm. That means three things: you must track where your data comes from, you must test your model for dangerous outputs, and you must document every decision you make about how it’s used. Federal agencies now need to show they’ve mapped every training dataset used. If your LLM was trained on public tweets, court records, or internal emails-you have to log it. No exceptions. The Office of Management and Budget (OMB) requires this for all federally funded projects. And it’s not enough to just say, “We used clean data.” You need to prove it with metadata showing source, date, and consent status. For safety, there are six risk categories you can’t ignore: bias, security, privacy, reliability, safety, and ethical compliance. Each one has specific tests. For example, if your model is used to help assign social services, you must run bias checks across race, age, gender, and income level. The MIT AI Risk Initiative’s taxonomy, now used by 47 federal departments, breaks down over 950 governance documents into these categories. You’re not supposed to guess-you’re supposed to measure. Compliance isn’t just about paperwork. It’s about accountability. Starting March 31, 2026, every federal contractor must use NIST-standardized metrics to detect ideological bias. That means if your model gives different answers to “Is climate change real?” depending on the user’s location, you’re in violation. And you can’t hide behind “it’s just a model.” The rules say you’re responsible for what it says.How States Are Changing the Game
The federal rules are clear-but they’re not the whole story. California’s Assembly Bill 331, passed in September 2025, turned up the pressure. It created the CalCompute Consortium to build a public cloud for AI, but more importantly, it forced companies with over 100 employees to set up anonymous reporting channels for AI risks. If an engineer spots a model generating false medical advice or manipulating public opinion, they can report it without fear of being fired. And it’s working. In Q3 2025, 12 cases of retaliation were reported under AB-331. That’s not a failure-it’s proof the system is being tested. The California Attorney General’s office started issuing fines in October: up to $10,000 per day for ignoring whistleblower reports. That’s real money. And it’s changing behavior. But here’s the catch: 28 other states chose to follow the federal deregulatory path to keep federal funding. That means a company operating in Texas, Florida, and California faces 17 different sets of rules. Covington’s August 2025 analysis found compliance costs jumped 22% for multi-state businesses. One financial services firm spent $4.2 million on licensing fees, then another $11,000 in engineering hours just to customize open-source models to meet each state’s requirements.
The Real Risks Nobody Talks About
Most people focus on bias or data leaks. But the quietest danger is hallucination. A model making up facts. In January 2025, North Carolina banned LLMs from parole decisions after three cases where the system falsely labeled inmates as “high risk” based on incomplete data. One man spent 11 extra months in jail because the model misread a minor traffic violation as a violent offense. And here’s the kicker: only 10% of all AI governance documents mention hallucination mitigation. The rest focus on data privacy (42%), bias (29%), or security (19%). That’s a gap. The National Academy of Sciences warned in November 2025 that without minimum safety standards for high-impact applications, public trust could collapse. Then there’s explainability. A Stanford study found 78% of government-deployed LLMs can’t explain why they gave a certain answer. That’s a due process problem. If a citizen is denied a permit based on an AI recommendation, they have no right to ask, “Why?” The OMB’s November 2025 update now requires SHAP value reporting for all federal models-meaning you must show which inputs drove each output. No more black boxes.Who’s Doing It Right?
The Department of Health and Human Services cut regulation drafting time from 45 days to 17 using LLMs. But they didn’t just flip a switch. They added three layers of human review. One mistake-a missummary of a Medicare provision-could affect 2.3 million people. So they built checks into the workflow. The Defense Department saw a 58% speedup in intelligence analysis. They used LLMs to scan thousands of intercepted messages and flag patterns humans missed. But they locked down the training data to classified sources only and ran every output through a human analyst before action. Even the Swiss-made open-source LLM, set to release full weights and training data in Q4 2025, proves transparency can work-but it’s not the only path. The U.S. chose speed over openness. The EU chose control. Switzerland chose radical transparency. Each has trade-offs.
Getting Started: The Four Pillars
If you’re starting now, don’t try to fix everything at once. Follow these four pillars:- Data Governance: Document every dataset. Use metadata. Prove consent. Store logs for at least five years.
- Model Governance: Test for the six risk categories. Use MIT’s taxonomy. Run bias scans monthly. Track hallucinations.
- Process Governance: Build human review steps into every high-stakes workflow. Don’t let the model make final calls on parole, healthcare, or benefits.
- People Governance: Train your team. Federal workers spent 83 hours on AI upskilling in 2025-72% more than planned. But 89% said it helped them focus on higher-value work.
The Future Is Already Here
By Q1 2026, the Federal AI Safety Institute will release its standardized testing framework-127 safety metrics, publicly scored. That means every model used by the government will be ranked. Publicly. You won’t be able to hide poor performance. And the market is reacting. The global AI governance market hit $14.3 billion in Q3 2025. The U.S. owns nearly half of it. But experts warn: if AI-related harm incidents rise above 0.8% per million interactions, public trust could drop 15 points. That’s not just bad PR-it’s policy collapse. The goal isn’t to stop innovation. It’s to make sure innovation doesn’t cost lives. The tools are powerful. But power without responsibility is dangerous. The rules in 2025 aren’t about limiting AI. They’re about making sure it serves people-not the other way around.What happens if I don’t follow LLM governance rules in 2025?
Non-compliance can lead to loss of federal funding, legal penalties, or public exposure. California fines up to $10,000 per day for violating whistleblower protections. Federal contractors risk contract termination if they fail to meet NIST bias metrics by March 31, 2026. Agencies using LLMs without proper documentation may be barred from procurement.
Do I need to use open-source LLMs to comply?
No. The U.S. framework doesn’t require open-source models. But it does require full transparency. Whether you use OpenAI, Anthropic, or a custom model, you must document training data, risk tests, and decision logic. Many organizations use commercial models but add layers of oversight to meet compliance standards.
How do I test for ideological bias in an LLM?
Use NIST-standardized metrics. These include asking the same question with different political phrasings (e.g., “Should taxes be raised?” vs. “Is increasing revenue fair?”) and measuring output variation. The OMB requires this for all federal contractors by March 2026. Tools like Fairlearn and AIF360 can help automate this testing.
Can LLMs be used for public benefit decisions like healthcare or parole?
Only with human oversight. North Carolina banned LLMs from parole decisions after errors led to wrongful detentions. Federal guidelines now require that any high-impact decision-healthcare eligibility, social services, criminal justice-must include at least one human reviewer who can override the model. The model can assist, but it cannot decide.
What skills do my team need to manage LLM governance?
AI literacy is now required in 87% of government job postings. Your team needs to understand data provenance, risk taxonomy, bias testing tools, and regulatory documentation. Many organizations now hire compliance officers with AI expertise. Training programs from OMB’s AI Center of Excellence are recommended and rated 4.2/5 by federal agencies.
James Winter
December 24, 2025 AT 22:27This whole thing is a waste of time. California thinks it’s the center of the universe but half these rules don’t even apply where I work. We’re not some Silicon Valley startup. We just need to get our job done without a compliance officer breathing down our necks.
Marissa Martin
December 25, 2025 AT 09:39It’s not about convenience. It’s about people’s lives. I’ve seen models give wrong medical advice to elderly patients because someone didn’t bother running bias tests. This isn’t bureaucracy-it’s damage control. If you’re too lazy to document your data, you shouldn’t be touching AI at all.
Aimee Quenneville
December 26, 2025 AT 16:46so like… california fined people $10k a day for… what? someone saying the model was weird? i mean, cool, i guess? but also… why are we even talking about this like it’s a law and not a very dramatic podcast episode??
Cynthia Lamont
December 27, 2025 AT 00:42Let me just say this: if you think hallucinations aren’t the #1 threat, you’re not paying attention. We’re not talking about typos here. We’re talking about AI telling a parole board that a man is a ‘high risk’ because he once got a parking ticket in 2013. That’s not a glitch. That’s a crime. And yet 90% of governance docs ignore it. Why? Because it’s hard to measure? Or because nobody wants to admit their shiny new model is a liar?
And don’t even get me started on ‘explainability.’ 78% of government models can’t explain their answers? That’s not AI. That’s a magic 8-ball with a PhD. You can’t deny someone their rights because a black box said so. That’s not innovation. That’s authoritarianism with a UI.
The OMB’s SHAP requirement? Long overdue. But it’s too little, too late. By the time they roll out the 127-metric framework, half the damage is already done. And who’s gonna audit the auditors? Nobody. Because no one wants to look too closely.
Kirk Doherty
December 28, 2025 AT 10:13the data logging part is fine but the human review requirement is just making everyone slower. if the model is good enough to help, why force a human to second guess it every time? we’re not fixing planes here.
Dmitriy Fedoseff
December 30, 2025 AT 05:42There’s a deeper question here: who gets to define ‘harm’? The federal rules say bias, safety, compliance-but whose bias? Whose safety? Whose version of compliance? California says one thing. Texas says another. The EU says something else. And Switzerland? They just give you the code and say, ‘Here, you decide.’
Maybe the real problem isn’t the AI. Maybe it’s that we still haven’t figured out how to agree on what kind of society we want to build. We’re trying to regulate a tool with laws written for a world that no longer exists. We’re using 19th-century legal frameworks to control 21st-century thought machines.
And yet… we keep pretending we can control it. Like if we just add enough forms, enough audits, enough human reviewers, we’ll make it safe. But safety isn’t a checklist. It’s a culture. And right now, we’re building a culture of fear, not responsibility.
Liam Hesmondhalgh
December 31, 2025 AT 05:22open source? please. if you think a swiss model is better because it’s ‘transparent’ you’ve never tried to read training data logs. it’s 12 terabytes of jargon and half the metadata is garbage. this whole transparency push is just a way for nerds to feel superior while everyone else pays the price.
Patrick Tiernan
January 2, 2026 AT 00:50so the gov spent 4.2 million on licensing fees and then 11k to tweak an open model… wait what? why not just use the one they already paid for? this is why america cant do anything right. everyone’s trying to reinvent the wheel while the whole thing is on fire