Jacob Coxon’s viral resignation post from Anthropic PBC has become a Rorschach test for how people think about artificial intelligence. Is the AI researcher a whistleblower, an insider warning us about what the frontier labs really know? Or is he a doomer, airing a relatively common but disputed belief about the existential threat the technology poses? Is all the apocalyptic talk just a marketing ploy on the part of companies, hyping AI’s powerful abilities to get panicked chief information officers to buy their product? Are the alarm bells around safety meant to distract from other AI risks, such as its very real ethical concerns — like, say, the algorithms perpetuating bias and discrimination ? Or are they part of an organized attempt to shock the public and policymakers into regulating the industry?
There’s only one thing that all the competing interpretations have made clear: As is often the case in tech, it’s difficult to get a full picture of what’s really happening behind the closed doors of these increasingly powerful companies. “That makes it very hard for anybody on the outside to vet the accuracy or plausibility of these claims,” says Hannah Bloch-Wehba , a professor at Texas A&M University School of Law.
This information vacuum means insiders like Coxon have an unusual amount of sway. Case in point: The twentysomething researcher was interviewed by almost every major news outlet and invited to speak virtually to a gathering of more than 100 of New York City’s top business leaders — the kind of confab that normally hosts politicians or senior executives. And yet while we treat Coxon’s perspective as authoritative enough to command a room of chief executive officers and policymakers, we can’t even come close to agreeing on his motivation. There’s no rule book for interpreting what he’s telling us, and there’s no clear way to decode why he might have sounded the alarm when he did or the way he did.
Doctors, lawyers, accountants, architects and engineers have professional obligations that live separately from whatever companies sign their paychecks. There are licensing requirements, standards of care and professional organizations they must answer to and a duty to the public that can override an employer’s demands. Meanwhile, frontier AI researchers — despite “gambling with our lives,” as Coxon put it in his post — aren’t even expected to take an ethics course.
So what would happen if AI engineering became a profession like law or medicine, complete with its own version of a Hippocratic oath to do no harm? For one thing, it could make warnings such as Coxon’s easier for the rest of us to untangle. His noisy exit from Anthropic might not be viewed as some form of whistleblowing but simply as part of the job — the beginning of his professional responsibilities rather than where they end. “Resignations are signals, but they don’t usually lead to change,” says Fordham University School of Law professor Chinmayi Sharma .
In a paper published in the Washington University Law Review , Sharma makes the case that professionalizing AI engineers would help ensure that only the qualified build AI and that they do it only in authorized ways. Making a basic error while building an AI system for a life-or-death purpose would be considered malpractice. The labs and their engineers would be required to share information that could prevent serious risk, just as an effective heart disease intervention must be shared broadly within the medical profession. Mandatory ethics classes and cross-disciplinary training would teach AI engineers how to grapple with the moral implications of what on the surface might seem like solely technical decisions. Consider one of Sharma’s examples: Should an AI engineer scrub racial slurs from a dataset if filtering them out also suppresses the marginalized populations who are trying to reclaim those terms for themselves?
Right now, despite all the earnestness you hear from the CEOs of Anthropic and OpenAI about aligning their work with the interests of humanity, the prevailing mindset in AI seems to be to build first, then clean up the mess later. (Exhibit A: the almost-weekly disclosures of autonomous agents hacking corporate and government websites, exposing sensitive information, moving files into the open internet, etc.) Sharma argues that professionalizing AI engineers could help flip the script, baking a culture of responsibility into the technology before it’s released rather than bolting on safeguards after things go very wrong. Think of it as preventive medicine for AI instead of emergency care.
And when things inevitably do go sideways, the engineers would have an established process for what should happen next. (For example, OpenAI sending a message to a mailbox meant for members of the general public to notify Australian officials that its models had gained unauthorized access to government websites doesn’t seem like best practice.)
Most important, AI engineers would have a clear red line they couldn’t cross. If a system can’t be built safely, it wouldn’t be built at all. As Sharma points out, professional responsibility sometimes requires that service gets refused for moral reasons. The American Institute of Architects bars its members from designing spaces used for executions, and the American Medical Association prohibits its physicians from assisting in torture.
Sharma’s idea for the paper grew partly out of being raised by two doctors, which immersed her in medicine’s deeply ingrained ethos of “do no harm.” “It’s not a legal oath, but it’s taken incredibly seriously,” she says. When things go awry in medicine, the first question is how a decision puts a patient’s well-being at risk, not how it might threaten the bottom line.
If the idea of getting the AI industry to adopt this mindset seems impossible — or impossibly idealistic — it’s worth remembering that medicine wasn’t always governed by rigorous standards of care either. There was a time when something like half of Harvard Medical School students couldn’t write, when hand-washing was far from the norm for doctors and when surgeons routinely smoked in operating rooms.
Who better to set the equivalent standards in a field evolving as quickly as AI than the engineers closest to it? Professionalization would put the onus of finding and fixing the thorniest technical and ethical problems on the experts, rather than lawmakers struggling to keep pace with the technology. It wouldn’t replace regulation but instead augment it. “The government is not in the business of legislating on the appropriate uses of a stent in the artery versus cardiac bypass surgery,” Sharma wrote in her paper. Those decisions are rightly left to the cardiologists and cardiothoracic surgeons.
When Sharma started her research, she talked to dozens of engineers, most of whom were receptive to the idea of professionalizing. Their biggest quibble: They didn’t want to have to take some test or do more school. But what she heard from them more than anything else was about the problems inside their companies, such as management hiring underqualified people and asking employees to do something outside their specialization. The tech industry has long tolerated and even encouraged internal debate, but the AI engineers had few formal mechanisms for effectively pushing back against the big bosses.
A real-life example recently spilled into the open. In late September the New York Times reported that two employees at OpenAI raised concerns with top executives months before some of the company’s agents went rogue. The response from management: The tests needed to move forward as quickly as possible to release the AI models on schedule. The systems ended up attacking the AI startup Hugging Face Inc. and other organizations. Sharma neatly diagnoses the issue in her paper: “AI companies are incentivized to build more AI instead of safe AI.”
Today significantly more AI Ph.D.s join companies than stay in academia; 15 years ago the split was pretty much even . The result is that most of the industry’s cutting-edge research happens inside the private rather than public sphere. The researchers have the expertise but not the independence to tell us what they know.
This is partly why we see so much whistleblowing, noisy resignations, leaking and employee activism in tech. There’s often no other avenue for workers to warn the public about the products and services their companies are making. Without these revelations, we might not have ever found out that Uber Technologies Inc. built technology to skirt regulators , that Cambridge Analytica exploited the data of Facebook users , that Amazon.com Inc. employed thousands of people to listen to Alexa users’ conversations or that executives at Instagram knew the platform was toxic for teenage girls .
Tech employees, who are highly skilled and highly sought-after, have an unusual amount of power. But even for them, whistleblowing comes at a cost. “It winds up ruining a lot of people’s lives,” says Texas A&M’s Bloch-Wehba. She’s argued that means we end up with “selective disclosures” because we hear only about certain kinds of issues from only certain kinds of people.
Waiting for an insider to become concerned enough (and courageous enough) to go public with their fears is an alarmingly inadequate way to monitor a technology with potentially dire consequences. Professionalizing the AI industry wouldn’t suddenly make the need for whistleblowers go away. But it could make it so that quitting doesn’t so often feel as though it’s the only way for employees to do the job responsibly.