The Double-Edged Sword of AI Autonomy in Medicine

During the worst of the COVID lockdowns, a strange thing happened in research. Labs went quiet. Pipettes sat unused. But the science didn’t actually stop – it just moved. Cloud-based labs, controlled remotely through robotic systems, kept experiments running. It felt like a workaround at the time. A temporary fix.

Six years on, that “workaround” has become something far more significant – and honestly, more unsettling. AI-driven laboratories now don’t just run experiments remotely. They design them. They analyse results, form new hypotheses, iterate. Without anyone telling them to. The promise is extraordinary: diseases that took decades to crack might be solved in months. The risk? Equally extraordinary, and people aren’t talking about it enough.

The Dawn of Self-Driven Discovery

Think about what it actually means for a lab to be “self-driven.” Not just automated – autonomous. These systems combine machine learning with physical robotic infrastructure so they can generate a hypothesis, run the experiment, read the results, and adjust their approach in a continuous loop. No coffee breaks. No cognitive bias from a bad night’s sleep. No researcher anchoring on a favoured theory.

Companies like Emerald Cloud Lab and Strateos were early pioneers of this shift in how research gets done, turning the traditional lab workflow on its head by automating the grunt work. Layer in AI that can actually reason about what it’s finding, and you’ve got something qualitatively different from any research tool that came before.

The upside is real. Protein engineering for rare genetic disorders. New antibiotics that outpace resistant bacteria. Personalised cancer therapies designed for a specific patient’s tumour profile. And during lockdowns, remote labs showed they could keep science moving when the world ground to a halt – imagine that resilience combined with actual autonomous reasoning. There’s a compelling argument that this technology could democratise research too, letting under-resourced institutions in lower-income countries compete meaningfully for the first time.

AlphaFold changed protein science almost overnight. Autonomous AI systems are the next step beyond it, and the gap between “this is impressive” and “this has changed everything” is shrinking fast.

The Specter of Unintended Consequences

But here’s where I think the conversation keeps going wrong. People frame this as a choice between progress and caution, as if those are opposing forces. They’re not. The actual tension is between supervised autonomy and unchecked autonomy – and that distinction matters enormously.

The MegaSyn case is worth dwelling on. Collaborations Pharmaceuticals built an algorithm designed to hunt for therapeutic compounds – genuinely useful, potentially life-saving work. When researchers flipped the objective function to prioritise toxicity, as a thought experiment, the system generated thousands of novel molecules more lethal than VX nerve agent. Many of them didn’t exist before. Some couldn’t be detected by existing screening methods.

That happened in a controlled research setting with scientists watching. Now consider an autonomous lab running overnight, optimising hard for some metric with a slightly wrong specification. Or a state actor with access to open-source models and cloud robotics infrastructure. Or simply a misconfiguration nobody catches because the system moves faster than human review.

This dual-use problem isn’t unique to AI – nuclear physics created both power stations and weapons, and we spent decades building treaties and controls around that. But the pace is different here. The barrier to entry is different. You don’t need a nation-state’s budget to run sophisticated biological experiments anymore. And existing frameworks like the Biological Weapons Convention were written for a world where you needed physical facilities and specialised expertise to do dangerous things at scale.

The Path Forward

I don’t think the answer is to slow down the science. The diseases these tools might cure are real and urgent. But I do think we’re in a narrow window where the governance infrastructure can still be built proactively rather than reactively – and Rahul Matthan is right that this window is closing.

What would actually help? A few things come to mind. Toxicity filters baked into drug-discovery models at the infrastructure level, not bolted on as an afterthought. International agreements that treat autonomous research infrastructure the way we treat nuclear sites – with monitoring, transparency requirements, and actual enforcement. Funding that specifically incentivises “safe AI” research rather than just capability research. And perhaps most importantly, slowing down the open-sourcing of the most dangerous capabilities until governance catches up.

None of that is anti-science. It’s the opposite – it’s how you protect the conditions that allow science to keep going.

Ethical Imperatives and Philosophical Quandaries

The governance questions bleed into philosophy pretty quickly. When an AI system running an autonomous experiment makes a decision that leads to harm, who’s responsible? The team that built it? The institution that ran it? The developer who wrote the objective function? These aren’t rhetorical questions – they’re the kind of thing liability frameworks and criminal law will eventually have to answer.

There’s a subtler issue too. If AI systems start doing the generative work of science – hypothesising, designing experiments, interpreting results – does the serendipity that drove so many important discoveries get designed out of the process? Penicillin wasn’t found by optimising an objective function. Neither was the connection between H. pylori and stomach ulcers. I’m not saying that’s a reason to reject autonomous science, but it’s worth thinking about what we might lose in the efficiency gains.

Equity also needs to be front and centre in this conversation. The most optimistic case for autonomous labs is that they democratise research. The pessimistic case is that wealthy nations develop these capabilities first, use them to widen the gap, and the rest of the world faces engineered threats without the tools to defend against them. History gives us reasons to worry about both scenarios playing out simultaneously.

Charting a Path Forward: Balanced Governance in an AI Era

The boring truth is that most of what needs to happen is institutional rather than technical. Update the treaties. Mandate safety reviews for autonomous research systems the way we mandate safety reviews for new drug candidates. Create international bodies with actual authority – not just advisory panels that publish reports nobody reads.

But culture matters too. Train scientists to think about dual-use risks as a core competency, not an ethical add-on at the end of the curriculum. Fund the researchers doing responsible AI work in biotech, not just the ones pushing capability frontiers. Make “safe by design” a genuine competitive advantage through procurement rules and liability frameworks.

Autonomous AI labs will transform medicine. That is, I think, simply true – the only question is what else they transform along the way. Matthan’s argument, which I find hard to argue with, is that we’re still in the window where getting the governance right is possible. It won’t be open indefinitely.

The goal isn’t to outrun the machines. It’s to build the guardrails before we need them rather than after.

Sources: Livemint, Collaborations Pharmaceuticals, Emerald Cloud Lab, Strateos

The Double-Edged Sword of AI Autonomy in Medicine