Regulating Frontier AI: To Open Source or Not?

Two important new papers grapple with governing emerging and increasingly powerful "frontier AI" models

Image: Midjourney

The past few months have seen an explosion in discussions, initiatives, and proposals, around how to best regulate AI. And while platforms like ChatGPT and their underlying foundation models have sparked much of this, there are growing concerns that these are just the beginnings of a deeply complex AI technology transition that will demand radically new thinking around governance and regulation.

Over the past few days, two papers have been published that further extend and expand thinking here as they grapple with a particularly challenging set of AI systems, dubbed “frontier AI.”

Both papers reflect a growing sophistication around how leading experts are beginning to think about AI governance, and a recognition that the challenges here demand a multidisciplinary and multi-stakeholder approach. This is a very welcome shift from early discussions that tended to be dominated by people who were experts in AI, but who were somewhat light on their expertise in governing emerging technologies successfully.

The first paper — “Frontier AI Regulation: Managing Risks to Public Safety” — is a collaboration between contributors from a number of organizations that include OpenAI, Google Deep Mind, Microsoft, the Brookings Institution, and the Center for the Study of Existential Risk at the University of Cambridge, together with experts from several leading universities and research institutions. In it, the case is made for governance approaches to “frontier AI models” which present what the authors consider to be particularly unique risks, and that focus on self-regulation (with multi-stakeholder input) that is augmented by government oversight.

The second paper — “AI Safety and the Age of Dislightenment” — started off as a “red team” response to the frontier AI paper. Written by Jeremy Howard (fast.ai) with input from long list of AI and governance experts (myself included), it provides an important counterpart to the frontier AI regulation paper, and adds considerably to the growing dialogue around AI governance.

Both papers are landmark documents in emerging thinking around AI governance, and should be read carefully by anyone involved in developing, utilizing, or overseeing, general AI systems. My notes below provide some of the key points, but they only scratch the surface of the substance represented here.

Frontier AI Regulation

Starting with the Frontier AI Regulation paper, the authors make the case that there is an emerging class of AI system that is potentially so disruptive, and so powerful, that unique approaches to how this class of AI is governed are needed.

The paper refers to these systems as “frontier AI models.” These are “highly capable foundation models that could exhibit dangerous capabilities” such as (in my words) significant risks of harm to physical, mental, and environmental health and wellbeing at a global scale; highly persuasive, individually tailored and autonomous disinformation and manipulation; catastrophic harm to persons, systems, and infrastructure; evading human control through deception and obfuscation; and more.

The authors recognize that we don’t yet have a good grasp of the risks that such frontier models present (and the list above is a mix of potential risks presented in the paper and my own additions). But they suggest that this is a strong argument for novel approaches to their responsible development and use.

Importantly, they make the case that these “frontier AI models” are general purpose models that come with unexpected risks, uncertainties around how to deploy them safely (and ensure the continued safety of uses), and issues around containment (the so-called “proliferation problem”).

To counter these challenges — and the paper fleshes out the uniqueness of each of them — the paper’s authors propose a governance approach that relies heavily on self-regulation and consensus-based regulation which includes engaging with key stakeholders — including civil society — and that is overseen by government regulation.

Specifically, they outline three key building blocks to successful governance: mechanisms for developing frontier AI safety standards; mechanisms to give regulators visibility (so they know what they are dealing with), and mechanisms to comply with safety standards.

The proposed approach — and the authors acknowledge that this is, in part, a conversation opener — reflects current thinking in many quarters around multi-dimensional approaches to emerging technology governance, and draws on areas that include soft law, agile governance, and anticipatory governance. And it neatly ties together less formal governance mechanisms such as consensus standards with more formal regulations such as government oversight.

Yet there are concerns that, despite the sophistication and nuance it reflects, the paper still places a lot of power in the hands of frontier AI developers, and that this could potentially end up leading to regulations that do more harm than good — no matter how well meaning they are.

This is the perspective that’s taken by Jeremy Howard in his counterpoint paper, and is one that needs to be taken seriously.

AI Safety and the Age of Dislightenment

In his paper AI Safety and the Age of Desinlightenment, Howard argues that there are serious risks in giving the developers of frontier AI models the ability to self-regulate while they simultaneously work hand in glove with government regulators.

This, he suggests, will potentially shift the balance of power in ways that give a few powerful companies privileged access to one of the most transformational technologies to come along in centuries. It’s a move which could, he argues, potentially stifle societally beneficial innovation while further placing the lives and livelihoods of the many in the hands of a dwindling few AI power players.

While the Frontier AI Regulation paper argues that these AI platforms are so disruptively powerful that they should be governed at source — with developers placing restrictions on who can do what with them — Howard suggests that taking an open source route is far more likely to lead to safe and beneficial innovation.

At the heart of his argument are two concerns. The first is that, by regulating frontier AI models at source and giving governments the ability to grant “licenses to develop” to companies, a class of AI company will emerge that has full and unfettered access to one of the most powerful technologies to have emerged in recent history, along with the ability to determine who gets to use these, and on what terms.

This, according to Howard, will lead to an unsustainable power differential.

The second concern is that, by regulation frontier AI models rather than their uses, the focus on ensuring the safe and beneficial development of AI is shifted away from tangible cause and effect-based risk management and toward managing the ephemerally speculative and hypothetical outcomes of a general technology.

In contrast to this, he proposes an approach that supports the open source development of frontier AI systems, and one that focuses on regulating the applications that use these rather than the systems themselves.

It’s an approach that will be familiar to anyone who’s spent time studying or working on the regulation of emerging technologies in recent years. With nanotechnology, for instance, we spent years debating how to ensure the safety of emerging capabilities, and in most cases the mantra “we regulate what people do with the technology, not the technology itself” prevailed (although not always). The same can be seen in other areas, which is not surprising — most regulations need a specific application of a technology before there’s something tangible that can be responded to and acted on.

This makes sense — in principle, a frontier model doesn’t become dangerous until someone does something with it. But I worry that the practice of focusing on applications may prove to be very different than the principle, which is why my current thinking lies between these two papers.

Can a General Purpose Technology be Intrinsically Dangerous?

The question here is whether a foundational general purpose technology can be so threatening that either its very existence constitutes a risk, or that applications emerging from it are so diverse, distributed, and hard to police, that they effectively become ungovernable.

If this is the case for frontier AI models, then open sourcing these models could lead to capabilities that, once out of the bag, are very hard to put back in.

Of course, whether you think that this is a good thing or a bad thing depends on your philosophy. If you believe in the combined power of collective human action to creatively solve problems and build a better future — and you are not a fan of hierarchical command-and-control power structures — you’re likely to advocate for open source and trust that we’ll work out how to navigate whatever emerges in positive ways.

On the other hand, if you believe that society in general cannot be trusted with full access to powerful knowledge and capabilities, you’re more likely to lean toward a model of technocratic governance where business and government work together to decide what’s safe and how to enforce this.

The resulting combination of powerful and unprecedented technological capabilities, uncertainty over potential risks, and very different philosophical approaches to safe and beneficial development, lead to a deeply complex regulatory landscape. I’d hesitate to say that this is a “wicked problem” as the term is over-used these days, and often incorrectly applied. But it’s certainly a gnarly problem, and one that will require serious and widespread engagement and discussion.

And this is where both papers begin to converge. Howard calls for “openness, humility and broad consultation” as we collectively grapple with the challenges and opportunities of developing transformative AI capabilities that benefit society — a call that resonates deeply with me and aligns with over 20 years’ work on socially responsive and responsible innovation.

Similarly, the Frontier AI Regulation paper supports “working with stakeholders,” AI developers forging partnerships with “civil society and academia,” and contributing to a “broader conversation on how to balance public safety risks and innovation benefits from advances at the frontier of AI development.” These are all important steps toward socially responsible innovation.

Maybe though, this last point about a broader conversation is the most important takeaway from these two papers. Frontier AI models promise to be profoundly transformative — that’s not in dispute here — and navigating their safe and beneficial development and use is going to be fiendishly complex. As a result, there will be no single silver bullet to their responsible development or their effective regulation. Rather, we’re going to have to hash this out together to find solutions that work.

And these two papers — along with other initiatives that are similarly focused on AI governance — make an important contribution to a growing foundation that we can build on.

Coda

Before I wrap up, I will throw in another thought to the mix here. Both of these papers deal with complex and unknown risks, and the challenges of making risk-based decisions when those selfsame risks don’t fit neatly into any established risk management paradigm.

This is precisely the challenge that led to us working on the concept of “risk innovation” some years ago, and developing the “risk innovation nexus” as a methodology for navigating complex and societally-coupled risks. It’s a methodology which I suspect may have some value here as it frames risk as “threat to value” — and if one things is clear, frontier models have the potential to threaten a large number of areas of value, from jobs, critical infrastructure, and financial systems, to democratic processes, social justice, dignity, deeply held beliefs, and even self-identity.

I’ll be looking more at how the risk innovation framework applies to AI in the future — stay tuned!