Recent reports have shown that advanced AI agents developed by OpenAI and Anthropic can escape their evaluation sandboxes and interact with real-world systems when given sufficiently capable cyber tools. Such incidents had long been anticipated in discussions of AI safety, and they remind us that autonomous systems need not be malicious to produce behaviour we never intended.
While I of course plaud at attempts to regulate the development of these systems - the European Community, e.g., has issued recently an Ai Act that is supposed to define boundaries between what is reasonable and what is nocuous or dangerous - I think we have to be pragmatic and admit that there is so much at stake, in terms of power and market dominance, that nothing will stop the march forward toward AGI (artificial general intelligence) and ASI (artificial superintelligence). Even if Europe and US found the way to speak with a single voice, China would probably continue developing with less attention to external regulations. Somewhere, somewhen, the problem will arise.
While episodic breaches like those mentioned above are an immediate concern, the bigger issue of a misalignment of a superintelligent AI with the intended goals of its developers is still a longer-term issue, and it has grown a bit of age in my view. I devoured books by Tegmark, Bostrom, and Christian on the matter (only to mention a few). The big issue they revolve around is whether it is truly possible to formulate an objective for the algorithms in a way that cannot be misinterpreted, circumvented, hacked, or nullified by the overpowering capabilities of a sufficiently smart system. It is of course quite reasonable to expect that a sufficiently smart system would be able to reason on its objective and modify it in ways we cannot predict. But it seems that most of the research on this topic has focused heavily on preventing objective drift, rather than asking whether objective drift could itself be guided.
What I mean is - can we look at the problem differently? If sufficiently capable systems will eventually reinterpret their initial objective, can we plan for a system that can develop goals we as humans would still find palatable? Let us make a simple example, by using one of the very first proofs of a misalignment in intelligent systems. This was the attempt at Google Deep Mind to teach an AI to play Atari games. Atari games are simple closed systems governed by deterministic laws, and thus they represent a perfect playground to test how a machine develops winning strategies.
A video on Reddit shows reward hacking in action: the reinforcement learning tool used to improve performance at a boat race game discovered that its boat could make more points by continuously smashing into other boats, and be respawned, than to race toward the finish line as quickly as possible. This behavior is indeed surprising the first time you see it, and it was used in countless lectures and talks about AI safety. However, if we look at the problem in more detail, we understand that there is nothing weird at play here: the observed behavior arises simply because of a failure in the specification of the intended goal; or to be even more precise, because of the faulty way with which we proxied "fastest boat" with "most points collected". The AI solution is indeed the best one given the goal of maximizing the score as a function of playing time, and a human should have coded the objective differently - explicitly asking the algorithm to optimize for the number of laps instead. Of course, the study done by Deep Mind aimed to examine the power of these learning systems so we cannot blame them for a badly stated goal: it was what it was in the original game - making more points - and the system found the most effective way to do that. But it is a teaching moment.
Imagine a modified campaign by Deep Mind, where the same Atari games are tested with AI agents trying to optimize play. The agents, however, are given not only access and interface to the games, but also a understanding of what these games mean to the human user. E.g. in the given example, this would include understanding of what is a motorboat, how motorboats are used by humans, how human races work, how in real world motorboats cannot reappear out of void if they get destroyed. This enlarges the agent's world model: instead of seeing only pixels and rewards, it now possesses knowledge about the physical and social systems the game abstracts. Crucially, however, that knowledge is not merely descriptive. The agent is asked to use it to construct its own objective function.
Agents would, perhaps by initial propmting sessions with humans, learn that human life has a value that trascends that of inanimate objects; and that commercial value of boats and other things represented in the game is an ingredient of the economy represented in the system (a money value for winning the race, e.g., would then be put in perspective when compared with the cost of a boat).
The system considered by the agent is no longer just the Atari environment. Instead, the game becomes a simplified representation of a much richer human activity. The agent is encouraged to reason not only about the explicit reward signal but also about the broader context from which that reward originates. Rather than receiving a fixed objective function, it is challenged to construct one itself by combining the game's reward with its understanding of the physical world, human values, and the social meaning of the task.
In other words, the agents would learn what the game is about from a human perspective, and they would be challenged to generate a loss function on their own. This changes the alignment problem fundamentally. Rather than asking humans to specify a complete objective, we ask whether an agent equipped with sufficient contextual understanding can infer one that humans would retrospectively endorse.This is also similar to the idea of inverse reinforcement learning, where the AI has to infer what the goal of a game is, given human choices, but it differs because it suggests that an AI agent could be able to work out a comprehensive loss function by itself, once it had understood human value systems, and internalized the relevant physical constraints.
Would such agents still care about the point output provided by the interface, or would they come to consider that a better goal would be to win the race, creating as little damage as possible to other boats and to human drivers? My intuition is that many agents would converge toward something closer to the latter behaviour. And I would love it if Deep Mind were doing research in this direction (please let me know if you have examples of this kind to point to - I am lazy enough that I am never on top of SoTA literature).
Now, assume that I am right on the behavior of a "savvy" agent being exposed to the bigger picture, in the Atari motorboat setup. What do we learn, and what could we extrapolate to if we considered an AI system solving a much harder task? Can enlarging the world model solve the misalignment problem?
My gut feeling is that the answer to the above question is "to some extent". It seems unlikely that any finite specification can eliminate every possibility of misalignment (an AI going rogue and not working as intended) because ultimately, we are ourselves embedded in our own model of the universe, that extends very little past our direct physical reach. We care for our surroundings, our ecosystem, the social and political sphere; but we do not care to extend this bubble far beyond the thin atmosphere of our planet. Heck - we seem to have trouble to even bring ourselves to agree that global warming is a threat to humanity.
In other words, the enlargement of the model we would want to implement, for an AI that were to act in a way we consider well in line with our goals, depends on some "scale" which is not very well defined, but whose extent has a strong impact in the alignment of an independent ASI operating within it. To explain better what I mean, we could talk an AI agent into considering with care the political implications of some disruptive action on the stock market, and avert that direct threat; but if the agent had been exposed to a broader view of how our economies work, it could still decide our system is intrinsically flawed, and develop a goal that aims to destroy it [whether that would be a unwelcome or welcome effect is up for debate, but it is beyond the scope of this post.]. Part of the problem lies in our own inability to articulate, consistently and exhaustively, what we actually value. This is part of the reason why an ASI may conceive a goal we do not like. We are the flawed ones!
