The debate on the risk connected with the development of superintelligent systems has been going on for a while now, and in the last few years it has intensified considerably - especially since large language models have established themselves as powerful new oracles, mathematics superpowers, and code-writing wizards. So the very recent declarations by the CEOs of OpenAI and Anthropic, among others, cannot be considered as bolts from the blue. However, we are sitting on a very steep slope of AI improvements, and thus a reassessment of the situation on a very short-period basis is in order.
A couple of years ago I wrote a paper together with a few experts in various scientific disciplines, where we examined the risks and benefits of future AI developments and related consequences through the lens of scientific research. The 60-page document is published by IEEE Access in gold open access and is available here. In it, we identified 10 specific risks to human society and assessed their impact and the likelihood of manifesting themselves in the short, medium, and long term. Our assessment is summarized in the graph below.
As you can see, two years ago a set of 25 authors - among them distinguished computer scientists and hard science experts - gave the threat of "Inception of misaligned AGI", labeled "I" above, a very low probability in the short and medium term (<10 years, 10-30 years), and a low probability in the 0.001 range for long term (>30 years). We also considered a number of other risks, listed in the table below; of relevance to the discussion is the one labeled "C", which was assessed by us at or around 10% likely in the short-medium term. The graph considers both likelihood and severity, and the colour code is meaningful as it indicates what is overall considered most worrisome: in that sense, AGI misalignment was recognized as the only risk reaching the highest level even if we gave it a relatively low likelihood of occurrence.
Below is reported the table of the risks we discuss in the paper:
Why have I brought up that article and its assessments here? Because it is a gauge of the speed at which things are evolving -mindboggling, for sure. I cannot poll my co-authors for this, but if I were to re-assess the likelihood and the severity of risks C and I today, less than 24 months after I did that exercise, I would significantly change the earlier figures. For C, I would put the likelihood as HL throughout, even much closer than 10 years away; and for I, I would put the likelihood as ML already in the next couple of years.
What has recently been going on
But let us first discuss what is in the news these days. One datum is the resigning from Anthropic of Jacob Coxon, who explained that the company is internally discussing when, and not if, the inception of AGI is going to happen - and the considered time range is in months from now, not many years. Another one is the assessment he makes concerning the probability that the developed AGI ends up annihilating humanity, putting it at 10%. A third datum is the declaration of Amodei, the CEO of Anthropic, who comes forward and admits that the risk is there, and that some agreed-upon steps have to be taken together, to slow the pace and give us time to build the necessary safeguards.
Whether one believes that the chance of an end to human civilization is 1% or 10% matters little to me. The very fact that the risk is clear and significantly non-zero is everything that matters. Humanity has faced threats of annihilation in the past only during the darkest days of the cold war, when the US and the USSR had their finger on the trigger. Today, we are subjected to a clear danger not because of geopolitical dynamics, but because of the greed of AI developers. A coherent effort by governments to put brakes to those developments would suffice to reduce the risk to an acceptably near-zero value, but this is not happening because governments today are remote-controlled by the same developers, through enormous amassed capitals. In a sense, the present risks are children of the absurd level of wealth a deregulated capitalism has put in the hands of very few individuals.
Whatever the source of our present headache, it seems rather clear to me that with the enormous profits guaranteed by achieving AGI first, and with the geographically distributed nature of the players, there are little chances to see some of them deliberately slow down or accept to be put under internal scrutiny. What else can we do?
Safeguards
One thing we can do is to try to analyze what really are the risks, dissect them, and try to put together specific defences to each. We should not focus on the movie plot of an AGI becoming malignant and deliberately targeting humans as its own goal. Rather, we should consider that AGI will, in almost any likely scenario, want more resources for whatever goal it decides to have. This will leave humans in the state of expendable collaterals, much like a ant-hill in a building construction site. I consider this risk one that we can defend against, if we plan ahead. Human control to energy sources and internet highways should be enforced; data centres should be accessible and their shut down made possible to governments - again, in a semi-manual way. A number of other safeguards of various levels of detail could be conceived. This kind of activities should become a priority of the department of defence of any country.
In my opinion the main risk, however, remains human-driven. As little faith we can have in the benign nature of a future AGI, humans remain more dangerous than sentient automated systems by far. The weaponization of AGI powers, in the hands of some deranged individual or group of fanatics, deserves careful consideration. While construction of nuclear weapons has remained since WW2 an impossible wish for terrorist groups and even most states, due to the complexity of the task and the shortage and control of uranium sources, with AI we do not have that safeguard: the possibility of finding cheap, fast ways to engineer new molecules with huge harming potential is already today something we must reckon with.
The biggest risk is already a fact: authoritarianism
And there is another risk that has at its roots the same cause that is powering the acceleration of AI research. It is the one labeled "C" in the graph and table above, and we are already seeing it at play in the US and other countries. The enormous wealth concentrated in the hands of few multi-billionaires has created a situation where governments are increasingly unable to withstand the pressure of lobbyists. And the push, of course, is toward authoritarianism, because that is the simplest way to silent dissent, prevent democratic forces from functioning, and continue amassing power and more wealth. The old "divide et impera" motto seems the way this plays out: by supporting far-right parties in Europe, for example, they undermine the cohesion of the European Union, which is seen as an obstacle toward a completely free-of-rule market and complete control. And authoritarian regimes also support a situation where regional wars continue to erupt, favoring weapons producers. So there. Do you think I falled prey of conspiracy theories? You are entitled to your own opinions. For now.
