In this chapter, we ask what economic modeling can tell us about the likelihood that firms will invent an AGI: how much and for how long must they sustain investment in R&D to obtain such an invention? We develop a novel Real Options Model, one that uses a stochastic compound Poisson process, to explicitly take into account that a radical innovation such as an AGI is subject to much more uncertainty than typical business investments – which also helps throw light of the breakthroughs and backlashes that have characterized periodic AI winters. An important conclusion from the modelling in this chapter is that it will most likely be largely government-funded agencies and/or a few large corporations that will invent an AGI, if it is ever invented.

In this chapter, we depart from the observation that a strong motivation for large firms to invest substantial amounts into R&D for an AGI is due to the winner-takes-all effects it may bestow on them. This feature, while important to incentivize AI investment, has the downside that it implies that AI arms races may take place. And the danger of an AI arms race is that it may result in an inferior AGI from a human safety perspective. In this chapter, we model such an AI arms race as an innovation contest and show how a government can steer such an arms race so as to obtain a better outcome in terms of the quality of the AGI. A crucial insight from our modeling is that the intention (or goals) of teams competing in an AGI race, as well as the possibility of an intermediate outcome (“second prize”), may be important.

To answer the question of what responsible AI means, the authors, Jaan Tallinn and Richard Ngo, propose a framework for the deployment of AI which focuses on two concepts: delegation and supervision. The framework aims towards building ‘delegate AIs’ which lack goals of their own but can perform any task delegated to them. However, AIs trained with hardcoded reward functions, or even human feedback, often learn to game their reward signal instead of accomplishing their intended tasks. Thus, Tallinn and Ngo argue that it will be important to develop more advanced techniques for continuous high-quality supervision – for example, by evaluating the reasons which AIs give for their choices of actions. These supervision techniques might be made scalable by training AIs to generate reward signals for more advanced AIs. Given their current limitations, however, Tallinn and Ngo call for caution when developing new AI: we must be aware of the risks and overcome self-interest and dangerous competitive incentives in order to avoid them.

Over the coming decades, artificial intelligence and robotics will continue to grow rapidly in power and scope, exerting a transformative impact on our lives. Seven trends are likely: the machines will become progressively more versatile and multifunctional in their capabilities; they will be designed to improve their own software and hardware over time, as they interact with their environment; the line between AI and robots will gradually blur, as machines come to permeate our society at all levels; these machines will be entrusted with complex practical tasks that require them to develop sophisticated commonsense knowledge about the human social world; the machines will be interconnected in functional networks that multiply their powers; such machines will need to be able to refuse to obey certain kinds of human commands, raising fundamental questions about who controls their actions; and the logic of arms races will apply to such machines, impelling nations and corporations to develop increasingly powerful machines as quickly as possible, with only a secondary concern for caution and safety.