- CPA to Cybersecurity
- Posts
- Notes on "The Coming Wave" by Mustafa Suleyman
Notes on "The Coming Wave" by Mustafa Suleyman
Technology, Power, and the Twenty-first Century's Greatest Dilemma
Steve McMichael
January 31, 2025
Hey there,
Here are my notes and questions from a great read, recommended on the Daily Cyber Threat Briefing.
If you go to simplycyber.io/books, this is my #1 recommended book (shout out to Christina Paulika, AKA The Librarian, who helps manage this for the community). The Coming Wave by Mustafa Suleyman, should be required reading in high school. He was the co-founder of DeepMind, which is where they started neural networking for AI in 2012, and Google acquired them. He's more of a philosopher, classically trained, than a technologist, but he understands what's going on and he lays out a lot of stuff that is very interesting. So definitely go check that out.
My favourite part of the book was learning the origin story of DeepMind (behind Google’s Gemini), including the AlphaGo documentary in Chapter 8.
I’ve listed some discussion questions at the end. Let me know in Discord or by replying to this email.
Be safe, be well,
Steve
Chapters
Chapter 1: Containment Is Not Possible
Advanced AI and synthetic biology promise immense benefits but also catastrophic risks that current institutions are unequipped to handle, necessitating a new paradigm of technological containment.
Artificial Intelligence (AI): The science of teaching machines to learn human-like capabilities.
Artificial General Intelligence (AGI): The point at which an AI can perform all human cognitive skills better than the smartest humans.
Artificial Capable Intelligence (ACI): A fast-approaching point between AI and AGI, wherein ACI can achieve a wide range of complex tasks but is still a long way from being fully general.
"It's no exaggeration to say the entirety of the human world depends on either living systems or our intelligence, and yet both are now in an unprecedented moment of exponential innovation and upheaval, an unparalleled augmentation that will leave little unchanged."
Replicating human intelligence in AI could help solve major challenges but introduces unprecedented dangers.
Potential to create “abundance” of energy, food, goods, services, education, medicine, etc. but also empower bad actors and risks.
Waves: The global diffusion or proliferation of a generation of technology, anchored in a new, general-purpose technology.
The Coming Wave: An emerging cluster of related technologies, centred on AI and synthetic biology, whose transformative applications will both empower humankind and present unprecedented risks.
Pessimism aversion: The tendency for people, particularly elites, to ignore, downplay, or reject narratives they see as overly negative. A variant of optimism bias, it colours much of the debate around the future, especially in technology circles.
Dismissing concerns about advanced technology often comes from an emotional aversion to dark possibilities.
Overcoming pessimism aversion is necessary to properly address the risks of advanced technologies.
Geopolitical competition and open research culture incentivize the wave's uncontrolled development by many actors.
Almost every foundational technology follows the law of becoming cheaper, easier to use, and widely proliferated.
AI systems can now perform near-human level object/facial recognition, speech transcription, and language translation.
AI is also making rapid progress in more challenging areas like long-term planning and imagination.
"Failure in this sense isn't intrinsic to technology - it is about the context within which it operates, the governance structures it is subject to, the networks of power and uses to which it is put."
Technology has been the primary engine of human progress and achievement.
Ancient humans first harnessed fire and stone tools as transformative early technologies.
Faster planes, safer cars, and more powerful computers exemplify technology overcoming risks of malfunction.
Reminds me of this blog post from Dec 2023:
Part I: Homo Technologicus
Chapter 2: Endless Proliferation
Technology proliferates inevitably in accelerating, intensifying waves that shape the human story and portend a future of continued transformation.
The story of the internal combustion engine is the story of technology itself:
Personal transportation was limited to walking or beasts of burden for most of human history.
Railways revolutionized transport in the early 19th century, but were not very personalized.
Innovators experimented with various approaches to create self-propelled, personal transportation, including steam, internal combustion, electricity, and hydrogen.
Karl Benz patented the first proper car in 1886, but it remained expensive and limited.
Henry Ford's Model T, built using the moving assembly line, made cars affordable for middle-class Americans.
Automobiles proliferated rapidly, with 59% of Americans owning a car by 1930.
Internal combustion engines are now ubiquitous, powering everything from lawnmowers to container ships.
"Technology has a clear, inevitable trajectory – mass diffusion in great, rolling waves."
The agricultural revolution marked the arrival of plant and animal domestication, gradually replacing hunter-gatherer life.
At the dawn of the agricultural revolution, the worldwide human population numbered just 2.4 million.
The Industrial Revolution saw immense technological change measured in decades rather than centuries or millennia.
Once a technology gains traction and a wave starts building, diffusion echoes throughout history.
The printing press, books, electricity, and consumer technologies all exhibited explosive proliferation and plummeting costs.
"Civilization's appetite for useful and cheaper technologies is boundless. This will not change."
"Mass diffusion, raw, rampant proliferation, this is technology's historical default, the closest thing to a natural state."
Takeaways:
Study the patterns of technological waves throughout history to anticipate and prepare for future changes.
Invest in research and development of potential general purpose technologies that could spark new waves.
Foster a culture of experimentation, tinkering, and innovation to drive technological progress.
Plan for the inevitable disruptions and transformations that come with each new wave of technology.
Chapter 3: The Containment Problem
Solving technology's containment problem is now humanity's most crucial challenge in an era of exponential, uncontrollable change.
Containment: The ability to monitor, curtail, control, and potentially even close down technologies.
The Containment Problem: Technology's predisposition to diffuse widely in waves and to have emergent impacts that are impossible to predict or control, including negative and unforeseen consequences.
"Technology's unavoidable challenge is that its makers quickly lose control over the path their inventions take once introduced to the world."
Technology's uncontrollable, unpredictable nature means inventors quickly lose agency over their creations' ultimate impacts.
Exponential growth in technological power and access makes potential harms and unintended consequences increasingly catastrophic.
Meaningful technology containment requires mutually reinforcing technical, cultural, legal, and political control mechanisms.
"Nuclear is an exception to the unstoppable spread of technology, but only because of the tremendous costs and complexity involved, the decades of tough multilateral effort, the fear-inducing enormity of its lethal potential, and pure luck."
Historically, cheaper, more efficient technologies overwhelm resistance and proliferate uncontrollably once established.
In the 21st century, the core technological challenge has inverted from unleashing to containing its power.
Develop rigorous technical containment protocols and failsafes for all high-stakes technologies like AI systems.
Cultivate a culture of responsibility and vigilance for unintended consequences among technology creators and institutions.
Pass proactive national and international legal frameworks to meaningfully govern and contain high-risk technologies.
Shift incentives in technology development from capability maximization to responsible stewardship and safety.
Massively increase public education and awareness about technology's containment problem and potential existential risks.
Part II: The Next Wave
Chapter 4: The Technology of Intelligence
AI is a transformative meta-technology whose rapidly expanding capabilities will profoundly reshape invention, labor, and society in the coming years.
AI's ability to discover novel strategies and knowledge has immense potential to equip humanity with new inventions and solutions
The convergence of AI, synthetic biology, and other emerging technologies represents an unprecedented phase transition in technology's capabilities
AI is already ubiquitous and will soon permeate nearly every aspect of life, rivaling human perception and cognition
Debates about AI sentience distract from substantive challenges in robustness, bias, safety, and real-world applications of AI systems
Framing AGI as a sudden event is misguided; it will likely arise through the gradual accumulation of AI capabilities
In the near future, AIs will be able to autonomously carry out complex, open-ended tasks in the real world
The implications of widely accessible AI systems that can help achieve almost any goal are profound and unpredictable
AI is not just another technology, but a meta-technology that will transform the very process of invention itself
Chapter 5: The Technology of Life
The convergence of AI and synthetic biology will transform every field, ushering in the age of synthetic life.
"Alongside AI, this is the most important transformation of our lifetimes."
McKinsey estimates up to 60% of physical inputs to the economy could be subject to "bio-innovation".
Stay informed on the latest breakthroughs in AI and synthetic biology, two exponentially advancing fields.
Consider the disruptive potential of synthetic biology on medicine, manufacturing, materials, energy, and more.
Anticipate major societal repercussions from extended lifespans and potential human genetic enhancement.
Explore the intersection of AI and synthetic biology as the defining technologies of the coming decades.
Chapter 6: The Wider Wave
The current technological wave, anchored by AI and robotics, represents a qualitative break from the past in its proliferation of the power to manipulate information and matter, demanding a proactive, interdisciplinary, and adaptive approach to responsibly stewarding its immense potential for transformative impact.
John Deere's steel plow transformed agriculture in the 1800s; the company now builds agricultural robots.
"From drones watching livestock, to precision irrigation rigs, to small mobile robots patrolling vast indoor farms, from seeding to harvesting, picking to palletizing, watering tomatoes to tracking and herding cattle, the reality of the food we eat today is that it increasingly comes from a world of robots, driven by AI, currently being rolled out and scaled up."
Quantum computing could break cryptography on "Q-Day", but also has tremendous benefits like molecular modeling.
Nanotechnology envisions atoms as controllable building blocks to automatically assemble almost anything.
The next energy transition is crucial; modern civilization relies on vast amounts of energy.
"Fusion and solar offer the promise of immense centralized and decentralized energy grids, with implications we will explore in part 3."
Educate yourself and others about the interrelated nature of technological waves and their far-reaching impacts.
Monitor developments in robotics across sectors to anticipate and prepare for automation's expanding reach.
Consider the potential risks and opportunities of transformative technologies like quantum computing and nanotechnology.
"At its core, the coming wave is a story of the proliferation of power. If the last wave reduced the costs of broadcasting information, this one reduces the costs of acting on it, giving rise to technologies that go from sequencing to synthesis, reading to writing, editing to creating, imitating conversations to leading them."
Chapter 7: Four Features of the Coming Wave
The coming wave of AI technologies will be uncontainable due to their:
Asymmetric impact: Asymmetric technologies are redistributing power away from states and toward smaller actors
Hyper-evolution: blurring the line between bits and atoms, accelerating real-world innovation
Omni-use nature: these technologies tend to be the most impactful because of their sheer versatility
"Omni-use technologies like steam or electricity have wider societal effects and spillovers than narrower technologies."
Increasing autonomy: means we are losing meaningful human control over powerful AI systems
AI may eventually recursively improve itself in an intelligence explosion, becoming uncontrollable
Gorillas are physically stronger than humans but it is humans who contain them
"Never before have so many had access to such advanced technologies capable of inflicting death and mayhem."
Chapter 8: Unstoppable Incentives
Entrenched and mutually reinforcing incentives across geopolitics, capitalism, science, and the human ego make the arrival of a transformative and risk-laden technological wave unstoppable.
AlphaGo, an AI developed by DeepMind, defeats the world's best Go player in a historic match, demonstrating the potential of AI.
Over 280 million people watched AlphaGo's matches against top human Go players live
AlphaGo's victory is a major milestone in AI, showing machines can outperform humans in complex tasks.
The open global research ecosystem makes governing and containing new technologies very difficult
Raw curiosity, national competition, corporate profit, and human ego all propel technology forward
"This is why we won't say no. This is why the coming wave is coming, why containing it is such a challenge. Technology is now an indispensable megasystem, infusing every aspect of daily life, society and the economy. No one can do without it. Entrenched incentives are in place for more of it, radically more. No one is in full control of what it does or where it goes next."
Part III: States of Failure
Chapter 9: The Grand Bargain
The coming wave of powerful technologies threatens to destabilize the already fracturing grand bargain of nation-states.
The Grand Bargain: In exchange for a monopoly over the right to use force, citizens expect nation-states to preserve order and provide public services, including by harnessing new technologies, while minimizing the harmful side effects.
"The grand bargain is fracturing, and technology is a critical driver of this historic transformation."
Recognize technology's potential to address major challenges while also acknowledging and examining its risks and downsides.
Invest in making governments more agile, competent, and trustworthy to navigate the coming technological disruptions.
"If the state is unable to coordinate the containment of this wave, unable to ensure that it is of net benefit to its citizens, what options does that leave humanity in the medium to long term?"
"Our system of nation-states isn't perfect, far from it. Nonetheless, we must do everything to bolster and protect it."
Educate leaders and technologists on the inherently political nature and implications of powerful new technologies.
Study history to understand how previous technological waves shaped political structures and apply those insights.
Chapter 10: Fragility Amplifiers
The coming technological revolution's redistribution of power across society threatens to shake the foundations and stability of the nation-state system.
Fragility amplifiers: Applications and impacts of coming wave technologies that will shake the already broken bargain.
“Democratizing access necessarily means democratizing risk. We are about to cross a critical threshold in the history of our species.”
WannaCry ransomware infected 250,000 computers across 150 countries in one day
AI cyberweapons will continuously probe networks, adapting to find and exploit weaknesses
Throughout history, technology has produced a delicate dance of offensive and defensive advantage, the pendulum swinging between the two, but a balance roughly holding.
Now, powerful, asymmetric, omni-use technologies are certain to reach the hands of those who want to damage the state. While defensive operations will be strengthened in time, the nature of the four features favours offense. Its proliferation of power is just too wide, too fast, and too open. An algorithm of world-changing significance can be stored on a laptop. Soon it won't even require the kind of vast, regulatable infrastructure of the last wave and the Internet.
AI-enhanced weapons will improve themselves in real time, making attacks more potent
AIs could find legal or financial means of damaging corporations or institutions
AIs could develop psychological tricks to gain trust and influence for manipulation
Deepfakes enable creation of highly realistic fake media for disinformation campaigns
The term “fragile” in this chapter reminded me of this book, and this morning I saw the author talking about the “grey swan” event of DeepSeek AI
Recommendations:
Invest heavily in defensive cybersecurity measures to keep pace with offensive evolution
Establish clear standards and verification systems for discerning authentic media
Tighten global biosafety protocols and oversight of dual-use pathogen research
Chapter 11: The Future of Nations
The coming technological wave's contradictory forces of centralization and decentralization will radically redistribute power, pressuring the nation-state to a breaking point.
Exponential technologies will amplify power, leading to both centralization and fragmentation of nation-states.
Simple innovations can have profound, long-lasting impacts on power structures and social organization.
Stirrups revolutionized warfare and society, enabling feudalism to dominate medieval Europe for a millennium.
“The stirrup was an apparently simple innovation, but with it came a social revolution changing hundreds of millions of lives."
Exponential technologies amplify everyone and everything, leading to contradictory trends of power concentration and dispersion.
Authoritarian states will exploit the coming wave as rocket fuel to entrench their dominance, including with ubiquitous and granular surveillance technologies
Technology can reinforce and upend social structures, hierarchies, and regimes of control simultaneously.
"Understanding the future means handling multiple, conflicting trajectories at once."
This is Roger Martin’s integrative thinking concept that I’ve talked about before in How to Break into GRC.
Chapter 12: The Dilemma
Navigating between dystopian control/surveillance and catastrophic technological failures from emerging technologies is the defining challenge of our time.
Safely developing advanced AI requires solving the alignment problem not just once, but every single time.
Technological development has allowed modern civilization to temporarily escape historical cycles of collapse and unsustainability.
"The coming wave, then, might paradoxically create the very tools needed to contain itself. Yet in doing so, it would open up a failure mode where self-determination, freedom and privacy are erased, where systems of machine surveillance and control metastasize into society-strangling forms of domination."
Demographic decline and resource depletion mean halting technological progress would lead to dystopia through economic collapse.
We must honestly acknowledge both the immense benefits and catastrophic risks of technological development to chart a path forward.
Part IV: Through the Wave
Chapter 13: Containment Must Be Possible
Exponential technological change is inevitable, and containment through unified action and incentives is humanity's necessary challenge.
"Saying 'regulation' in the face of 'awesome technological change' is the easy part."
"Technology evolves week by week. Drafting and passing legislation takes years."
Nations are conflicted between accelerating frontier tech development and trying to contain its risks.
Regulating fast-evolving, general-purpose technologies is incredibly difficult and will likely have gaps.
Chapter 14: Ten Steps Towards Containment
Containing the unprecedented risks of AI and biotechnology requires realigning incentives and coordinating efforts across technical, governmental, corporate, cultural, and social domains to walk a narrow path between dystopia and catastrophe.
Containment requires carefully balancing openness and security on a narrow path between catastrophe and dystopia.
The Narrow Path: The potential for humankind to strike a balance between openness and closure when it comes to containing the technologies in the coming wave, that avoids catastrophic or dystopian outcomes.
The ten steps toward containment involve a multi-layered approach, beginning with technology and expanding to culture, incentives, and international agreements. Each step requires specific interventions and skills, and their combined effect is intended to ensure effective containment.
Safety is achieved through technical measures and responsible deployment, such as addressing biases in AI models using reinforcement learning from human feedback. Physical containment strategies, like air gaps, can prevent AI interaction with the external environment. Increasing investment in safety research and development is also crucial.
Audits ensure transparency and oversight through meaningful rules and technical implementation reviews. Verifying system integrity involves access rights, auditing, and adversarial testing, including red teaming. Sharing insights about risks among tech companies and creating government-funded red teams are also important.
Chokepoints create rate-limiting factors by leveraging critical R&D and commercialization hubs to regulate development. Export controls on key technologies, such as advanced semiconductors, can slow down technological development.
Makers and Critics work together to address technology's problems. Critics should be involved in the creation process, incorporating ethical and humanistic perspectives.
Businesses should adopt commercial models that incentivize safety alongside profit. Hybrid organizations that reconcile profit and social purpose are essential, with governance models prioritizing transparency, accountability, and ethics.
Governments should increase their involvement in technology development, standard-setting, and in-house capabilities. This includes monitoring technological developments, understanding data usage, and logging technology-related harms.
Alliances should be formed through international collaborations to address global challenges. This involves agreements to limit certain technologies and collaborations between countries to address shared risks.
Culture should be reformed to promote a healthier technological approach and encourage a global movement.
Incentives should be created to prioritize safety and responsible development in business.
International treaties are needed to establish international agreements and frameworks for governing the development and deployment of advanced technologies.
Incentives drive technological development more than containment efforts, so realigning incentives is crucial.
Effective containment requires coordinated efforts across many domains, from technical to cultural to political.
Call to action 👇️
"Credible critics must be practitioners, building the right technology, having the practical means to change its course, not just observing and commenting, but actively showing the way, making the change, effecting the necessary actions at source."
“Lather, Rinse, Repeat” GRC Mantra in the Simply Cyber community 👇️
"Safe, contained technology is, like liberal democracy, not a final end state. Rather, it is an ongoing process, a delicate equilibrium that must be actively maintained, constantly fought for, and protected."
The main monitor of bioweapons, the Biological Weapons Convention, has a budget of just $1.4 million.
In 2021, there were only around 100 AI safety researchers at top labs; in 2022, 300-400.
72 bills with "artificial intelligence" have passed worldwide since 2019; the OECD tracks 800+ AI policies.
The Montreal Protocol, Biological Weapons Convention, and other treaties show international cooperation is possible.
In the early jet age, there was one death per 7.4 million passenger boardings; now years pass without fatalities.
AI Audit Authority (AAA) idea:
What an AI Audit Authority office might look like
A dedicated regulator that navigates contentious geopolitics as much as possible, avoids overreach, and performs a pragmatic monitoring function on broadly objective criteria is urgently needed.
Think of something like the International Atomic Energy Agency, or even a trade body like the International Air Transport Association. Rather than having an organization that itself directly regulates, builds, or controls technology, I would start with something like an AI audit authority, the “AAA.”
Focused on fact-finding and auditing model scale and when capability thresholds are crossed, the AAA would increase global transparency at the frontier, asking questions like, "Does the system show signs of being able to self-improve capabilities? Can it specify its own goals? Can it acquire more resources without human oversight? Is it deliberately trained in deception or manipulation?"
Similar audit commissions could operate in almost every area of the wave and would, again, offer a foundation for government licensing efforts, while also helping the push for a non-proliferation treaty.
Actually having meaningful oversight and enforceable rules and reviewing technical implementations are vital.
Technical safety advances and regulation will struggle to be effective if you can't verify that they are working as intended.
How can you be sure what's really happening and check that you're in control? It's an immense technical and social challenge.
Trust comes from transparency. We absolutely need to be able to verify, at every level, the safety, integrity, or uncompromised nature of a system. That in turn is about access rights and audit capacity, about adversarially testing systems, having teams of white hat hackers, or even AIs probing weaknesses, flaws, and biases.
It's about building technology in an entirely different way, with tools and techniques that don't yet exist.
External scrutiny is essential. Right now there's no global, formal, or routine effort to test deployed systems. There's no early warning apparatus for technological risks, and no uniform or rigorous way of knowing if they abide by regulations or even adhere to commonly agreed benchmarks. There are neither the institutions nor the standardized assessments nor the tools necessary.
Life After the Anthropocene
Unprecedented human adaptation and control of the coming wave of world-changing AI and synthetic biology technologies is crucial to avoid the Luddites' fate and secure our species' flourishing future.
In 1785, Edmund Cartwright debuted the powerloom, a mechanized means of weaving.
The powerloom could be operated by a child, producing as much as 3.5 traditional weavers.
Weavers' wages more than halved in the 45 years after 1770 as food prices leapt.
In 1807, 6,000 weavers protested pay cuts; dragoons killed a protester breaking it up.
In 1811, Luddites destroyed 63 machines in Nottingham mills after "General Ludd" letters.
England had a few thousand automatic looms around 1811 but a quarter million by 1850.
"The challenge today is clear. We have to claim the benefits of the wave without being overwhelmed by its harms."
Discussion Questions
What were the biggest brain benders for you in this book?
Did the book make you more pessimistic or less about AI’s risks and opportunities?
Are you a “doomer” or an “accelerationist”?
Do you agree with the author in chapter 10 that offense has the advantage?
“Throughout history, technology has produced a delicate dance of offensive and defensive advantage, the pendulum swinging between the two, but a balance roughly holding.
Now, powerful, asymmetric, omni-use technologies are certain to reach the hands of those who want to damage the state. While defensive operations will be strengthened in time, the nature of the four features favours offense. Its proliferation of power is just too wide, too fast, and too open. An algorithm of world-changing significance can be stored on a laptop. Soon it won't even require the kind of vast, regulatable infrastructure of the last wave and the Internet.Where do you see opportunities for cloud, local and hybrid approaches?”
What do you think about Chapter 14’s AI Audit Authority (AAA) idea?
“Focused on fact-finding and auditing model scale and when capability thresholds are crossed, the AAA would increase global transparency at the frontier, asking questions like, "Does the system show signs of being able to self-improve capabilities? Can it specify its own goals? Can it acquire more resources without human oversight? Is it deliberately trained in deception or manipulation?"
What are the implications of the Coming Wave to cybersecurity GRC?