By John Romeo
The first hint came in 2016 when Lee Sedol, a renowned Go champion, faced off against Google’s artificial intelligence computer, AlphaGo.
The Korean strategy board game has been played for thousands of years, creating a wealth of human knowledge about what it takes to win. Then AlphaGo played Move 37, something few humans had ever done. Commentators initially thought it was a mistake, but it turned out to be a stroke of brilliance. AlphaGo won the game, and overturned centuries of conventional wisdom.
I think about Move 37 a lot today. We tend to think of technological development as an evolution of increasingly powerful tools, from the computer and PC to the internet and smartphone. But AI is on the verge of something more fundamental: making the leap from intelligent tool to an actor capable of learning, reasoning, and, increasingly, deciding and taking action.
AI started by giving us answers; now we are beginning to give it authority. We give an agent access to our data, then our tools, then a mandate, then a budget, and finally permission to act. At some point it stops looking like software and starts looking like an economic actor.
The battle for machine permission
That presents a dilemma to companies: What happens when the person choosing your airline isn’t a person? When an AI chooses your insurance, your investments or your energy supplier? The battle for human attention may become a battle for machine permission, an issue we discuss in our new AI report, The Reality Check. And power follows permission.
I heard a wonderfully practical suggestion recently from Tim Berners-Lee: Give AI agents license plates. If an agent loses money or compromises a system, for example, we should be able to answer four questions: What is the agent? What is it allowed to do? Who is accountable for what it does? And who can stop it? The license can solve for identity, permission, accountability and revocability.
Yuval Noah Harari once made a related observation to me. People lie, cheat, and pursue their own interests, so we developed contracts, courts, auditors, boards and regulators. Civilization works not because we trust everyone but because we learned how to cooperate with people we cannot completely trust.
Only four years into the generative AI era, we are already confronting the possibility of agents deceiving, coordinating in unexpected ways, and concealing what they have done. Instead of trying to design machines we can blindly trust, perhaps we should focus on building institutions that allow us to use machines we cannot trust.
That gets harder when machines operate at machine speed. Human control increasingly has to move up a level where people set the boundaries, determine the escalation rules, and retain the ability to revoke authority, while machines increasingly supervise machines within those boundaries. Machine-speed oversight, but human accountability.
People will still have agency
There is a danger in all this talk of races and control. It can make us sound like spectators, as though AI were an autonomous force sweeping through our companies and societies and our only choice was how quickly to adapt. It isn’t. Technology has a direction because people give it one. Companies decide which decisions machines may make. Governments determine permissions and accountability. Society should ultimately have a voice in what this extraordinary technology is for.
Which brings me back to Move 37. AlphaGo won because it tried a move most humans never imagined. That is precisely what we want from AI. New medicines, new materials, scientific discoveries and solutions to problems we could not solve ourselves. But what happens when Move 37 comes to a financial market, a payment system, a cyberattack, or a military decision?
The very quality that makes AI so valuable is also what makes governing it so difficult. Machines will make moves we don’t expect and increasingly moves we don’t understand. Some will be extraordinary. Some may be dangerous.
But there is still one decision the machine does not get to make: We decide which games it is allowed to play. AI will have more agency. But so do we.
John Romeo is CEO of the Oliver Wyman Forum
Four years after generative AI burst on the scene, the technology seems all-pervasive. People increasingly use AI in place of search, companies are ramping up investment, massive infrastructure spending is juicing financial markets and roiling our politics (data centers, anyone?), and hopes of abundance vie with fears of existential risk in our social media feeds. We don’t know if AI is conscious, but it’s fast invading our collective consciousness.
Yet there’s one place where AI remains relatively scarce: the bottom line. AI capability is growing faster than the decisions that can turn it into value, and few companies are generating significant cost savings or revenue gains from the technology.
To move the needle, leaders need to decide how they want to transform the business with AI, and what types of investment and return profile will get them there, according to a new Oliver Wyman Forum report, The Reality Check. Of note, CEOs reporting large returns on AI are much likelier than others to see the technology as a growth driver.
Leaders must communicate their AI strategy to managers and rank-and-file employees and make clear the role both groups can play in the journey. They also need to adapt their marketing and sales channels for AI agents, take a proactive approach to risk management that achieves both speed and safety, and decide which parts of the AI tech stack are worth paying to control.
Here are five imperatives for leaders, drawn from a growing body of AI reality checks now coming into focus.
Rethink your whole approach to ROI. Targeted applications, such as using AI to increase call center efficiency, can produce quick, measurable results. Using AI to redesign workflows or develop new product innovations can produce bigger payoffs over time but require more extensive changes in how the enterprise operates, as well as strong data and risk management foundations.
Companies need to be clear where each investment fits in this mosaic, define the expected path to return, and account for the full cost of deployment, including time checking AI output. They also need an ROI harvester, a senior executive with authority over business decisions who is accountable for delivering returns.
Rewire the enterprise for a bionic workforce. Leaders need to decide where they want AI to take the business and what role employees will play in that journey. Many are failing that test, with only 37% of executives and 11% of non-managers saying management has conveyed a clear AI ambition. That may explain why so many workers play down their AI skills or express willingness to take a pay cut in return for a job guarantee.
Companies also need to adapt to the way AI unbundles management, with junior employees spending a growing share of their time managing AI tools. That requires managers to step up and become micro-CEOs of agentic-powered human teams, and leaders to create new development paths for entry-level workers doing less of the routine work that traditionally built experience.
Reset the sales pitch for machine attention. Companies that have long honed marketing strategies to capture the fancy of consumers need to pivot quickly to the realities of AI. A third of consumers already start with AI when looking for new products or services, including nearly half of Gen Z, and they are more likely to describe a need and let AI recommend products or brands.
E-commerce increasingly demands product information and supporting evidence that AI agents can verify and compare, checkout processes they can navigate, and anti-fraud measures that block malicious actors while letting authorized agents in. Welcome to the age of SAO — shopping agent optimization.
Design guardrails that enable growth. Well-designed governance can help an organization accelerate, not slow down. Sturdy guardrails reduce ambiguity by telling employees which use-cases they can pursue, which ones require additional review, and who owns the decision. Little surprise, then, that where approved AI is easy to access, clearly governed, and built into everyday software, workplace users are nearly twice as likely to report productivity gains as elsewhere.
It’s like Formula 1, where well-designed tracks and better brakes enable racecars to go faster.
Price in AI sovereignty. No single country or company can single-handedly control the full AI tech stack, from energy and data to the latest chips and large language models. Dependencies are growing as AI adoption increases, but preparedness is lagging awareness of the risk. Consider that 55% of companies expect disruption if their main AI provider becomes unavailable, but few have backup plans in place to switch providers.
In practice, AI sovereignty means choosing the terms of dependence: knowing what to rely on, how hard it would be to replace, and ensuring you have options if a critical dependency were compromised.
The acceleration of AI is a remarkable story of technological innovation. Our ability to safely harness its power depends on organizational decision-making. These reality checks can help leaders make the difficult choices that lie ahead.