Quoting Machines of Loving Grace

Quoting Dario Amodei's Machines of Loving Grace (via)

I am not as confident that AI can address inequality and economic growth as I am that it can invent fundamental technologies, because technology has such obvious high returns to intelligence (including the ability to route around complexities and lack of data) whereas the economy involves a lot of constraints from humans, as well as a large dose of intrinsic complexity. I am somewhat skeptical that an AI could solve the famous “socialist calculation problem” and I don’t think governments will (or should) turn over their economic policy to such an entity, even if it could do so. There are also problems like how to convince people to take treatments that are effective but that they may be suspicious of.

I was intrigued by this assertion and had a somewhat long discussion with Gemini about why that's the case to help me build a mental model about the problem and it used a game analogy that was quite useful so trying to reproduce as much of that conversation as possible below.

To understand Dario Amodei’s skepticism about AI fixing the economy, you have to realize that humanity is actually playing two completely different genres of games on the same server: Player vs. Environment (PvE) and Player vs. Player (PvP). Amodei is confident AI will dominate the PvE game—science and biology—because it is essentially a speedrun against a static boss. When an AI learns to fold a protein or stabilize a fusion reactor, it is playing against the laws of physics, and physics never changes its attack patterns or tries to trick you. The "boss" of cancer doesn't update its code just because you found a cure, meaning high intelligence yields massive returns.

The economy, however, is a PvP game where you are competing against eight billion other humans, and this introduces "The Meta." In a PvP game, the moment a developer introduces a powerful new item or a rule patch—like a new tax policy to fix inequality—the players immediately change their strategy to counter it. This is the gaming equivalent of the Lucas Critique; if an AI tries to balance the server by taxing income, the high-level players stop taking income and start taking loans against their assets to dodge the "nerf." The AI isn't solving a puzzle; it is entering an endless loop of patching exploits while the players constantly find new ones.

I then asked if this is not closer related to poker as it has very simple rules as a game but the meta of interaction between players make it an impossible problem to announce it as solved.

The poker analogy is actually much sharper than the video game one because it perfectly captures the difference between Game Theory Optimal (GTO) play—which AI has mastered—and the messy, human "meta" of the economy where rules are socially enforced rather than coded.

In a vacuum, AI has already "solved" poker by calculating the Maximum Expected Value (Max EV) of every possible hand. This is the PvE side of the equation: if you treat the economy purely as a mathematical problem of resource allocation, the AI knows exactly when to "bet" (invest) and when to "fold" (cut losses) to maximize global GDP. The laws of probability, like the laws of physics, don't change; a flush beats a straight every single time, just like a specific protein structure cures a disease every single time. If everyone sat down at the table and agreed to play strictly by the math, the AI would generate the most wealth for everyone efficiently.

However, the economy is a PvP table where the players are constantly trying to change the rules of the house or collude under the table. While the AI is playing perfect GTO poker to maximize the pot for everyone, the human players are engaging in "The Meta"—bluffing not just with their cards, but with their political power. The billionaire players might threaten to leave the casino entirely if the "blind" (taxes) is raised, or they might bribe the dealer (regulators) to peek at the deck. The AI can calculate the perfect bet size to distribute the winnings fairly, but it cannot calculate for a player who flips the table or convinces the other players that the AI is cheating.

Ultimately, Amodei’s point is that you can build a machine that plays the cards perfectly, but you cannot build a machine that forces humans to stop bluffing or cheating. In science (PvE), nature never bluffs; an electron doesn't try to deceive the microscope. In the economy (PvP), the "optimal" move often fails because humans care about status, power, and perception, not just the raw chip count. The AI sees the math of the hand, but it is blind to the "tell" of human politics, meaning it will likely win the scientific pot while losing the social game.