Back to notes

Notes

Post AGI Classics

06/06/26.

the goal is to have an opinion on the future, not a surface level summary of it nor a sheep influenced one. by testing the existing beliefs against a wide range of data and a diverse mix of writings the hope is to develop a strong perspective to defend and act on.

The intelligence curse

AI corporate pyramid

AI still seems to be a tool; the AI employee hasn’t happened (yet). Every startup I talk to is trying to hire more. Counter to that, founder friends with bigger teams report not hiring any more SWE. But for some reason it feels like we are here already:

tried comparing the job market to the unemployment rate, but it's hard to make sense of it given everything else that’s going on, high level, it seems like unemployment and job openings have been pretty similar since 2025.

first wave of agents arrive, companies will either of do nothing (regrets it quickly), fire everyone, adopt it slowly. the process goes bottoms up, first entry positions, then junior, middle management, etc... at some point in between managers manage more AI's than humans (expect a lot of observability tooling/startups to be needed here).

the first affected will be the most competitive sectors (startups/swe/finance/hedge funds), public resentment grows, earning reports increase, all this driven by competition, firms that decide not to will simply die.

at some point AI does all intellectual labor, some firms will only need the C-suite and some will not even need humans, the only humans earning are the shareholders.

then it argues (2.3 caveats): "currently it seems like tasks that require planning and execution over longer-time horizons will take longer for AI to automate, and it will be harder for AI companies to train on non verifiable domains". both are wip from the labs as of right now.

some white collar job (trust/connection based) will be harder to automate. displaced humans will join blue collar professions aka physical work

finally (2.4) future of work suggests that as ai increases labor supply, wages go down, fixed inputs (resources, land, etc) goes up, and humans eventually get a minimum biological wage to below subsistence levels.

inferring some 2nd order effects of this:

as humans get displaced into physical labor, a lot of retraining and human coordination seem necessary. if physical labor is cheap and keeps getting cheaper, you can imagine collecting physical data does too. it'd mean that humans keep having value as long as they can do novel (data) things in the physical world

what’s still not clear is:

if humans reach a point where they no longer provide value and don’t earn, then the companies that still exist, who are they doing work for? an economy only between corporations, is that even possible?

3. Capital, AGI, and Human Ambition

the 3rd essay from the intelligence curse argues that the key AI economic effect is that capital (resources) becomes a much better substitute for labor (human resources).

  1. the same capital can deliver much more
  2. the value of labor goes down, along with human power
  3. it becomes harder for humans to achieve outlier outcomes relative to their starting point

finding good talent isn’t easy, but it matters less if you can convert raw money into top 1% labor. how? more land, more electricity, more data centers, more compute, more data. the value of fixed inputs goes up.

"Everyone with enough money to burn on GPUs gets the AI star researcher." we don't need anything else once we get here.

the pyramid replacement stops. "the possibility of someone achieving success and changing the world is important for avoiding stasis and continued progress."

3.4 enforced equality: if UBI or redistribution works in some way, it's plausible that immigration stops. why would rich countries care about non-citizens?

3.5 the new Medici families: "My great great great grandfather was a MTS at anthropic." a permanent aristocracy.

what does claude has to say?

"it's a price question: humans stay employable as a resource only while the cost of a human doing a task stays below the one of a machine doing it"

4. defining the intelligence curse:

nations that stop depending on human capital for economic output.

why? powerful actors care about you for the ROI of your labor: corporations, government, etc. they educate you, build roads, give you perks. if the above becomes true, you lose bargaining power, and none of this is necessary.

think of intelligence as a natural resource. countries that have decoupled national economic output from human capital through natural resources are known as rentier states (venezuela, saudi arabia, norway). quality of life becomes less tied to the country’s economic results.

rentiers (corporations) become the largest taxpayers. when labs account for 15%+ of the economy, the intelligence curse has taken off.

the question is: who are the powerful actors producing anything for?

so can the economy sideline human consumption? the essay says yes: ultra-wealthy people colonizing mars, excavating the ocean, or just maximizing their pleasure.

4.4 can we break the intelligence curse?

  1. we need effective institutions that can redistribute wealth, similar to how Norway built a $1.7T sovereign fund.
  2. put human labor at the center, as Saudi Arabia is diversifying (caveat: intelligence is not finite).

I have hopes for the first, but capitalism and competition seem incompatible with the second.

5. will we ever be free of economic pressure?

“neither competitive pressures nor human greed have any intrinsic stopping point.”

nations have no option but to compete, or be completely dominated. if the US decides to slow down, others get ahead. humanity could only avoid this if we coordinated as a whole (choice-transition), but that sounds ridiculous. geopolitics would not allow it.

the good outcome

  • humans have economic value
  • a high standard of living for everyone (abundance)
  • no single actor or oligarchy monopolizes AGI

how do we build technologies that help people remain economically relevant?

given our competitive nature, we must find a way to make humans + AI have higher economic output than AI alone, at least for as long as possible

6 breaking the intelligence curse (part 1)

the goal is to avert it is by applying a “swiss cheese” mitigation model: stacking multiple safety measures on top of each other so that no single failure leads to disaster.

this does not mean centralizing the technology or locking down all labs. that would only concentrate godlike power in the hands of a few actors.

instead, focus should be on misuse. we are already seeing labs limit cybersecurity, restrict certain uses, and release weaker models. biology may be next. but the same logic applies to everything: hardware design, CAD, and any top 1% skill that can be weaponized specially in the physical world.

we also need more work on alignment and interpretability, along with stronger observability and monitoring.

part 2, humans in the loop.

augmenting humans, ensuring the symbiosis of humans x ai > ai.

at least for as long as possible. otherwise, this starts looking like a Foundation (Asimov) problem: the goal is to shorten a predicted dark age from 30k to 1k years, and make it more of a transition period.

the concern is that RSI seems to be here (every lab and neolab is working on it).

  1. we need more evals that measure human x ai vs ai, so we understand where we are and keep finding ways to keep humans >. GDPVal reporting human x ai results costs and speed improvements.
  2. interfaces that help humans process more bits per second.
  3. human feedback and steering over agents with long horizons.

eventually, humans might be forced into BCI.

then it argues about individuals having their own finetuned model where their local knowledge and taste has value.

its extremely difficult to see this working out for 2 reasons, 1) labs will buy any data that it's valuable, at any cost, and they will buy it from people before people know how valuable it is 2) why would Opus-10 with your knowledge be better than Opus-10 alone?

data might be the last job standing.

part 3. decentralization and policy

distributed training runs, open source AI, and local compute.

user-specific alignment: build an economy of agents where your agent represents you in the digital economy, does the work, and is guided by your judgment, taste, and tacit knowledge (tax the individual, not the agent).

as said above, tacit knowledge will likely be bought as data before humans understand its value. if so, this won’t work.

we’ll need fast human upskilling and retraining, especially in areas that bottleneck the AI economy: non-verifiable tasks, physical work, and trust- or connection-based fields.

on policy: ban AI from owning assets. we’ll need new coordination (sort of digital advocates) and verification systems, we could return to a version of democracy where every decision is actually voted, not taken by a few representatives.

"governments should prepare for a world where lots of regular people don't provide immediate value"

7. history is ours to write

  1. governments should be forecasting.
  2. people in think tanks should be preparing post-AGI policies.
  3. if you are at a lab, self-critique.
  4. if you’re young, get ambitious: help humans stay economically relevant and spread abundance.
  5. if you’re a VC, fund projects that keep humans in charge.

now, back to work.

AI Enabled Coups

what would enable a small group of people to use AI to seize power?

very timely (Fable was just banned) to read AI Enabled Coups by @TomDavidsonX to dive, through multiple lenses, into the different scenarios playing out now.

the answer:

  • secret loyalties
  • singular loyalties
  • centralization

why?

  1. loyalties: with AI deployed in key institutions, a small group of people could steer these institutions in whatever direction they want. consider a military with autonomous systems.
  2. centralization: a small group of people with unprecedented intelligence in every field, over the rest of society.

key points that help clarifying what’s happening today:

  1. leaders of frontier AI projects, government, and the military are the most likely to be in a position to perform a coup.
  2. reducing the likelihood of something like this would almost certainly slow down productivity, so economic incentives are not aligned.
  3. capability nerfing can result in more centralization and outsized power.
  4. open source is great for preventing centralization and increasing proliferation, but it comes with alignment risk. as of now, it’s not possible to prevent misuse by bad actors.
  5. neolabs are great for preventing centralization.
  6. RLaaS companies and inference providers will be key to preventing a centralized scenario where all intelligence comes from only a few places.
  7. cybersec’s gradual deployment has been handled well*: access to key institutions first (Project Glasswing), then gradual deployment everywhere else.
  8. nations’ competitive pressure only increases rushed adoption of autonomous AI systems in the military and government.
    • one could argue that anthropic fighting the government was a good thing, because it forced the government to work with multiple labs instead of relying on a single provider, slowing down centralization.

    as capabilities become more powerful and dangerous, access will likely be increasingly restricted

  9. we will need more safety institutions with good governance structures. we don’t have enough public knowledge or awareness of models’ deception and persuasion capabilities (or secret loyalties). we also need better forensic data analysis, and third-party and red-teaming audits for government use of AI, and for AI itself.
  10. no single individual should be allowed to control more than a certain number of AI systems in key institutions. models must follow the law.

some open questions:

  • is the first to achieve RSI a winner take all scenario?
  • can you balance misuse and privacy or are we destined to highly intrusive monitoring systems

forethought.org/research/ai-en…

AI-Enabled Coups: How a Small Group Could Use AI to Seize Power

The British Industrial Revolution in Global Perspective (2009)

the industrial revolution recipe can help understand where we are going, a high disparity between two forms of capital (wages vs energy), plus the right incentives (institutions), drives technological shifts.

in 1760 Britain had both key ingredients:

  • high wages with very cheap energy (lots of coal)
  • parliamentary power, private property, and defined taxation

how does the shift develops?

early on, it’s mostly an isoquant shift: from (labor, coal) = (0.8, 0.2) to (0.2, 0.79). it’s a small gain, mainly from shifting capital. but at some point, industry by industry, isoquants shift inward, meaning you can produce more with fewer inputs.

do other nations fall behind?

new technologies first fit one country’s factor prices, giving it an early edge, but as that same nation improves the technology, others can now benefit and the advantage disappears

when do shifts result in "better for everyone"?

technology can turn labor scarcity into surplus, driving wages down. specialization and higher living standards usually follow decades later, but without labor rules the first generation often suffers.

Situational Awareness

Some predictions:

  • Most important and TBD: Inflection point: 2027, automated AI researcher.
    • ASI follows very soon after AGI, imagine 1M Alec Racfords that work 24/7. Wouldn't 100 of him be able to solve robotics or any research problem?
  • Total AI investment will be 1T by 2027, I think we are close to this by 07/2026.
    • 1T year is only 3%, 1996-2001 Telecoms invested ~1T, 1841-1850, 40% GDP, equivalent 40%

  • Compute is all that matters: OpenAI vs Anthropic, not rly as of 07/2026.
  • America will become a war factory but for chips, it's just a race for 100B cluster.

  • The Manhattan Project of AI
    • Buildout of DC can't be done under dictatorships

    • weights need to be protected, labs, individuals, data will all be of national interest.
    • Superintelligence is all what matters, and whoever get’s there first has to protect it at all costs
    • A pause is non sense and risky, china other nations will not, and we’ll be fucked
    • No international treaty on safety, when is decisively clear US will win, offer deal to china

    • The radical thing is not this, it's letting private CEOs wielding military power, and becoming benevolent dictators
  • if AGI doesn't happen through this decade (the Compute decade), we'll have slow progress after that
  • Superalignment
    • “Will they honestly answer our questions” “reliably follow our instruction” “will they deceive humans”
    • AI will start suggesting decisions that will be impossible for humans to understand their real intention
    • We need more mech interp, reasoning interp, and automated alignment work, Safety research tasks
    • We need to figure out creative oversight strategies
    • Even if best case scenario it just buys us some margin of error.
    • Training smaller models to handle oversight for us, highlight fishy things
    • Unlearning biology + chemistry + cyber will be necessary (yes it's happening)
  • 100B revenues by 2025/2026, and we'll start seeing by 2028 10T valuations

People generally fail one estimates for being too optimistic, not in AI, the opposite

  • MATH dataset was ought to be pointless (too hard), saturated 3 months after
  • Is taste a bottleneck? if so is this the 80/20 remaining for humans? and for how long?
  • How do you help humans monitoring (processing more bits per second)
  • There was more variance expected within labs, we have not, everyone has converged to the same
  • Models capabilities improve 3x faster than children, gg.
  • Societal rollout in medical or legal professions where connection is necessary will make it slower

Predicting if a large scale experiment will be successful from only a small one is a printing machine (AGI would solve it) as well as optimizing GPU workloads in general