Notes

Ankur Moitra: So much of our society now is geared towards winning and looking good on paper, and I always wonder to what end. You start off in high school and you can start, as soon as you walk in the door, optimizing your APs and your extracurriculars so that you can get to an elite school. And then when you get to that elite school, you might skip classes because, you know, maybe the important thing is to prepare for your interviews so that you can get a job at a frontier lab, and then you can get some nice, you know, restricted stock units and live out your life. And that’s great. You won. You won at every stage, but so what?

You know, for me, the, you know, the best way to live a rich and fulfilling life is just to pretend you’ve already won, and then figure out what you would do in that scenario. I remember the advice I got when I started at MIT as faculty. Sanjeev Arora told me, “You know, the easiest way to get tenure is just to pretend you already have it and do what you would have wanted to do all along.”

So I don’t know if that advice applies to, you know, graduate students, to high schoolers too, but I think my advice stays the same. Pretend you already have tenure. What would you now do? ‘Cause a lot of the people we’re giving advice to have really so many different opportunities. Finding something that you really believe in is the only way I know to be fulfilled.

…

Lex Fridman: And I think there’s a lot of wisdom in what Ankur says. You can be paralyzed by that fear, or you can imagine, “I’ve already won. What would the person who has won do in this situation?” And that alleviates the fear. It actually takes the– gives you the optimal path under uncertainty.

From Lex Fridman Podcast #503 (video, from 3:27:58); text from the official transcript.

What surprised me is that I’ve used this method myself before: imagine a goal has already been reached, and ask what you would do then. It’s a very good way to think, because it helps you work out what you really want, instead of being pushed along by all kinds of constraints.

Alex Kontorovich: I’m constantly talking to AI systems, and I’m learning a lot of math a lot faster than I’d be able to. Because the way math is written, you know, I sometimes give a paper to a grad student to read, and they come back a week later, and I say, “So how far did you get?” “Well, I didn’t really get past the first page, to be honest.” “Oh, really? Tell me.” “Uh, well, actually, I didn’t get past the first paragraph.” I say, “Okay, fine. Why not?” “Well, the first sentence is by the perturbation theory of non-local systems and I don’t know what any of that means, so that’s kinda where I’m stuck.” I’m like, “Okay. You don’t know how to read a math paper. You don’t– You skip the first couple of paragraphs. Go straight to, you know, the Lemma 3.” “And that’s where you can start working and work your way backwards to what the thing is doing.”

But in the future, there is no “I don’t know what this means,” ‘cause you just upload it to an LLM, and you start asking it questions, and it will zero in on the zone of proximal development, the place where here’s everything that you know, and here’s the next thing you need to hear so that you start learning. And if people use that effectively, they can really supercharge their education. I don’t just mean kids. I mean old fogies like me- … can learn new fields. You know, to learn a field, like this is what our graduate students are going through. They have to completely rewire their brains. Like, learning math is still hard.

Up until past the master’s level into the PhD level, you are not expected to produce new mathematical theories. Everything that they can do is something that is in a textbook somewhere. And yet it’s still extraordinarily, extraordinarily difficult for people to- … Learn to solve problems that are solvable by lots of other people, never mind problems that aren’t solved. So learning math is hard. Learning new math is even harder, discovering math, whatever, working out solutions to math problems. But I think it’s going to get easier, and this tool is gonna be an amazing resource for people to be able to get to that zone of proximal development much more rapidly and develop their understanding much more rapidly.

Lex Fridman: Yeah, I’ve been having so much fun on topics I’m not even… And I would love to go back to grad school with the tools we have now. But one of the ways my brain likes to learn is constructing visual examples, toy examples, the simplest possible example that still contains the insight there, and especially if it’s interactive. Like, if I can, like, move stuff, it’s like, oh, you immediately, like, get it. All these words that the community agreed mean something, you’re supposed to know all the notation, all that kind of stuff. If I can just know that this little symbol means this, oh, okay, cool, all right. And just everything kind of falls into place.

Also, you fall in love with the subject, and I feel like I can take on any problem that’s completely outside of my field. Like, I can understand, like, protein folding. I can, like, play with the parameters and play. I can take a DeepMind paper and actually understand by taking any aspects of it and playing with it, and what’s a simple example? Ask crazy aforementioned pothead questions. So yeah, this is super fun. The question is whether people will be motivated to learn. This is the problem, is the problem of human motivation. That’s the real question.

Steven Brunton: Well, the key ingredients there are curiosity, asking good questions, and being able to call bullshit. Like the- … critical advisor skills of, like, getting information where you’re not the expert, and knowing if you need to dig deeper or push back. And I think, you know, there will always be people who are curious and not curious, and who ask good questions and don’t. We need to be really promoting that more than we ever have. It’s a little scary to me that we’re expecting our undergraduates, our young students, to interact with these tools when they’re learning, when their brains aren’t fully formed. We’re asking them to have critical, not just critical reasoning skills themselves, but, like, advising skills. How do you take in information and tell if it’s true or not, or how to question it? And that’s a huge task for us in education, is how do we- … train people to have good advisor skills when they’re kids?

Lex Fridman: Yeah, I think the hardest problem isn’t undergrad. The hardest problem is, like, middle school ‘cause, like, that, I… What is the point at which homework starts getting hard? That, at that point, why do the homework?

Steven Brunton: There’s the temptation, right?

Lex Fridman: That thing will just… At first, I’m saying, see, I’ve come to appreciate the value of doing difficult things because of all the suffering, but I think because I was forced to. But if you’re not forced to do it because LLM can, AI can just help you, I don’t know what that, what kind of society that creates.

Alex Kontorovich: But math has the longest head start on that question because calculators came out, what, 70 years ago? And we still ask kids to learn their times tables. At least some of us do, and those who said, “No, kids don’t need to learn times tables,” you know, to the extent that teachers listened to that and kids listened to that and didn’t learn- … they really suffered. There’s a great, great amount of human suffering resulting from people saying you don’t need to know times tables ‘cause calculators can do it. And my hope is that we can learn from that and say just because a calculator can do it, you’re gonna be much better off if you can also do it yourself. Doesn’t mean if I have to do some calculation, obviously I’m reaching for a calculator.

But I could in, in principle do it myself, and that’s what allows every time you give yourself that skill and master a skill. Kids love mastering skills. How happy are they when they learn to tie their shoelaces for the first time? You would think it’s like, “It’s just shoelaces. Who cares?” No, it’s the greatest thing ever. You wanna celebrate every little bit of independence and you want to be able to think. You wanna be independent from the machines. Just ‘cause the machine can do something doesn’t mean you don’t wanna have the capacity to do it yourself.

From Lex Fridman Podcast #503 (video, 3:11:34–3:17:45); text from the official transcript.

This passage is very meta and very illuminating. The metacognitive way of learning they describe is itself worth learning, and what they say about the changes the AI era brings is also valuable.

Alex Kontorovich: But we also don’t wanna outsource so much– nearly so much decision-making to these systems for exactly that reason.

Steven Brunton: Yeah, we have infrastructure and protocols in place already for safety critical and high impact decisions. We have the FAA, we have the American Medical Association, the Bar Association for these things that affect humans. And I think of, you know, how do we incorporate AI into our general processes in the entire world on two axes. We have, you know, how much do we trust the output, so verifiability or trustworthiness, and on the other axis is how high stakes is it if it’s wrong. If AI spits out a wrong proof, nobody goes without power. It’s fine. So it’s very low stakes. And it is actually extremely verifiable, so you know very quickly if it’s right or wrong, and it’s low stakes if it’s wrong. So that’s the perfect use case is this open-ended design space. Designing a better vehicle, you know, a better car design that’s lower drag, more fuel efficient. Low stakes if it gets a wrong design because you can always test it in a wind tunnel and verify if it did or didn’t work.

So that, like, design optimization space is perfect for AI, and we as humans have to be very careful about not bleeding into– We need a firewall of what needs to actually be verified, and we have protocols, we have systems, legal systems, and infrastructure for verification. But it is something we need to think about. If we get lazy- … We can– It’s a very slippery slope. We need to have that firewall.

From Lex Fridman Podcast #503 (video, from 3:02:13); text from the official transcript. The image is a frame from the video at 3:03:30.

A good way to classify and think about where AI can be applied.

Ankur Moitra: So the way that this works is it’s an example of something called online algorithms. So you have a sequence of options coming. One through N. They come in each time step, and your goal is, you know, when someone comes along, you know their ranking relative to everyone else you’ve seen before, but you don’t know their ranking relative to the people who are yet to come.

And your goal is, you know, to make a decision when the person comes along, do you commit or do they move on, in such a way that you have a good chance of finding the absolute best candidate. And so the point is that you can actually get a constant competitive ratio algorithm that what you do, the best strategy is to wait some amount of the time, not make any decisions, don’t accept anyone. And then once that probationary window has passed, you take the first person who’s better than anyone you’ve seen thus far. And that’ll give you a good competitive ratio in the sense that you can prove theorems. Assuming the inputs arrive in a uniformly random order, you’re likely to do pretty well compared to as if you knew what that sequence of inputs was in advance.

From Lex Fridman Podcast #503 (video, from 2:44:30); text from the official transcript.

Its real lesson is to explore first to build a baseline, then switch to deciding, and not to wait until the very end to choose. The exact 37% only holds for the classic model and should be taken as an intuition.

This may be a very useful tool, but before using it you should also know its limits so you don’t fall into them.

Ankur Moitra: I think also students have so much anxiety about, you know, what publications are gonna look like that one of the hardest parts is, you know, when I’m thinking about a project, I’m really trying to think about it in a sort of multi-year time horizon because I have some meta vision that I wanna realize, like how can we use learning theory techniques in quantum many-body systems? And there are a lot of stepping stones on route to that, but some of the most exciting problems are, you know, many years out. But students are so worried about the incoming AI tidal wave, you know, the future of work, universal basic income. Are any of us gonna have jobs? That they’re really thinking in three- and six-month horizons, and then it’s hard to find problems that we both care about just because- You know, we’re thinking about different worlds.

Lex Fridman: You don’t think we should encourage them to continue to think long term? I mean, I understand that-

Ankur Moitra: Oh, I do, but they don’t always listen to you. I mean, I don’t know if I’m right or wrong. You know, maybe AI will solve all of our problems, but it seems like the only ones that are actually worth us working on at this moment in time are the ones we’re still gonna care about in a bunch of years, and you should only be working on future-proof problems. If you’re working on things that, you know, you’re in a rush because you think the next generation of models will definitely solve those problems, then maybe just go on vacation and don’t solve that problem.

From Lex Fridman Podcast #503 (video, from 2:25:46); text from the official transcript.

Ever since the shock the AI wave brought, I seem to have forgotten that I used to be able to think about problems like this. It’s a really good way of thinking, and I should try to pick it back up.

Although, as of October 30, 2026, there is no evidence that the Binance hack had anything to do with AI, I believe it did. I think there is a good chance the hackers used AI, directly or indirectly, in the attack.

As an engineer, I know full well that hackers can use AI to launch large-scale attacks with ease. Hacking is, at its core, finding vulnerabilities in software. In the past that process usually depended heavily on people and was expensive to carry out, and what AI is best at is reading code tirelessly, in many threads at once, line by line, to find vulnerabilities.

Because computer software is so complex, hackers can use all kinds of techniques to get around the visible guardrails, directly or indirectly, so a very strict set of guardrails is necessary. That is why I think it is reasonable for Anthropic to strictly restrict users from hacking, even if the restrictions are sometimes too strict. Of course, until the whole world has been hit by large-scale hacking, they look too strict.

There is another kind of attack that Anthropic mentions again and again, one I don’t know much about and whose consequences the public perhaps hasn’t yet had to bear: chemical weapons attacks. Since the hacking guardrail, one of the two big guardrails, has turned out to be real, I am willing to believe that the attacks behind the other guardrail may be real too, namely chemical weapons attacks. Compared with hacking, this kind is probably more harmful in most cases. Other AI developers may have set up guardrails for it long ago, but at least in the public sphere Anthropic looks like the one that has gone deepest. That helps build public trust in Anthropic, and the hacking consequences coming true will strengthen that trust.

In the long run, I think Anthropic’s safety-first strategy is underrated and effective. People will lean toward products that take responsibility for safety, not irresponsible ones.

Every time another company’s product has a similar accident, public trust in Anthropic goes up. That looks a little dark, since other companies’ trouble works in Anthropic’s favor, but either way it will make companies and the public trust Anthropic more. Other AI companies may copy Anthropic’s approach, which is good and right for AI safety as a whole. Beyond that, though, it could produce something like the “Apple” and “Android” landscape: Anthropic is “Apple” and the other companies are “Android”. Android can do many things better than Apple, but it all evolved from Apple’s design philosophy, yet Android will never become Apple; they will lose the sharpness and distinctiveness of their own brands.

I hope other companies realize this early and invest more in independent safety research, at a scale equal to Anthropic’s or greater. Only then can they truly compete with Anthropic across the board; otherwise they can only be followers, not competitors.

AI is Transforming Intelligence

We believe that Powerful AI, a highly capable and impactful form of frontier AI characterized by the following key attributes, will become widely available within a few years:

• cognitive capabilities matching or exceeding those of top human experts across most professional and scientific disciplines;

• the ability to navigate all interfaces available to a human working virtually, including text, audio, video, and direct computer control;

• the ability to reason through complex tasks autonomously over extended periods—lasting hours, days, or weeks—and seek clarification and feedback when needed;

• the ability to coordinate across multiple instances and collaborate with both humans and other AI systems; and

• the ability to interface with the physical world by digitally controlling laboratory equipment, robotic systems, and other machines through digital connections.

If Powerful AI is directed at solving humanity’s hardest problems and the risks are managed responsibly, the potential is extraordinary. The possibilities include compressing decades of scientific progress into years; accelerating drug discovery, disease detection, and the delivery of personalized therapy; providing individually tailored education and expert financial and legal advice; advancing power grids and battery design for greater efficiency and reliability; and extending to the developing world capabilities that took wealthy nations generations to build.

In the prospectus: Prospectus summary: AI is Transforming Intelligence (original filing on SEC EDGAR)

I broadly agree, especially with the line “cognitive capabilities matching or exceeding those of top human experts across most professional and scientific disciplines.” If AI can solve math problems (a while ago I saw that Navier–Stokes, one of the Millennium Prize Problems, had been solved), then what professional problem is there that AI can’t solve?

Mathematics has always been seen as the hardest kind of problem human intelligence can solve. If even this can be solved, what reason is there that other problems that depend on intelligence can’t be?

Enterprise Strategy. Our enterprise strategy is built on the belief that frontier AI is most valuable in real-world settings, as deployment across enterprises is where economic impact is greatest and where the improvement loops from customer feedback are richest. Accordingly, we have organized our research priorities, our product roadmap, and our go-to-market and Applied AI teams around enterprise use cases, which range from routine knowledge work to the most complex problems. The customer feedback we gather directs our research focus and makes Claude more capable over time. This broadens where Claude can be deployed and further expands enterprise adoption, which helps ensure that the benefits of our AI reach people and organizations at scale.

In the prospectus: Prospectus summary: Overview (original filing on SEC EDGAR)

In other words, what enterprises say matters a lot, and the optimization and development are mainly built around enterprise needs.

We manage compute as a single, fungible pool from multiple strategic partners, and we dynamically direct available compute between workloads based on where the marginal unit is expected to be most productive at any given time.

In the prospectus: Prospectus summary: Our Approach to Compute (original filing on SEC EDGAR)

In other words, they’ve built a dynamic compute platform: when the consumer side needs more compute, more goes to the consumer side, and whatever frees up goes to research and training.

We have achieved this through our research breakthroughs and our multi-chip compute architecture across AWS Trainium, Google TPUs, NVIDIA GPUs, and AMD GPUs, enabling us to orchestrate workloads to optimize for price and performance.

In the prospectus: Prospectus summary: Our Approach to Compute (original filing on SEC EDGAR)

And it’s one unified dynamic compute platform that spans hardware platforms.

To build AI is to harness cognition itself, and anything so powerful can be used for ill as well as good.

In the prospectus: The founders’ letter (page image) (original filing on SEC EDGAR)