The idea is closely related to Richard Sutton’s concept of the bitter lesson, which describes how sheer scale has consistently won out over more specific, complex methods of training AI models. Note that scale is not the only source of improvements in model capabilities—algorithmic efficiencies (“compute multipliers”), which allow us to (for example) achieve higher capabilities with the same amount of compute, have played a major role. But these efficiencies are themselves found using the scaling laws: researchers test new ideas and keep the ones that improve the scaling trend the most.
In the prospectus: The founders’ letter (page image) (original filing on SEC EDGAR)
This is a big lesson. As a field, we still have not thoroughly learned it, as we are continuing to make the same kind of mistakes. To see this, and to effectively resist it, we have to understand the appeal of these mistakes. We have to learn the bitter lesson that building in how we think we think does not work in the long run. The bitter lesson is based on the historical observations that 1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term, and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.
One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great.
From Rich Sutton, “The Bitter Lesson”: paragraph 6, paragraph 7 (Chinese translation on this site)