Does the bitter lesson have limits?

Summary

Recently, “the bitter lesson” is having a moment. Coined in an essay by Rich Sutton, the bitter lesson is that, “general methods that leverage computation are ultimately the most effective, and by a large margin.” Why is the lesson bitter? Sutton writes: The bitter lesson is based on the historical observations that 1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term, and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach. Sutton walks through how the fields of computer chess, computer go, speech recognition, and computer vision have all experienced the bitter lesson. Lately, people have been citing the bitter lesson a lot. A Blow to the Human Ego The first time I read “The Bitter Lesson”, I thought of Donna Haraway, one of my professors at UCSC. During a seminar on the history of science, she presented her list of the four major blows to the human ego (which I previously wrote about): These concepts undermined humans’ imagined central position in the universe. The bitter lesson slots in naturally here; really a sub-point of cyborgs, a topic which Haraway has been exploring for decades. Within this framework, it’s easy to believe! My instinctual belief reminds me of an exchange from a 2010 RadioLab segment with Neil deGrasse Tyson: ROBERT KRULWICH: Is it your working bias that if I came to you with a new discovery in which we were less important, or a discovery which proposed that we were more important, that you would guess that my scientific discovery that said we are less important is more likely to be right? NEIL DEGRASSE TYSON: No doubt about it. That’s correct. Now you call th...