Thoughts on watching Translating Claude’s thoughts into language
This idea is pure genius, and it carries a simple truth, as if it should have been this way all along. So why did no one think of it before?
Activations can’t be read directly, so you train a model specifically to turn them into text, then convert that text back into activations to check whether it’s right.
This process can be repeated until the model can explain the activations correctly.
Pure genius, and very generous too: they generously shared this remarkable idea along with a lot of technical detail.