Comment by jszymborski

I enjoyed this. With the hindsight of today's LMs, people might get a kick our of reading Claude Shannon's "Prediction and Entropy of Printed English" which was published as early as 1950 [0], and later expanded on by Cover and King in 1978 [1].

They are fun reads and people interested in LMs like myself probably won't be able to stop thinking about how they can see the echos of this work in Bengio et al.'s 2003 paper.

[0] Shannon CE. Prediction and Entropy of Printed English. In: Claude E Shannon: Collected Papers [Internet]. IEEE; 1993 [cited 2025 Sep 15]. p. 194–208. Available from: https://ieeexplore.ieee.org/document/5312178

[1] Cover T, King R. A convergent gambling estimate of the entropy of English. IEEE Trans Inform Theory. 1978 Jul;24(4):413–21.