Finnegans Wake has extremely high Shannon entropy; frontier models extremely low

Surprising: frontier models (Claude, ChatGPT, Deepseek V4) produce the most predictable text of any local AI model or human text I’ve ever tested.

Not surprising: Finnegans Wake is off the charts, by far least predictable—Shannon & Lydia Liu proved right—and Hemingway the most predictable human.

Horizontal boxplot comparing information density (BLT 1B bits/char) across 27 text sources, ordered from most compressed (top) to most information-dense (bottom). Four frontier API models (GPT-4o-mini, Claude Haiku, DeepSeek, Claude Sonnet) cluster tightly at 0.85-0.90 bits/char, below Shannon's English rate of 1.0 bits/char marked by a dashed vertical line. Seven local aligned models (red) span a wide range from DeepSeek-7b aligned (0.94) through OLMo aligned (1.43), with most falling between 1.0 and 1.3. Seven base models (green) occupy 1.04-1.50, overlapping substantially with human text. Eight human text sources (blue) range from Hemingway (1.12) and abstracts (1.20) through dream reports (1.24), waking journals (1.26), and Basic English stories (1.39) up to C20 fiction (1.50) and Joyce (2.30). OLMo aligned is a notable outlier among aligned models, with higher information density than its own base model. DeepSeek-7b base (1.04) is unusually low for a base model, sitting near Shannon's threshold alongside Hemingway.

Lydia Liu, The Freudian Robot, p. 37:  "...interesting questions that his experimental work raises for us is: how does a stochastic view of writing correlate to the received theories of language, literature, and modernism on the one hand and to psychoanalytical speculations about the unconscious on the other? This question is pertinent to our inquiry because many of the earlier modernist literary and psychoanalytical experiments on language, automatic writing, and thought-reading had anticipated Shannon’s Printed English and his “mind-reading machine” in numerous ways. For instance, Shannon cites James Joyce’s Finnegans Wake as one of the texts exemplifying the lower threshold of redundancy and higher entropy rate in his stochastic model of Printed English. What makes entropy and its possible linkage with Freud’s Todestrieb (death drive) such an interesting problem for the study of digital media is the ways in which certain ideas migrated into psychoanalysis first and then got into information theory. Furthermore, Freud’s work and psychoanalysis in general may suggest some interesting clues as to the shared theoretical impulses or implicit exchanges among information theory, cybernetics, and modernist literature. Spanning across these moments of broad intellectual confluences is the techne of the unconscious that continually articulates itself to digital writing, machine, and social engineering. We turn next to the invention of Printed English by Shannon and its implications for a theory of digital writing."

Quoting Aligned models have less information/entropy than Shannon measured for English.

Part of Libidinal Language Models.

Originally posted on Bluesky.

← All notes