Aligned models have less information/entropy than Shannon measured for English

Shannon measured the information rate of English at ~1 bit per character. According to a byte-level LLM measuring next-character predictability in LLM & human text (diaries, abstracts, dreams, fiction), aligned models produce sub-English information rates & LLM text is more predictable than humans’.

Bar chart comparing information density (BLT bits/char) of AI-generated prose versus human text. Shannon's English rate (1.0 bits/char) shown as dashed red line. Aligned OLMo models (SFT, DPO, RLVR) fall below the line at 0.89–0.99 bits/char. The base model sits just above at 1.14. All human text types are higher: waking reports (1.24), abstracts (1.28), dreams (1.32), and fiction (1.49). Alignment compresses model output below the information density of all measured human writing.

Part of Libidinal Language Models.

Originally posted on Bluesky.

← All notes