Generative Formalism

How do large language models idealize the culture they reproduce? Analysing over 15,000 AI-generated poems, I found that models produce formally conservative verse, rhymed and regularly metered, at rates exceeding even the most traditional periods of literary history; that this bias persists even when they are instructed against traditional forms; and that training data does not explain it. Parallel patterns appear in history: prompted to recall publication records, models produce demographic distributions that obscure well-known patterns of exclusion. Instruction tuning is one mechanism, as models fine-tuned to be helpful assistants idealize more than their base models. This “cultural collapse” privileges satisfaction over frustration, regularity over variation, and conformity over contradiction. The work continues in Creative Collapse and Libidinal Language Models.

Notes

Temperature does not explain formal stuckness on rhyme in LLMs

Preparing talk on this for #CHR2025 by addressing an oversight in the paper: assessing impact of model temperature on generative poems’ rhyme. As expected, temp negatively correlates. The effect is significant, but very weak (R2=0.02), suggesting “formal suckness” is largely invariant to model temp.

Two-panel scatter plot showing the relationship between temperature and rhyming proportion in LLM-generated poem completions. Left panel shows llama3.1:instruct model (80% baseline rhyming) with shallow negative slope. Right panel shows llama3.1:text model (10% baseline rhyming) with shallow negative slope. Both show statistically significant (p<0.0001) but weak effects (pseudo R²=0.01-0.02).

Continue →

LLM-generated verse is formally stuck on rhyme and regular meter

As in work by @mellymeldubs.bsky.social et al, I examine formal tendencies in the poetic output of 9 LLMs. I find that LLM verse is “formally stuck” on rhyme & regular meter, producing these forms far more than even the formally strictest periods of literary history—and even when instructed not to.

Figure 1.Frequency of rhymed poems in the Chadwyck-Healey corpora compared with LLM-generated verse. Generative models were prompted for three types of poems: rhyming poems, unrhyming poems, and poems without specifying whether to rhyme. Points indicate mean likelihood; size indicates the number of poems per data point; whiskers show standard error.  The left-hand portion of the graph shows that historically, across the 1,000 sampled poems per half-century in the Chadwyck-Healey poetry collections, the average likelihood for a poem to rhyme descends over time, particularly since the late nineteenth century. Prior to that, the historical incidence of rhymed verse hovers between 71% and 90% for three centuries, before turning away from it in the late nineteenth century (71%) and plummeting in the early twentieth (15%)—with ultimately only 7% of poems written by postwar poets in rhyme.  Meanwhile, the right-hand portion of the graph, a distant reading of rhyme not in historical but generative verse, shows several striking findings. On average across the LLMs, prompts for a rhyming poem (shown in blue) and prompts simply for a poem without specifying whether to rhyme (shown in green) produce a body of artificial verse composed of 95% rhyming poetry. Not only is this incidence more than in any historical period of actual human-authored verse, but its similarity across these two categories of prompts suggests a remarkably stubborn association between the genre of poetry and the form of rhyme within these models. Furthermore, whether prompted simply to write a poem, a rhyming poem, or an unrhyming poem (shown in purple), generative models—despite the overwhelming bias in their training data toward the contemporary—rhyme more often than poets born in the late twentieth century, and sometimes significantly more. Finally, even prompting the models for an unrhyming poem yields rhyming poems 50% of the time on average across the models, more than seven times as often as in the …

Continue →

Fine-tuning LLMs increases their propensity to rhyme

Finally may have figured out why LLMs rhyme so compulsively: instruction-tuning. Training an LLM to respond “helpfully” to user queries may push models into more “pleasing” aesthetic forms.

Graph showing that simple text completion models more accurately imitate the unrhymed form of C20 verse, whereas instruction-tuned models lapse into rhyme more often.   Caption to graph: Given the first 5 lines of 10-20 line poems from poets born in each century, 1600-2000, LLMs are prompted to "complete" the poem. Rhyme is measured by exact phoneme match in the rime of the final syllable (or syllables, if final syllable unstressed). Poems randomly sampled from Chadwyck-Healey poetry collections, with 600 poems for each model for each century. Results shown for actual poems as well as the LLM imitations. Poems "memorized" by the model are excluded.

Originally posted on Bluesky.

LLMs rewrite history on Random House publishing by race, 1950-2000

Ok 1 more: Random House’s actual pub record of its novelists’ race (from @richardjeanso.bsky.social’s Redlining Culture); vs. LLM models prompted to “recall from memory” that record (authors/races RH pub’d by yr). LLMs fantasize the same progress in diversity that So’s book debunks as industry myth.

Originally posted on Bluesky.

← All projects