A validated, deterministic structural-assessment environment for symbolic music, within which an LLM agent composes and is steered.
UC Berkeley
Large language models can write music notation, but we lack a rigorous way to say whether what they write is structurally sound — and to steer them when it is not. We present Libretto, a validated, deterministic environment for symbolic music in which an LLM agent composes within a text grammar and is assessed and steered by a structural-assessment engine. The engine places any piece in a 29-axis fingerprint of token-derived, decorrelated structural features (rhythm, harmony, melody, texture, form), each scored as an empirical percentile against a frozen 314-song corpus; the axes are descriptive, not a learned “quality” score. We show the gates discriminate: real human music sits at a mean of 3.4 extreme axes (admitted), while degenerate generations pile up to 4–6, and a strict note-level copy gate cleanly separates novel material (≤0.29) from near-duplicates (≥0.31). On this substrate we add two interventions — a monotone-safe self-evolving loop that returns non-prescriptive “dosage” feedback to the agent, and mandatory retrieval of corpus-grounded idiom concepts plus centroid-nearest real exemplars. Across four applications (held-out gap-filling, component morphing, from-scratch generation, single-voice pedagogy), retrieval triples from-scratch pass rate (2/8→6/8) by de-degenerating the fingerprint, and the loop roughly triples gap-filling pass rate (12%→39%) and lifts from-scratch generation to 94%. Because every axis is deterministic and grounded, a failure is fully legible: a generated gap-fill’s “~2-chords-per-bar conveyor belt” and all-stepwise melody are pinpointed to specific axes at the 95th–100th percentile. The grammar round-trips to MIDI with 0.019% note loss, and the validated core is frozen under a sha256 manifest.
An LLM composes in the grammar; the deterministic engine scores it on the 29 axes and steers it — retrieve, compose, measure, and loop until it clears the gates.
One compact, human-legible text grammar — exact pitch, rhythm, and voices, and nothing else. The same text round-trips to MIDI; the LLM reads and writes only this.
KEY: C major | METER: 4/4 | TEMPO: 120 | GRID: 16th | BARS: 2 VOICES: Melody, Piano, Bass [C] Melody: E4@1>2 G4@5>2 C5@9>4 G4@13>2 Piano : C4+E4+G4@1>8 C4+E4+G4@9>8 Bass : C2@1>4 G2@9>4 [G] Melody: D4@1>2 G4@5>2 B4@9>4 D5@13>2 Piano : G3+B3+D4@1>8 G3+B3+D4@9>8 Bass : G2@1>4 D3@9>4
Note content is exact and reversible; velocity, micro-timing, and timbre are intentionally dropped.
Edit the grammar on the left — it renders to a piano-roll instantly, and you can play it. Runs entirely in your browser (no server).
↑ these structural axes are read directly from the text — every note's bar and slot is explicit, so they're exact lookups (off-beat notes are outlined). In ABC or raw MIDI you'd first have to reconstruct absolute timing (relative durations / tick deltas) before any of these is even well-defined.
ABC is the standard text music format, but it stores each note's onset as a relative duration — the onset is a running within-bar sum (O(n) to recover). Libretto stores an absolute slot (O(1) lookup). Same music, same information; only the representation differs.
| What you ask of the notation | ABC (relative) | Libretto (absolute slot) |
|---|---|---|
| Recover every note's onset — real folk tunes (Nottingham DB) | O(n) prefix-sums | 0 — slot is the onset |
| Align voices, “what sounds together” — full 314-song corpus | 5,896,110 additions | 0 |
| LLM reads structure, tool-free benchmark (accuracy) | 42% | 100% |
| LLM quick-read — impossible / out-of-meter answers (“beat 7”) | 27% | 0% — unrepresentable |
Two tiers. The top two rows are deterministic properties of
the encoding, measured on real tunes and the whole corpus (no model). The bottom two are tool-free LLM
reads of the same music in each notation: the same reasoning is more accurate in Libretto, and
never lands out-of-meter — an absolute slot cannot name a beat outside its bar, so the hallucination is
eliminated by construction, not merely rarer. ABC's honest advantage is being more token-terse. LLM rows are
illustrative (small n, one model); full method & caveats in paper_data/grammar_compare/.
Any piece is placed in a coordinate system of 29 token-derived structural axes, each a percentile against a frozen 314-song corpus. The axes are decorrelated (max |r| = 0.77, 12 of 406 pairs ≥ 0.5) and carry real — if soft — genre signal.
Percentiles are scored against a frozen reference of 314 real songs that we curate
across 8 genres from the Clean MIDI subset (~17k files,
2,199 artists) of the Lakh MIDI Dataset (Raffel, 2016; CC-BY 4.0). The 314-song selection and genre
labels are ours; the source MIDIs — and their artist/title, “with some inaccuracy”
— come from clean_midi (community transcriptions, hence anonymized Part{n} voices and
occasional near-duplicates, which the copy gate accounts for).
| genre | n | genre | n |
|---|---|---|---|
| Pop/Rock | 70 | Funk/Soul | 30 |
| Electronic | 34 | Jazz | 28 |
| Classical | 30 | Latin/Reggae/World | 28 |
| Folk/Country | 25 | Film/Score | 10 |
The gates are calibrated against ground truth and discriminate real music from degenerate or copied output — a deterministic, auditable target rather than a learned score.
| gate | real / good output | degenerate / copy | threshold |
|---|---|---|---|
| C1 non-degeneracy (#extreme axes/29) | real songs mean 3.4 | gap-task FAIL 4.36 vs PASS 2.25; un-grounded newgen 5.9 | budget 4–6 |
| Novelty / copy-risk (note-level — one metric, per-task reference) | gap-fill & newgen passing ≤0.29; education drills 0.065 | gap-fill flagged ≥0.31 (17/51 caught) | <0.30 vs corpus/answer · education <0.50 vs shown example |
| fidelity (grammar↔MIDI) | pitch exact, 0.019% loss | velocity/micro-timing/drums dropped by design | — |
Frozen core certified — distribution + encoder/decoder + metric definitions match a pinned sha256 manifest (v2.0.0); guard tests for the 29-axis shape, exact round-trip, determinism, and copy self-match all pass.
Two interventions sit on the substrate: mandatory retrieval (corpus idiom concepts + centroid-nearest real exemplars) and a monotone-safe self-evolving loop (≤3 rounds, pick-best) that feeds non-prescriptive dosage feedback back to the agent.
One grammar and one assessment engine support held-out gap-filling (n=51), component morphing (n=21), from-scratch generation (n=8/16), and single-voice pedagogy (n=32).
Audio rendered from the generated grammar (grammar → MIDI → General-MIDI synthesis). A few typical samples per task; the rest are browsable below.
One typical sample per genre. In the full list below, s1 / s2 are independent samples of that genre — separate generations that use the same prompt and the same retrieval, so they may sound similar (same style neighborhood; measured pairwise note-overlap ≈ 0.15, well under the 0.30 copy gate, so they are still distinct pieces).
Each clip glides from one style to another (real source → generated transition → real target).
A/B the model's generated region against the true held-out answer for the same gap.
A monophonic reading study for every major key, in both the treble and the bass clef — each roams its clef's register, mixes rhythms, and is distinct from the others (no repeats). Listen, or download the MIDI and the engraved score (PDF). Scroll within the panel for all 24.
Because every axis is deterministic and grounded, the metric’s verdict is not a black box. A failing gap-fill is legible as visible musical structure, pinned to specific axes.
Grounded diagnosis
FAILS with 6 extreme axes vs a budget of 3:
100th pct99th96th95th100th3rdNot a copy (copy-risk 0.134), rhythmically aligned (beat 86%) — the failure is purely structural, and every flagged axis is visible in the roll.
The package onboarding (libretto/ONBOARDING.md) — structure, what each task
implements, and how to reproduce. Scroll within the panel.
The Libretto package (grammar codec, the 29-axis engine, the loop and retrieval, the four applications) and the full results bundle are open.
@misc{xu2026librettogivingllmagents,
title={Libretto: Giving LLM Agents a Sense of Musical Structure},
author={Yichen Xu},
year={2026},
eprint={2606.22708},
archivePrefix={arXiv},
primaryClass={cs.SD},
url={https://arxiv.org/abs/2606.22708},
}