Libretto — a structural-assessment environment for symbolic music

Abstract

Large language models can write music notation, but we lack a rigorous way to say whether what they write is structurally sound — and to steer them when it is not. We present Libretto, a validated, deterministic environment for symbolic music in which an LLM agent composes within a text grammar and is assessed and steered by a structural-assessment engine. The engine places any piece in a 39-axis fingerprint of token-derived, decorrelated structural features (rhythm, harmony, melody, texture, form), each scored as an empirical percentile against a frozen 1,523-song corpus; the axes are descriptive, not a learned “quality” score. We show the gates discriminate: real human music sits at a mean of 3.4 extreme axes (admitted), while degenerate generations pile up to 4–6, and a strict note-level copy gate cleanly separates novel material (≤0.29) from near-duplicates (≥0.31). On this substrate we add two interventions — a monotone-safe self-evolving loop that returns non-prescriptive “dosage” feedback to the agent, and mandatory retrieval of corpus-grounded idiom concepts plus centroid-nearest real exemplars. Across six applications (held-out gap-filling, cross-genre morphing, from-scratch generation, accompaniment reconstruction, anomaly detection, single-voice education), retrieval keeps generations on-genre (6→8 / 11) by de-degenerating the fingerprint, and the loop turns failures into passes — newgen convergence 64→100%, gap-fill gate 2→12/33, education 17→24/24. Because every axis is deterministic and grounded, a failure is fully legible: a generated gap-fill’s “~2-chords-per-bar conveyor belt” and near-stepwise melody are pinpointed to specific axes at the 95th–100th percentile. The grammar round-trips to MIDI with 0.019% note loss, and the validated core is frozen under a sha256 manifest.

The Libretto package (grammar codec, the 39-axis engine, the loop and retrieval, the applications) and the full results bundle are open.

What you ask of the notation	ABC (relative)	Libretto (absolute slot)
Recover every note's onset — real folk tunes (Nottingham DB)	O(n) prefix-sums	0 — slot is the onset
Align voices, “what sounds together” — full 1,523-song corpus	5,896,110 additions	0
LLM reads structure, tool-free benchmark (accuracy)	42%	100%
LLM quick-read — impossible / out-of-meter answers (“beat 7”)	27%	0% — unrepresentable

gate	real / good output	degenerate / copy	threshold
C1 non-degeneracy (#extreme axes/39)	real songs mean 3.4	gap-task FAIL 4.36 vs PASS 2.25; un-grounded newgen 5.9	budget 4–6
Novelty / copy-risk (note-level — one metric, per-task reference)	gap-fill & newgen passing ≤0.29; education drills 0.065	gap-fill flagged ≥0.31 (17/51 caught)	<0.30 vs corpus/answer · education <0.50 vs shown example
fidelity (grammar↔MIDI)	pitch exact, 0.019% loss	velocity/micro-timing/drums dropped by design	—

genre	n	genre	n
Pop/Rock	396	Funk/Soul/R&B	291
Electronic/Dance	241	Jazz	170
Folk/Country	149	Hip-Hop/Rap	71
Metal	63	Classical	43
Reggae/Ska	41	Blues/Gospel	30
Latin	30

Citation

@misc{xu2026librettogivingllmagents,
      title={Libretto: Giving LLM Agents a Sense of Musical Structure},
      author={Yichen Xu},
      year={2026},
      eprint={2606.22708},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2606.22708},
}

Libretto (谱文)

Abstract

How it works

How music is represented

Try the grammar

Listen

From-scratch generation (newgen)

Style morphing (morph)

Gap-filling (gaptask) — model fill vs. real held-out region

Accompaniment — reconstruct a removed track (without · with generated · original)

Education

Get started

Quickstart

The grammar in one glance

The 39-axis environment

Applications, one environment

Why not ABC or REMI?

The metric is not gamed

Steering the agent

A failure you can read off the score

Data & provenance

Code & data

Citation