ContinuousBench: does DP synthetic text actually transfer knowledge?

June 1, 2026 · paper · dataset · eval harness

Co-led by Peihan Liu and Alex Bie; joint work with Lucas Rosenblatt, Weiwei Kong, Natalia Ponomareva, Gautam Kamath, Rachel Cummings, Roxana Geambasu, Yu Gan, and Lillian Tsai.

What's wrong with current evaluations for DP synthetic text?

Existing evaluations often miss this for three reasons:

1. The knowledge already in the base model. On IMDB, Yelp, OpenReview, and other familiar classification-style benchmarks, modern LMs can already solve these out of the box, even at ~1B scale. Hence, if a modern pretrained LM gets good numbers after training on DP synthetic data, that does not necessarily mean the synthetic data taught it anything new.
2. The metrics reward style, not knowledge. MAUVE, BLEU, classification accuracy, etc. can all improve when the synthetic text only gets the right "vibe". A DP synthetic copy can score well on these metrics while losing the specific facts in the source corpus, so good metric scores do not necessarily mean real information transfer.
3. Teacher-student distillation muddies the signal. Synthetic data is often generated by a stronger model and then used to train a weaker model for eval. The gains may come from the stronger model teaching the weaker one, rather than from knowledge learned from the private corpus.

The visible symptom is saturation. On the benchmarks the field actually uses, every method sits within a few points of the ceiling:

ContinuousBench

Each release packages three things: a fresh training corpus, a derived short-answer QA set, and a standardized training/eval harness. The benchmark is built around three design goals:

• Freshness. The corpus is constructed to be unseen by the tested model. In Geminon, we procedurally generate a Pokémon-inspired fictional world, and generate articles from it with Gemini. In News, we source news articles after the training cutoff from periodically released CommonCrawl-News dumps. The QA set tests knowledge from that corpus.
• Grounded QA. Each question is linked to supporting records in the corpus and has a normalized short answer, so evaluation can be automatic and verifiable.
• DP-learnable facts by construction. Each main test question is supported by at least 200 independent records, so the target is population-level knowledge rather than singleton memorization. We also keep a singleton split, where each fact appears in only one record, as a privacy sanity check.

To avoid teacher-student confounds, we use the same base checkpoint for the generator and the downstream evaluator. Geminon gives us a fully controlled fictional domain for studying DP learnability, while News gives us a continuously collected real-world corpus with natural messiness and time-evolving facts.

The gap under differential privacy

The main result is simple: synthesis works, while DP synthesis mostly does not. In the standard “fine-tune then sample” pipeline, public synthetic data (ε = ∞) nearly matches the real corpus. With DP, accuracy collapses. On Geminon, DP synthesis reaches only 4% at ε = 100 and 3% at ε = 10. On News, it reaches 26% at ε = 100 and 20% at ε = 10 — barely above no-training in both cases. Another promising training-free DP method, Private Evolution, does not close the gap either: it falls below no-training on both Geminon and News.

The pattern is clear. Today's DP synthetic text can preserve the form of a corpus, its style, topics, and document structure, while losing the factual content that makes the data valuable. On Geminon, for example, going from ε = ∞ to ε = 10 only drops MAUVE by 0.10, but drops QA accuracy by 89%. See example. If a method cannot recover facts repeated across hundreds of independent records, it has not really preserved the corpus :(