September 2026

What if AutoResearch experimented with data?

Provena's AutoCurator experimented with how to curate training data — and discovered a small, high-value subset that improved model performance.

161

Experiments

5.6%

Data selected

+5.2pp

ARC-Easy benchmark

Gain held

At 1-3x training budgets

AutoCurator

Improving Karpathy's AutoResearch with data curation

DiscardedSuccessful
AutoCurator experiment progressA stepped line chart showing two retained improvements across 161 experiments, ending with 5.6 percent of training data selected and a 5.2 percentage point benchmark gain.−0.10−0.11−0.12−0.13−0.14−0.15−0.16BENCHMARK MARGIN (HIGHER IS BETTER)0326496128161EXPERIMENTS #10% data selected5.6% data selected

01

Making data the variable

Karpathy's AutoResearch showed that AI agents can accelerate the path to better models by experimenting with model architecture and training.

Using the same NVIDIA ClimbMix training dataset, AutoCurator explored ways to select and curate the data, while keeping the model architecture and training compute fixed.

Could an autonomous researcher discover better training data?

02

What AutoCurator found

Across 161 experiments, most interventions failed. Two improved on the running best.

AutoCurator discovered a combination of three data-quality signals and thresholds that selected just 5.6% of the original data pool. The model was then trained exclusively on this curated subset.

With the same training-token budget as the baseline, ARC-Easy performance improved from 32.5% to 37.7% (+5.2pp).

BenchmarkBaselineAutoCuratorChange
ARC-Easy32.5%37.7%+5.2 points
PIQA55.0%53.3%−1.7 points
HellaSwag27.3%27.2%flat
3-benchmark mean38.3%39.4%+1.1 points

The ARC-Easy advantage held at 1×, 2× and 3× training budgets, without further degradation on the other benchmarks.

03

Validating the results

To reduce the risk of overfitting, we followed the research setup from NVIDIA's Climb paper , which introduced the training dataset.

We optimized on a validation set and confirmed that the gain held on a separate test set (+5.2pp validation, +5.0pp test). We also checked for test-data contamination, used general data-quality signals rather than benchmark-specific tuning, and trained on each document only once.

Finally, the best selection strategy transferred to a 4.4× larger scale, with even stronger results.

04

What we learned

  1. 01

    Better data unlocks more intelligence. AutoCurator identified just 5.6% of the original data pool that improved model performance at the same training-token budget.

  2. 02

    Most experiments won't work — and that's the point. Of 161 experiments, only two improved on the running best. Autonomous research makes it practical to test many hypotheses, explore a larger search space, and pursue promising directions.

  3. 03

    Research judgment matters as much as experimentation. AutoCurator independently developed validation checks, but needed guidance on when to explore broadly versus investigate promising directions more deeply. Improving that judgment is key to building better autonomous researchers.

05

What's next

These results establish a promising proof of concept on an 11.5M-parameter model. Next, we're extending AutoCurator to larger models, more datasets and new research questions — while developing better research agents and the infrastructure to scale them.

06 · Help us test it

Push the next experiment further with us

We’re looking for model builders and research teams to push this further with us.

Through Provena Research Partners, we'll contribute up to $10K in compute + research support to experiment on a model, training data and benchmark you care about.

Become a Research Partner