September 2026
What if AutoResearch experimented with data?
Provena's AutoCurator experimented with how to curate training data — and discovered a small, high-value subset that improved model performance.
161
Experiments
5.6%
Data selected
+5.2pp
ARC-Easy benchmark
Gain held
At 1-3x training budgets
AutoCurator
Improving Karpathy's AutoResearch with data curation
01
Making data the variable
Karpathy's AutoResearch showed that AI agents can accelerate the path to better models by experimenting with model architecture and training.
Using the same NVIDIA ClimbMix training dataset, AutoCurator explored ways to select and curate the data, while keeping the model architecture and training compute fixed.
Could an autonomous researcher discover better training data?
02
What AutoCurator found
Across 161 experiments, most interventions failed. Two improved on the running best.
AutoCurator discovered a combination of three data-quality signals and thresholds that selected just 5.6% of the original data pool. The model was then trained exclusively on this curated subset.
With the same training-token budget as the baseline, ARC-Easy performance improved from 32.5% to 37.7% (+5.2pp).
| Benchmark | Baseline | AutoCurator | Change |
|---|---|---|---|
| ARC-Easy | 32.5% | 37.7% | +5.2 points |
| PIQA | 55.0% | 53.3% | −1.7 points |
| HellaSwag | 27.3% | 27.2% | flat |
| 3-benchmark mean | 38.3% | 39.4% | +1.1 points |
The ARC-Easy advantage held at 1×, 2× and 3× training budgets, without further degradation on the other benchmarks.
03
Validating the results
To reduce the risk of overfitting, we followed the research setup from NVIDIA's Climb paper , which introduced the training dataset.
We optimized on a validation set and confirmed that the gain held on a separate test set (+5.2pp validation, +5.0pp test). We also checked for test-data contamination, used general data-quality signals rather than benchmark-specific tuning, and trained on each document only once.
Finally, the best selection strategy transferred to a 4.4× larger scale, with even stronger results.
04
What we learned
- 01
Better data unlocks more intelligence. AutoCurator identified just 5.6% of the original data pool that improved model performance at the same training-token budget.
- 02
Most experiments won't work — and that's the point. Of 161 experiments, only two improved on the running best. Autonomous research makes it practical to test many hypotheses, explore a larger search space, and pursue promising directions.
- 03
Research judgment matters as much as experimentation. AutoCurator independently developed validation checks, but needed guidance on when to explore broadly versus investigate promising directions more deeply. Improving that judgment is key to building better autonomous researchers.
05
What's next
These results establish a promising proof of concept on an 11.5M-parameter model. Next, we're extending AutoCurator to larger models, more datasets and new research questions — while developing better research agents and the infrastructure to scale them.
06 · Help us test it
Push the next experiment further with us
We’re looking for model builders and research teams to push this further with us.
Through Provena Research Partners, we'll contribute up to $10K in compute + research support to experiment on a model, training data and benchmark you care about.