AutoTrainess: Teaching Language Models to Improve Language Models Autonomously

arXiv 2026 51.6 method, system

TLDR

AutoTrainess is an LM agent that automates post-training of language models using agent-computer interfaces, outperforming CLI-only baselines on PostTrainBench.

Reasoning

Strengths: introduces a structured approach to automate LM post-training, shows empirical gains across models. Weaknesses: limited to post-training, not general scientific discovery; evaluation on a single benchmark.

Read-first score

Read-first score 51.6, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 31.

Recency 8%

100

Uses a gentle age decay so recent papers surface without erasing older foundations. 2026

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=baseline,benchmark,evaluation,experiment

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=checkpoint,code,repository

Topical relevance 42%

25.8

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 98.

Keyword Scores

automated experimentation

experiment design agent

research automation

automated research

autonomous research agent

AI for scientific research

AI scientist

scientific discovery agent

automated scientific discovery

literature review agent

survey generation

paper writing agent

Deep Analysis

Innovations

Agent-computer interfaces that expose post-training operations (planning, data preparation, training, evaluation, logging) as structured workflows rather than raw CLI actions.
Externalization of human expertise as explicit workflows, rules, and execution constraints to guide the agent toward reliable training behavior.
AutoTrainess system that enables language model agents to autonomously improve language models through iterative post-training.

Methodology

AutoTrainess provides a repository of agent-computer interfaces that structure the post-training pipeline into planning, data preparation, training, evaluation, and logging steps, embedding human-derived workflows and constraints. The agent is evaluated on PostTrainBench against CLI-only baselines using models like GPT-5.4 (Codex) and DeepSeek-V4-Flash (OpenCode).

Key Results

AutoTrainess with GPT-5.4 (Codex) achieves an average score of 26.94 on PostTrainBench, outperforming the CLI-only baseline (23.21), and improves DeepSeek-V4-Flash (OpenCode) from 12.13 to 19.58.

Resources

Paper