Paper Brief — AutoSynthesis: An Agentic System That Automates Meta-Analysis From Research Question to Published Report

What if you could transform a research question into a fully synthesized meta-analysis report without weeks of manual literature review? A new paper from researchers at LMU Munich, Politecnico di Milano, and MIT CSAIL introduces AutoSynthesis, an end-to-end multi-agent system that does exactly that. Given a natural language research question, it formulates search strategies, retrieves scientific literature, screens studies, extracts statistics, computes effect sizes, and produces a PRISMA-aligned meta-analysis report automatically. In tests across 28 studies, its pooled effect estimates came within 0.12 Hedges' g of expert-conducted manual meta-analyses, putting automated evidence synthesis on par with human researchers for the first time.

How AutoSynthesis Works: A Multi-Agent Architecture

AutoSynthesis is built on LangGraph and deploys a chain of specialized AI agents, each handling one step of the meta-analysis workflow. First, a planning agent translates the research question into a structured protocol specifying the meta-analysis objective, treatment and control conditions, eligibility criteria, target outcomes, and candidate moderators. This protocol becomes the shared reference point for all downstream agents.

Next, a search agent generates Boolean queries and retrieves papers from multiple scientific databases including arXiv, Semantic Scholar, CrossRef, OSF Preprints, and PubMed. A paper reading agent then fetches full texts and converts PDFs to structured Markdown using a cascading pipeline of MinerU, LlamaParse, Mistral OCR, and conventional text extraction as fallbacks. The eligibility assessment agent reviews each full text against inclusion criteria, logging exclusion reasons for full audit traceability.

The most novel part is the statistical pipeline. A study identification agent breaks papers into independent empirical components, mapping treatment groups, control conditions, and outcome variables. A statistical result extraction agent then performs a two-stage extraction: first identifying relevant statistical information, then extracting precise numerical values. A validation agent cross-checks the extracted statistics against the source text, and an analysis agent computes standardized effect sizes. Finally, a random-effects meta-analysis agent pools the effects, assesses publication bias, performs heterogeneity analysis, and produces the final report.

Key Results: Matching Expert Meta-Analysts

The researchers evaluated AutoSynthesis on a published benchmark meta-analysis from Hoelbling et al. (2025) examining the persuasive power of large language models. AutoSynthesis recovered 71.4 percent of the studies included in the human-conducted benchmark. Its pooled effect estimate deviated by only 0.12 Hedges' g from the expert analysis, a degree of variation that falls within the broadly accepted tolerance region of plus or minus 0.20 Cohen's d used in human reanalysis assessments.

The primary source of discrepancy was not in the statistical computation but in literature retrieval: some relevant studies were simply not found during automated searching, or their full texts were behind paywalls. When the same set of studies was available, the statistical outputs closely matched the human benchmark. This is an important finding because it isolates the bottleneck: the AI does the statistics well, but the search and retrieval layer is where improvement is most needed.

AutoSynthesis also supports heterogeneity analysis to examine how effect sizes vary across moderators, risk-of-bias assessment using established criteria, and publication bias diagnostics including funnel plots and Egger's regression. Every intermediate decision is logged, creating a fully auditable record of the meta-analysis process from initial search to final report.

Limitations and the Path Forward

The paper is transparent about its limitations. The evaluation was limited to a single benchmark meta-analysis, so robust performance across all scientific domains, study designs, and reporting conventions is not yet established. Automated literature retrieval remains constrained by paywalled content and PDF parsing reliability. Some studies do not report sufficient quantitative information to compute standardized effect sizes, a challenge shared with human meta-analysts.

The authors explicitly state that AutoSynthesis is not meant to replace expert reviewers. Many steps in the meta-analysis workflow, such as formulating the research question, specifying inclusion criteria, selecting moderators, and interpreting findings, require substantive expert judgment. The system is designed as a supporting tool, not a shortcut. The most promising near-term application is continuous updates of existing high-quality meta-analyses, where the methodological framework is already defined and validated, allowing newly published studies to be incorporated with minimal manual effort. This extends the concept of living systematic reviews to living meta-analysis.

What This Means for Builders

For founders building data-intensive products, AutoSynthesis demonstrates that agentic AI can automate workflows that previously required PhD-level domain expertise. The system directly replaces weeks of manual literature review, making evidence-based decision-making accessible to startups without dedicated research teams. If you are building in healthtech, edtech, or any domain where evidence synthesis matters, consider how automated meta-analysis could accelerate your product development cycle.

For AI infrastructure builders, the modular architecture is instructive. AutoSynthesis's cascading PDF parsing pipeline, multi-database search federation, and separation of LLM-based reasoning from deterministic statistical computation are design patterns worth studying. The system uses open-weight LLMs in addition to proprietary models, suggesting that reproducible scientific AI systems can be built without depending entirely on API-bound models.

For investors and product strategists, the existence of AutoSynthesis signals that the evidence synthesis market, currently dominated by expensive manual processes and proprietary tools, is ripe for disruption. A system that can produce PRISMA-aligned meta-analyses from a single natural language question has obvious applications in clinical research, policy analysis, education, and market intelligence. The gap between what AutoSynthesis does today and what a commercial product could do is narrowing fast.