ASI-Arch combined evidence index

105 candidates. Pinned data. One conservative list with explicit evidence gaps.

Method

1. Average all eight accuracy tasks equally, then percentile-normalize that average.
2. Percentile-normalize final training loss, lower being better.
3. Use only the judge component's mathematical bound: max(0,F−2) to min(1,F). Do not add raw composite fitness again.
4. Weight accuracy 60%, training loss 20%, judge 20%. Rank by the lower endpoint, a conservative maximin convention.

Weights are analyst choices. Score and rank intervals below reflect missing judge data and, for #562, missing final loss, not statistical confidence. The judge rubric also assesses performance, so it is not independent of benchmark quality. No matched PPL is scored because none was released.

Sensitivity

Weight-grid first-place counts: #879: 6 of 11 settings, #908: 3 of 11 settings, #1475: 2 of 11 settings. These are not probabilities.

Single ranked list

Each candidate: 8/8 accuracy metrics; 0/2 matched PPL values; judge component unpublished. Final training loss is available for 104/105; #562 is missing and receives an unknown 0–1 normalized-loss interval, not a measured zero. “Feasible rank” allows unknown judges to vary inside their bounds. Parameter metadata varies. Final-paper PPL is Wiki/LAMBADA at the separate 1.3B / 15B setting and is not weighted.

What this establishes

Use this list to prioritize tests. Accuracy alone favors #559; this default combined rule favors #879; other reasonable priorities favor #1475. No unique overall winner is established. Exact judge subratings, matched held-out PPL, and repeated-seed/equal-compute tests would make the decision firmer. There is no published final-scale 105-way comparison.

Full method

# ASI-Arch combined evidence index

This is a transparent decision aid, not an official benchmark or proof of a best architecture.

## Default rule
Average all eight accuracy tasks equally, counting each once. Convert this macro-average to a percentile within the 105-candidate cohort. Convert final training loss to a percentile with lower being better. Midrank percentile = (number below + (number tied - 1)/2)/(N-1), with direction reversed for loss.

Weights are 60% accuracy, 20% training loss, 20% AI-judge contribution. These analyst-selected priorities are not learned or endorsed by the paper. Parameter counts are shown, not penalized.

## Missing judge component
Do NOT add the composite fitness again. Its documented raw sum is F = L + B + J, where normalized loss, benchmark and judge components are each in [0,1]. The individual values are unpublished, so max(0,F-2) <= J <= min(1,F). Use this mathematical bound, not a fabricated judge measurement.

Rank by the LOWER endpoint of 100*(0.6*accuracy_percentile + 0.2*inverse_loss_percentile + 0.2*J). This is a conservative maximin rule, not a claim that the lower endpoint is the actual judge score. Both endpoints and feasible rank bounds are supplied. These are NOT statistical confidence intervals. The bounds assume stored fitness follows the documented ranges. The code requests a 1-10 judge range but its Pydantic schema does not enforce numeric limits.

The fitness's quantitative components are not added twice. However, the judge's own rubric includes performance (30%), innovation (25%) and complexity (45%). Conceptual correlation remains; its unpublished subratings cannot be separated.

## Perplexity and other missing data
All 105 have eight accuracy metrics. Final training loss is finite for 104; #562 has NaN, so its normalized loss is an unknown interval [0,1], not a measured zero. Loss percentiles use the 104 observed rows. BOTH held-out perplexity columns are NaN for ALL 105. No zero replacement, cross-model imputation or fabricated PPL is used. Training loss is not renamed held-out perplexity, and exponentiating it would not add independent evidence.

The final paper's 1.3B / 15B-token results are a different experiment. Only #879 and #703 can be exactly matched to exported records. Their published PPL and 14-task averages appear in supplementary columns and have ZERO weight here. Other named finalists are not guessed from similar names. No cross-scale average is taken. A genuinely complete all-data 105-way ranking cannot be recovered.

## Sensitivity and uncertainty
Alternative weights and normalizations are shown. The winner changes. The default is one disclosed convention, not an objectively unique answer. Weight-grid win frequencies are not probabilities. Feasible rank bounds reflect unknown judge values only, not seeds, test sampling or data contamination.

Keep all 105 unique IDs and internal names distinct, including variants sharing display names. There are 100 display names. Exported parameter metadata ranges from 411.95M to 817.77M; the paper's nominal setting does not establish equal parameters or compute. No repeated-seed uncertainty is reported for these rows.

## Reproduction
Run python build_unified_index.py beside gallery105_results_source.json. Python standard library only. The source excerpt preserves metadata and raw result fields from the pinned JSON, excluding program and diagram strings.

## Sources
- https://github.com/GAIR-NLP/ASI-Arch/blob/3113c51b04a1d4a7520a3b0238c6a08d3194d58f/data/106.json
- https://github.com/GAIR-NLP/ASI-Arch/blob/3113c51b04a1d4a7520a3b0238c6a08d3194d58f/database/candidate_manager.py#L76-L127
- https://github.com/GAIR-NLP/ASI-Arch/blob/3113c51b04a1d4a7520a3b0238c6a08d3194d58f/database/evaluate_agent/model.py
- https://github.com/GAIR-NLP/ASI-Arch/blob/3113c51b04a1d4a7520a3b0238c6a08d3194d58f/index.html#L1994-L2035
- https://arxiv.org/html/2507.18074v2#S4
- https://arxiv.org/html/2507.18074v2

Primary sources

• Pinned results
• Fitness code
• Judge rubric
• Revised paper