# ASI-Arch combined evidence index

This is a transparent decision aid, not an official benchmark or proof of a best architecture.

## Default rule
Average all eight accuracy tasks equally, counting each once. Convert this macro-average to a percentile within the 105-candidate cohort. Convert final training loss to a percentile with lower being better. Midrank percentile = (number below + (number tied - 1)/2)/(N-1), with direction reversed for loss.

Weights are 60% accuracy, 20% training loss, 20% AI-judge contribution. These analyst-selected priorities are not learned or endorsed by the paper. Parameter counts are shown, not penalized.

## Missing judge component
Do NOT add the composite fitness again. Its documented raw sum is F = L + B + J, where normalized loss, benchmark and judge components are each in [0,1]. The individual values are unpublished, so max(0,F-2) <= J <= min(1,F). Use this mathematical bound, not a fabricated judge measurement.

Rank by the LOWER endpoint of 100*(0.6*accuracy_percentile + 0.2*inverse_loss_percentile + 0.2*J). This is a conservative maximin rule, not a claim that the lower endpoint is the actual judge score. Both endpoints and feasible rank bounds are supplied. These are NOT statistical confidence intervals. The bounds assume stored fitness follows the documented ranges. The code requests a 1-10 judge range but its Pydantic schema does not enforce numeric limits.

The fitness's quantitative components are not added twice. However, the judge's own rubric includes performance (30%), innovation (25%) and complexity (45%). Conceptual correlation remains; its unpublished subratings cannot be separated.

## Perplexity and other missing data
All 105 have eight accuracy metrics. Final training loss is finite for 104; #562 has NaN, so its normalized loss is an unknown interval [0,1], not a measured zero. Loss percentiles use the 104 observed rows. BOTH held-out perplexity columns are NaN for ALL 105. No zero replacement, cross-model imputation or fabricated PPL is used. Training loss is not renamed held-out perplexity, and exponentiating it would not add independent evidence.

The final paper's 1.3B / 15B-token results are a different experiment. Only #879 and #703 can be exactly matched to exported records. Their published PPL and 14-task averages appear in supplementary columns and have ZERO weight here. Other named finalists are not guessed from similar names. No cross-scale average is taken. A genuinely complete all-data 105-way ranking cannot be recovered.

## Sensitivity and uncertainty
Alternative weights and normalizations are shown. The winner changes. The default is one disclosed convention, not an objectively unique answer. Weight-grid win frequencies are not probabilities. Feasible rank bounds reflect unknown judge values only, not seeds, test sampling or data contamination.

Keep all 105 unique IDs and internal names distinct, including variants sharing display names. There are 100 display names. Exported parameter metadata ranges from 411.95M to 817.77M; the paper's nominal setting does not establish equal parameters or compute. No repeated-seed uncertainty is reported for these rows.

## Reproduction
Run python build_unified_index.py beside gallery105_results_source.json. Python standard library only. The source excerpt preserves metadata and raw result fields from the pinned JSON, excluding program and diagram strings.

## Sources
- https://github.com/GAIR-NLP/ASI-Arch/blob/3113c51b04a1d4a7520a3b0238c6a08d3194d58f/data/106.json
- https://github.com/GAIR-NLP/ASI-Arch/blob/3113c51b04a1d4a7520a3b0238c6a08d3194d58f/database/candidate_manager.py#L76-L127
- https://github.com/GAIR-NLP/ASI-Arch/blob/3113c51b04a1d4a7520a3b0238c6a08d3194d58f/database/evaluate_agent/model.py
- https://github.com/GAIR-NLP/ASI-Arch/blob/3113c51b04a1d4a7520a3b0238c6a08d3194d58f/index.html#L1994-L2035
- https://arxiv.org/html/2507.18074v2#S4
- https://arxiv.org/html/2507.18074v2