Architecture Research Gallery

For each selected group, each component equals the lowest measured value in that group divided by the model’s measured value. All three inputs are lower-is-better: mean test cross-entropy, mean training CPU seconds, and mean prefill-plus-64-decode latency. Index = 100 × weighted average of those components. Weights are normalized to sum to one. The default 60/20/20 is an analyst-selected preference, not a learned or official score.

The quality component is a numerical transform of loss, not a ratio of intelligence. An index of 100 only means best on every selected component within this tiny group. Near-equal training times contribute near-equally; trivial timing variation is not magnified to a full 0–100 range. Weight zero omits a component’s contribution; all-zero weights leave the comparison unranked.

All displayed pilot rows have 3/3 inputs. A missing input leaves the index unavailable rather than filling it with zero. No confidence interval is claimed. Three seeds and noisy CPU timings cannot establish a robust global winner. Changing group or prompt length changes the reference and can change the ordering.

Training budgets were approximately two CPU seconds, not time to matched quality. Serving measurements were not quality-matched. Expanding to all four controls introduces different sizes and inductive biases, including a small untuned position-free attention control. GPU performance, energy, financial costs and frontier capability are unmeasured and excluded. The 200 tiny screens remain unranked because their quality evidence is inadequate.

View every raw pilot measurement

Requires PyTorch. This is the actual experimental adapter source, not official Sakana language-model code. It defines the models; use the research harness for training, data and checkpoints. Download models200.py · Upstream Apache-2.0 license

Archived v1 code below has a conditioning-coverage mismatch. Use corrected v2 source or a candidate’s complete file instead.

"""Experimental paper-derived DiffusionBlocks/AR/SAT/NAT CPU adapters.

Not official Sakana AR code. Equations/noise-band design follow paper2506.14202v3
and SakanaAI/DiffusionBlocks@ade0b08. Upstream Apache2license is retained separately.
All dense projection 'binary' operations emulate STE signs in FP32, not packed.
Generation takes prefixes only, never reference continuations.
"""
import math
from statistics import NormalDist
import torch
from torch import nn
from torch.nn import functional as F
N=NormalDist();PM=-1.2;PS=1.2;SMIN=.002;SMAX=80.
def cdf(s):return N.cdf((math.log(s)-PM)/PS)
def sigma(p):return math.exp(PM+PS*N.inv_cdf(p))
def boundaries(n):return [sigma(cdf(SMIN)+(cdf(SMAX)-cdf(SMIN))*i/n) for i in range(n+1)]
class Linear(nn.Linear):
    def __init__(self,a,b,binary=False):super().__init__(a,b,bias=False);self.binary=binary
    def forward(self,x):
        w=self.weight
        if self.binary:
            quant=torch.where(w>=0,1.,-1.)*w.detach().abs().mean(1,keepdim=True);w=w+(quant-w).detach()
        return F.linear(x,w)
class RMS(nn.Module):
    def __init__(self,d):super().__init__();self.weight=nn.Parameter(torch.ones(d))
    def forward(self,x):return x*torch.rsqrt(x.square().mean(-1,keepdim=True)+1e-5)*self.weight
class Block(nn.Module):
    def __init__(self,d,h,binary):
        super().__init__();self.h=h;self.d=d;self.n1=RMS(d);self.n2=RMS(d)
        self.qkv=Linear(d,3*d,binary);self.out=Linear(d,d,binary);self.up=Linear(d,2*d,binary);self.gate=Linear(d,2*d,binary);self.down=Linear(2*d,d,binary)
        self.time=nn.Linear(1,d,bias=False)
    def rope(self,x,pos):
        # Head widths2,3,4,...: rotate the even prefix; retain any odd final dim.
        k=(x.shape[-1]//2)*2
        f=10000.**(-torch.arange(0,k,2,device=x.device,dtype=x.dtype)/max(1,k));a=pos[:,None]*f
        u,v=x[...,:k:2],x[...,1:k:2];r=torch.stack((u*a.cos()-v*a.sin(),u*a.sin()+v*a.cos()),-1).flatten(-2)
        return torch.cat([r,x[...,k:]],-1)
    def forward(self,x,mask,pos,noise=None):
        if noise is not None:x=x+self.time(noise)
        b,t,d=x.shape;q,k,v=self.qkv(self.n1(x)).chunk(3,-1);q,k,v=[z.reshape(b,t,self.h,d//self.h).transpose(1,2) for z in (q,k,v)]
        q,k=self.rope(q,pos),self.rope(k,pos);a=F.scaled_dot_product_attention(q,k,v,attn_mask=mask).transpose(1,2).reshape(b,t,d)
        x=x+self.out(a);z=self.n2(x);return x+self.down(F.silu(self.gate(z))*self.up(z))
class Architecture(nn.Module):
    def __init__(self,spec):
        super().__init__();self.spec=spec;d=spec['width'];self.emb=nn.Embedding(256,d);self.norm=RMS(d)
        self.blocks=nn.ModuleList([Block(d,spec['attention_heads'],spec['binary_transformer_projections']) for _ in range(spec['depth'])]);self.diff=spec['family'].startswith('diffusionblocks');self.bands=boundaries(spec['depth'])
    def logits(self,x):return F.linear(self.norm(x),self.emb.weight)/math.sqrt(self.spec['width'])
    def forward(self,tokens):
        assert not self.diff,'Diffusion model has no ordinary AR likelihood forward'
        x=self.emb(tokens);t=tokens.shape[1];i=torch.arange(t);mask=i[:,None]>=i[None,:]
        if self.spec['family']=='ar_local16':mask&=(i[:,None]-i[None,:]<16)
        for block in self.blocks:x=block(x,mask,i)
        return self.logits(x)
    def training_mask(self,t):
        # x_i is the previous clean token for target y_i. Clean keys <=groupstart
        # are permitted; noisy keys only from the current generated group.
        k=self.spec['generation_block_size'];i=torch.arange(t);group=(i//k)*k
        mask=torch.zeros((2*t,2*t),dtype=torch.bool);mask[:t,:t]=i[:,None]>=i[None,:]
        mask[t:,:t]=i[None,:]<=group[:,None];mask[t:,t:]=(i[None,:]//k)==(i[:,None]//k)
        return mask
    def denoise(self,clean,z,sig,block,training=True):
        b,t,d=z.shape;past=self.emb(clean);sd=.5
        cin=torch.rsqrt(sig.square()+sd**2);cout=sig*sd*cin;cskip=sd**2/(sig.square()+sd**2)
        x=torch.cat([past,z*cin[:,None,None]],1);n=past.shape[1]
        cond=x.new_zeros((b,n+t,1));cond[:,n:]=(.25*sig.log())[:,None,None]
        if training:mask=self.training_mask(t);pos=torch.cat([torch.arange(n),torch.arange(t)+1])
        else:
            mask=torch.zeros((n+t,n+t),dtype=torch.bool);mask[:n,:n]=torch.ones((n,n),dtype=torch.bool).tril();mask[n:,:]=True;pos=torch.arange(n+t)
        hidden=self.blocks[block](x,mask,pos,cond)[:,n:]
        denoised=hidden*cout[:,None,None]+z*cskip[:,None,None]
        return self.logits(denoised)
    def loss(self,batch,g):
        x,y=batch[:,:-1],batch[:,1:]
        if not self.diff:return F.cross_entropy(self(x).flatten(0,1),y.flatten()),{'active_blocks':len(self.blocks)}
        interval=int(torch.randint(len(self.blocks),(1,),generator=g));lo,hi=self.bands[interval:interval+2]
        delta=math.log(hi)-math.log(lo);lo=max(SMIN,math.exp(math.log(lo)-.1*delta));hi=min(SMAX,math.exp(math.log(hi)+.1*delta))
        u=torch.rand(batch.shape[0],generator=g).tolist();sig=torch.tensor([sigma(cdf(lo)+(cdf(hi)-cdf(lo))*v) for v in u])
        target=F.normalize(self.emb(y),dim=-1);z=target+sig[:,None,None]*torch.randn(target.shape,generator=g)
        block=len(self.blocks)-1-interval;logits=self.denoise(x,z,sig,block,True)
        ce=F.cross_entropy(logits.flatten(0,1),y.flatten(),reduction='none').reshape(y.shape)
        weight=(sig.square()+.25)/(sig*.5).square()
        return (ce*weight[:,None]).mean(),{'active_blocks':1,'block':block,'objective':'weighted denoising CE, not AR NLL'}
    @torch.inference_mode()
    def generate(self,prefix,new_tokens,seed=1,passes=4):
        self.eval();g=torch.Generator().manual_seed(seed);tokens=prefix.clone();calls=0;generated=0
        while generated<new_tokens:
            if not self.diff:
                logits=self(tokens)[:,-1];out=logits.argmax(-1)[:,None];calls+=1
            else:
                k=min(self.spec['generation_block_size'],new_tokens-generated);b=tokens.shape[0];d=self.spec['width']
                z=torch.randn((b,k,d),generator=g)*math.sqrt(1+SMAX**2)
                schedule=[sigma(cdf(SMIN)+(cdf(SMAX)-cdf(SMIN))*i/(passes-1)) for i in reversed(range(passes))]
                for i,s in enumerate(schedule):
                    interval=min(len(self.blocks)-1,max(0,sum(s>=v for v in self.bands)-1));block=len(self.blocks)-1-interval
                    logits=self.denoise(tokens,z,torch.full((b,),s),block,False);calls+=1
                    if i+1<len(schedule):
                        target=F.linear(logits.softmax(-1),F.normalize(self.emb.weight,dim=-1).t());z=z+(schedule[i+1]-s)*(z-target)/s
                out=logits.argmax(-1)
            tokens=torch.cat([tokens,out],1);generated+=out.shape[1]
        return {'tokens':tokens[:,prefix.shape[1]:],'model_calls':calls,'generation_block_size':self.spec['generation_block_size'],'passes_per_group':passes if self.diff else 1,'cache':'none; full-prefix execution charged'}

Fresh non-overlapping validation and test windows were hash-checked; exact 64-byte scored blocks found in training were excluded. Public source splits had been used in prior research, so this is not a claim of previously untouched or semantically uncontaminated data.

All 12 saved checkpoints reproduced their test-window losses. Cached-versus-full logits and future-token causality were checked. A separate fixed-update replay reproduced one job bitwise; CPU-time-budget reruns can execute different numbers of updates.

Binary projection weights were simulated using floating-point storage. No packed binary kernel or binary runtime speedup was demonstrated. The attention control was small, untuned and position-free. CPU microbenchmarks were not quality-matched, and no GPU, energy, production-cost, frontier-intelligence or 600B-scale conclusion follows.

This used adapted upstream ASI-Arch stages with a bounded coordinator, not a reproduction of the original large search.