• $\phi_{q,p}: [0,1] \to \mathbb{R}$ — inner univariate functions
• $\Phi_q: \mathbb{R} \to \mathbb{R}$ — outer univariate functions
• Only 1D operations; no multivariate terms needed
• Existence guaranteed; functions may be non-smooth

Why does this matter for AI?

The universal approximation theorem tells us what MLPs can approximate. The KA theorem tells us something deeper: any continuous multivariate function fundamentally decomposes into sums of 1D functions.

The KAN insight (Liu et al., ICLR 2024)

If every multivariate function reduces to 1D functions, why not build networks where every learnable parameter is a 1D function? Each "weight" becomes a visualizable curve — interpretable, and in principle, symbolically regressionable.

MLP — fixed activations, learned scalars

• $\sigma$ is fixed (ReLU, GELU, …)
• $W_{ji}$ is a scalar — a single number per edge
• Nonlinearity sits at the node

KAN — fixed structure, learned functions

• Each $f_{i,j}$ is a learnable 1D function (a curve)
• No scalar weights — every edge carries a function
• Nonlinearity sits at the edge

B-spline basis $B(x)$

• Evaluated via Cox-de Boor recursion on a fixed grid
• $n_{\text{basis}}$ = grid_size + spline_order per input channel
• Default: grid=5, order=3 → $n_{\text{basis}}=8$
• Grid spans $[-1,1]$, extended with $k$ extra knots on each side

Grid update

After warm-up (step 500), knot positions adapt to the quantile distribution of actual activations — a once-per-training, non-differentiable operation.

Parameter count

# Cox-de Boor recursion (simplified)
def b_splines(x, grid, order):
    # x: (B, d_in) → bases: (B, d_in, n_basis)
    bases = (x >= grid[:,:-1]) & (x < grid[:,1:])
    for k in range(1, order+1):
        left  = (x-t_i)/(t_ik-t_i).clamp(1e-8) * bases[...,:-1]
        right = (t_ik1-x)/(t_ik1-t_i1).clamp(1e-8) * bases[...,1:]
        bases = left + right
    return bases

Standard transformer block

KAT Attention (Yang & Wang, ICLR 2025)

Replace linear Q, K projections with KANLinear feature maps $\varphi$, turning the dot-product similarity into a learned kernel:

Applied per-head on head_dim=64, not d_model=384. V and output projection remain standard linear layers.

Why replace only the FFN?

• FFN ≈ 2/3 of parameters — the largest compute block
• FFN is applied independently per token — ideal for KAN's 1D decomposition along feature axes
• Attention does relational computation; the benefit of splines on score functions is less clear

Shared architecture

Training

Dataset: GuppyLM-60k

arman-bd/guppylm-60k-generic — 60K instruction-response pairs in ChatML format. First-person fish-personality chatbot. BPE tokenizer trained on same corpus.

Why this scale?

At $d_\text{model}=384$, each KAN FFN layer has 147,456 edge functions — small enough to reconstruct and characterize every single one exhaustively. At GPT-2 scale ($d=768$) that's 4× more, making a full edge-level study infeasible.

The single-domain corpus provides coherent semantic categories for token-conditioned interpretability analysis.

Without masking

• Loss computed over all tokens including the user prompt
• Model learns to predict the prompt itself
• Outputs leak role markers — generates "assistant" as part of responses
• 4,000 training steps wasted on diluted signal

With masking  $y[\text{:prompt\_len}-1] = -100$

• Loss computed only on assistant tokens
• Role-marker leakage eliminated
• Clean, in-character fish-personality responses

The einsum structure is identical to KANLinear's spline path. Each $f_{i,j}(x)$ remains a visualizable 1D scalar function. No grid update needed.

# B-spline path          vs   MLP-edge path
splines = b_splines(x)          H = σ(einsum("bi,ih→bih", x, W1) + b1)
out = einsum("bik,oik→bo",      out = einsum("bih,oih→bo", H, W2)
             splines, W_spline)

Hidden size $h$ ablation

Bars show quality (inverted: shorter = worse). Lower val loss is better.

Safe Padé rational activation

• Numerator $\{a_k\}$ — shared across all groups
• Denominator $\{b_k\}$ — per group (g=8 groups)
• Denominator $\geq 1$ everywhere — numerically safe
• $m=5$, $n=4$: only $(m{+}1)+n{\cdot}g=38$ extra params over an equivalent MLP

GR-KAN FFN structure

Why rational functions?

• GPU-friendly: pure tensor ops — no Cox-de Boor recursion
• Smooth everywhere: no knot discontinuities
• Expressive: Padé approximants converge faster than polynomials of the same degree
• Fused CUDA kernel available via rational_kat_cu — fixes backward-pass memory bottleneck identified in FlashKAT (arXiv 2505.13813)

Initialization

rat₁ → identity; rat₂ → Swish (least-squares fit on $[-4,4]$). This enables weight transfer from pre-trained MLP transformers.

Reconstruction

Each edge $f_{i,j}(x)$ is reconstructed from model weights:

Evaluated on 200 points from each channel's grid range.

Three per-function metrics

MLP effective edge functions

Scalar contribution of input channel $i$ to output $j$, all other channels at zero. Sampled on $[-2,2]$.

NLS consistency across layers

Functional PCA

Top-4 components explain 99.9% of variance per layer (PC1: 60–62%, PC2: 30–32%, PC3: 7%, PC4: <0.3%). Note: $n_\text{basis}=8$ means the function space is at most 8-dimensional — this is partly parametric.

Experiment A — MLPEdge · WikiText-103

124M params · RTX A6000 · 20K steps · metric: perplexity

−26% — per-edge factored topology fails at scale

• einsum overhead: 11% throughput penalty
• Same $h=8$ per-edge budget is underpowered for WikiText-103 complexity at $d=768$

Experiment B — GR-KAN · ClimbMix

286M params · H100 NVL · 2520 steps · metrics: bpb + DCLM CORE

bpb gap ~4.2% · CORE gap −0.025 · canonical formula

★ Zhang et al. 2022, lm-harness zero-shot, different training data — indicative only.
Eval time: 287.2 s on MPS device.

Muon optimizer

Newton-Schulz-5 orthogonalization of the momentum buffer for 2D weight matrices. Active on CUDA only; falls back to AdamW on MPS/CPU. Converges ~2× faster per token vs AdamW, halving cloud compute cost.

Architecture

Each transformer block replaces the single GR-KAN FFN with 8 independent GR-KAN expert FFNs. A linear router selects the top-2 experts per token:

Active FLOPs ≈ top_k / n_experts = 25% of dense GR-KAN.

Load balance loss

$f_i$ = fraction of tokens routed to expert $i$; $P_i$ = avg router probability. Encourages uniform expert utilization; prevents collapse.

Configuration

Expected benefits

• Parameter efficiency: 8 specialized experts, only 2 active per token
• Each expert can specialize in different token types or syntactic contexts
• Combines GR-KAN's rational expressiveness with conditional computation

Ouro architecture (arXiv 2510.25741)

The same GR-KAN body is applied $T_\text{max}$ times — weight-tied, like an RNN at the transformer-block level:

• $\lambda_t$ = per-step exit probability
• Inference exits when $\text{CDF} > 0.8$ or after $T_\text{max}=4$ steps

Training objective

The entropy term $H$ prevents the exit distribution from always using $T_\text{max}$ steps — encouraging adaptive compute allocation.

EqR enhancements (arXiv 2605.21488)

• RI (Random Init): $z_0\sim\mathcal{N}(0,\sigma^2 I)$ added at embedding — breaks symmetry, enables breadth scaling
• NI (Noisy Iteration): $x_{t+1} = x_t + (1{-}\lambda)(\text{body}(x_t){-}x_t) + \beta\varepsilon$ — shapes attractor landscape

Breadth scaling

Run $B=4$ independent RI restarts at inference; select by lowest mean residual $\|x_t - x_{t-1}\|$. Trades latency for quality at test time — no retraining needed.

Why loop a GR-KAN?

• Rational functions define a smooth dynamical system — convergence is more stable than with splines
• Weight tying reduces parameters while increasing effective depth
• Exit gate learns adaptive compute: simple tokens exit early

• Masking is critical. Computing loss on all tokens dilutes signal by 31%. Architectural comparisons without it are unreliable.
• KAN topology matters more than the basis at small scale. MLPEdge matches B-spline KAN quality at 4× faster training, no grid update.
• KAT attention helps quality but costs speed. 14–15/16 accuracy vs 8/16 for dot-product, but 1.4× more inference latency.
• GR-KAN is the best FFN replacement. 30M params beats OPT-125M on ARC-Easy at 4× fewer params. At 286M params (g=8), only ~1.2% behind MLP on ClimbMix bpb, and 6.6% behind on DCLM CORE — with a striking +0.12 advantage on COPA.
• MLPEdge does not scale; GR-KAN does. MLPEdge is 26% worse than MLP at GPT-2 scale. GR-KAN is only ~1.2% worse on bpb — same scale, same protocol. Rational activations are what scales, not the per-edge factored topology.
• GR-KAN trails MLP at GPT-2 scale. Canonical Safe Padé formula: val_bpb 0.8834 vs 0.8474 (+4.2%), CORE 0.1286 vs 0.1540 (−0.025). MLP leads 20 of 22 tasks. The ~1.2% gap previously reported was inflated by a per-coefficient-abs denominator bug — now corrected.
• MLPEdge does not scale; GR-KAN partially does. MLPEdge is 26% worse than MLP at GPT-2 scale. GR-KAN is ~4.2% worse — same scale, same protocol. Rational activations are more favorable than per-edge MLPs, but the gap is larger than initially believed.

Still open

• GR-KAN at 286M is ~1.2% behind MLP on bpb; g=8 narrows the gap vs g=4. Does the trend continue at g=16? Can longer training close the 6.6% CORE gap? Ablations queued.
• COPA anomaly (+0.12 for GR-KAN): only 100 examples — needs replication on larger causal reasoning benchmarks.
• Can MLPEdge close the GPT-2 gap with higher $h$? Untested.
• MoE GR-KAN full benchmarks — in progress.
• Symbolic regression on rational edge functions — planned.
• ModuleChain staging recipe at 286M scale — next run.

Honest caveats

• GR-KAN GuppyLM result: single seed, 2K extra steps — not controlled
• GR-KAN 286M result: 2520 steps, ~1.32B tokens only; longer training may change the bpb and CORE gaps
• GR-KAN 286M result: 2520 steps, ~1.32B tokens only, single seed; longer training may change the bpb and CORE gaps
• Mann-Whitney $p$-value: edges share grids — not inferential

Near term

• Complete MoE GR-KAN benchmarks (WT103, ARC, HellaSwag)
• Fully benchmark LoopGRKAN adaptive compute and reasoning capabilities
• Multi-seed controlled comparison for GR-KAN vs MLP at GuppyLM scale
• Update paper with canonical GR-KAN GPT-2 scale results and formula correction

Research questions

• Do rational activations close the GPT-2 gap for KAN-topology models?
• Symbolic regression on GR-KAN rational edge functions
• Pruning circuits: which edges determine fish-domain vs generic behavior?
• Token-conditioned activation atlases for KAN edges

Scale

• Train GR-KAN at 125M+ params on a richer corpus (Pile subset)
• Test weight transfer: init GR-KAN from pre-trained MLP weights (KAT paper recipe)
• LoopGRKAN breadth scaling evaluation at inference time
• MoE routing patterns as an interpretability signal