In short
To understand what an AI system learns, researchers measure how much “information” its internal signals carry about the right answer. In large networks this can only be estimated, and current estimators depend on settings the researcher picks by hand, such as how finely to chop the signals into bins. Those settings can sway the result more than the network itself, so studies of the same question have reached opposite conclusions. We use a property of a widely used type of network (ReLU): it divides its inputs into many regions, and treats each region with its own transformation. Asking how much an input’s region reveals about its answer gives a measure of information read straight from the network. It is a measure of the information stored by the geometry of the network. Across 75 networks trained on three very different datasets, from handwritten digits to breast-cancer measurements, it agrees with established methods. When a network carves its inputs so finely that almost every example gets its own region, we merge regions that apply nearly the same transformation, which restores meaningful estimates. This gives researchers a more trustworthy way to study how AI systems learn, a step towards more transparent AI: instead of asking only how much information a network holds, they can ask where that information lives.
Abstract
Mutual information (MI) is fundamental to representation-learning analysis in deep networks, but in high dimensions MI estimators depend on hyperparameters whose bias often dominates the signal. We show that continuous piecewise-linear (CPWL) networks, including ReLU networks, admit an intrinsic information-theoretic analysis that avoids this pitfall: their inherent partition of the input space into convex linear regions assigns to every input a discrete region label Π alongside the continuous representation T and label Y. We introduce a decomposition of the mutual information I(Y; T) into interpretable geometric terms. We show that the dominant term, the routing information I(Y; Π), admits an estimable plug-in estimator, that in the well-sampled regime, approximates a population lower bound on the joint information I(Y; Π, T). When the partition saturates, a functional-equivalence quotient collapses regions implementing the same linear operator and restores informativeness; this extension has a single bounded, dimensionless parameter, which is reported through the quotient's compression ratio. The routing estimator agrees strongly with three established MI baselines applied to the same networks and data. The functional quotient recovers informative estimates in the regime bordering the well-sampled and under-sampled regime. We provide a framework for analysing the internal information geometry of CPWL networks, grounded in their inherent piecewise-linear structure rather than in external discretisation.
1. Linear regions and routing information
A trained network on the 7-class Composite dataset. Each colour is a linear region that contains held-out points; grey regions contain none. The routing information Î(Y; ΩD) is the plug-in mutual information between region and label over the N = 2,000 held-out points. Watch it rise with depth and during training, and how the number of regions grows.
2. The functional quotient
Deep networks create far more regions than the data can populate, and the plug-in estimate saturates. Regions whose active linear maps à differ by at most ε in relative Frobenius distance implement (nearly) the same function and are merged into one class, visiting regions in first-encounter order. Colours are quotient classes; thin lines are the original region boundaries. At the first layer nothing merges: distinct patterns are distinct maps.
3. Plug-in bias as ρ → 1
The plug-in estimate is only trustworthy when regions hold many points, i.e. when ρ = |ΩD| / N is small. Here routing is simulated with a known true MI: K equally likely regions, each with a home class kept with probability equal to the signal strength. As N shrinks relative to K, ρ → 1 and the plug-in estimate (solid) climbs to H(Y) whatever the truth, while Miller–Madow (dashed) overshoots below zero. Dots are the paper's 138 network cells (filled: plug-in, open: Miller–Madow), normalised by H(Y).
4. Routing information without learning
Left: a network trained on the true labels. Right: the same architecture trained on randomly permuted labels — it learns nothing about the true classes (test accuracy near chance), yet its partition still separates the held-out points, and the wide network keeps a routing information close to H(Y). High routing information alone does not show learned structure; read it together with ρ and accuracy.
BibTeX
@inproceedings{myliuskroken2026intrinsic,
title = {Intrinsic Information-Theoretic Analysis of {ReLU} Nets},
author = {Mylius-Kroken, Johan and Wetzer, Elisabeth and Ramezani-Kebrya, Ali
and Jenssen, Robert and Wickstr{\o}m, Kristoffer},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026},
eprint = {2610.36864},
archivePrefix = {arXiv},
primaryClass = {stat.ML}
}