ZERO Research Lab
Investigating how much of language-model inference is necessary computation.
- Evidence stage
- Evidence stage: Research hypothesis
- Status
- Early research · Planning
- Research domain
- Intelligence Systems
01 · Problem statement
Running a language model costs roughly the same amount of computation for every token, regardless of how difficult that token is to predict. As models are deployed more widely, inference, rather than training, becomes the dominant and recurring cost. Many existing efficiency techniques require retraining, architecture changes, or model-specific kernels, which limits who can use them and on which models.
02 · Research motivation
There is a substantial research literature showing that parts of transformer computation can be skipped, approximated, or reused for some inputs: early exits, layer skipping, speculative decoding, and cache management among them. Most of these results are tied to a specific model family or training procedure. ZERO asks whether the underlying redundancy can be detected and exploited in a way that is independent of the model, so that existing models benefit without retraining.
For a meaningful share of inputs, a measurable fraction of inference computation in existing language models is redundant and can be identified at run time cheaply enough that skipping it reduces total cost, without retraining and without an unacceptable loss in output quality. This is an untested hypothesis; it may be false, or true only under narrow conditions.
03 · Technical direction
- A
Map the redundancy
Survey and classify where redundant computation has been reported (layers, attention heads, tokens, repeated prefixes) and under which conditions.
- B
Define the measurement before the method
Fix quality metrics, baselines, hardware, and the accounting of detection overhead before any optimization is attempted, so results cannot be selected after the fact.
- C
Reproduce, then extend
Reproduce established techniques on small open models to calibrate the harness, then test model-agnostic variants against them.
- D
Report negative results
Publish what does not work with the same care as what does. A clear negative result is a valid outcome for this programme.
04 · Current verified status
- 01 Hypothesis (current stage)
- 02 Experiment
- 03 Prototype
- 04 Validated
- 05 Released
Research hypothesis: A stated, falsifiable idea. No experimental evidence yet.
Verified
- The research direction and scope are defined.
- The initiative is in planning. Literature review is the first work item.
Not yet done or demonstrated
- No experiments have been run.
- No computational savings have been demonstrated.
- No code, datasets, or models have been released.
Status source: Project brief; no code, experiments, or data exist yet. Checked .
05 · Known challenges
-
Detection overhead
Deciding what to skip costs computation; the decision must be cheaper than the work it avoids.
-
Quality measurement
Small degradations in output quality are hard to measure and easy to overlook with weak evaluations.
-
Model-agnosticism
Techniques that generalise across architectures may capture less of the available savings than model-specific ones.
-
Hardware dependence
Skipped operations do not always translate into wall-clock savings on real accelerators, because of batching and memory-bandwidth effects.
06 · Next milestones
- M1 Literature review and a written taxonomy of reported redundancy in transformer inference.
- M2 A documented measurement protocol covering quality metrics, baselines, hardware, and overhead accounting.
- M3 Reproduction of at least one established technique on a small open model to validate the harness.
- M4 A first falsifiable experiment, published as a research note regardless of outcome.
Milestones are planned work, listed in intended order. They are not dated commitments.
07 · Public resources
No public resources yet. This page will link to them when they exist.
References
External work this project builds on or compares against. Listing does not imply endorsement or affiliation.
- Leviathan, Kalman, Matias — Fast Inference from Transformers via Speculative Decoding (2022)
- Schuster et al. — Confident Adaptive Language Modeling (2022)
- Elhoushi et al. — LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (2024)
- Raposo et al. — Mixture-of-Depths: Dynamically allocating compute in transformer-based language models (2024)
- Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)