Rectified Language Flow

A rectified-flow language model that writes 1,024 tokens at a time, with no masking and no left-to-right order. Below, ReLF-Large and an autoregressive model of the same size generate side by side on measured wall-clock time.

0 ms

ReLF steps

ReLF-Large

AR-Large

Blocks

Block 1

At every sampling step all 1,024 positions hold a predicted token, and the whole block moves together along a short trajectory from noise to text; the autoregressive model fills the same block one token at a time. Each box is one token position, the clock is measured wall-clock time, and every frame is stored model output.



: one block at a time, batch size 1

Results

On OpenWebText with 64 function evaluations, ReLF attains lower generative perplexity than autoregressive baselines at every model size tested under matched diversity, and reflow straightens the flow for few-step generation.

(a) Quality and diversity at 64 NFE

(b) Wall-clock speedup

Quality, diversity and speed. (a) Quality–diversity trade-off of ReLF and ELF at 64 NFE, with AR temperature sweeps as reference; colors denote methods, markers model sizes, and the star real text. ReLF-Large reaches Gen. PPL 17.3 at entropy 5.51, close to real text (17.3 at 5.49). (b) Wall-clock speedup over AR for 1,024-token generation on one GPU, with KV caching for AR and CUDA graphs for both; ReLF after two reflow rounds at 16 NFE.

Generative perplexity and entropy against the number of function evaluations for three model sizes after zero, one and two reflow rounds.

Few-step generation across reflow rounds. Gen. PPL (top) and entropy (bottom) versus NFE for ReLF-Small, ReLF-Medium and ReLF-Large after zero, one and two reflow rounds. At 8 NFE, two rounds reduce Gen. PPL by 40%, 41% and 35%, respectively, relative to the model before reflow. Dotted lines are same-size AR models; long-dashed lines are real OpenWebText.

Time per batch and throughput of ReLF and autoregressive models at batch sizes 1, 4, 16 and 64, with the speedup over the autoregressive model above each ReLF bar.

Latency and throughput. ReLF at 16 NFE after two reflow rounds and AR on one GPU, with KV caching for AR and CUDA graphs for both. Top: time per batch of 1,024-token sequences; bottom: throughput in tokens per second, at batch sizes 1, 4, 16 and 64. Labels above the ReLF bars give the speedup over AR.

Ablations

Singular logits, Hellinger–SVD embeddings and cross-entropy training each lower generative perplexity at matched entropy, and a 32-dimensional flow space gives the best quality–diversity trade-off.

Generative perplexity against entropy with and without singular logits, over momentum guidance strength.

(a) Singular logits

Generative perplexity against entropy for Hellinger–SVD, Gaussian and one-hot embeddings.

(b) Embedding matrix

Generative perplexity against entropy for cross-entropy and MSE training over the guidance scale.

Generative perplexity against entropy for flow embedding dimensions 16 to 256.

(d) Flow embedding dimension

Generative perplexity and entropy against the number of function evaluations for the SDE and Euler samplers.

(e) Sampling budget

Generative perplexity and entropy against the guidance scale for the Small, Medium and Large models.

(f) Guidance and model size

Legend: SDE sampler solid line, Euler sampler dashed line in panels b and e; real data as a dashed reference line.

Ablations. ReLF-Small at 64 NFE unless indicated. (a) Singular logits under momentum guidance α. (b) Embedding matrix. (c) Cross-entropy and MSE training objectives over the guidance scale ω. (d) Flow embedding dimension. (e) Number of function evaluations at ω = 1.7. (f) Guidance response of the Small, Medium and Large models. Panels (a)–(d) show Gen. PPL versus entropy; (e) and (f) show Gen. PPL above and entropy below. The diversity floor is entropy 5.40. At entropy 5.45, cross-entropy training reaches Gen. PPL 24.1 versus 25.6 for MSE.

How it works

ReLF is trained with cross-entropy on positionwise token posteriors, and its velocity is the posterior-weighted mean of the token embeddings. Singular logits capture the divergent part of the posterior analytically, leaving the network a bounded residual and guaranteeing that the flow converges to a valid token embedding.

Architecture diagram: the noisy state and the self-conditioning input enter a Transformer whose residual logits are added to singular logits; the probability-weighted mean of token embeddings is fed back as the next self-conditioning input.

Architecture. From the noisy state Z_t and the self-conditioning input, the Transformer outputs residual logits f_θ, which are added to the singular logits s. The prediction X̂₁, a probability-weighted mean of token embeddings, is fed back without gradient as the next self-conditioning input (dashed).

Three panels: token probabilities along the trajectory with bounded logits and with singular logits, and the resulting flow paths in embedding space.

Singular logits connect commitment to embedding convergence. A four-token example in which both models share the bounded residual f_θ and the initial state Z₀. (a) With bounded logits, the token distribution p_t keeps mass on competing tokens. (b) Adding s = log r_t drives p_t to a one-hot distribution. (c) Correspondingly, the flow ends at an embedding mixture (dashed) or at the token embedding E(w₁) (solid).

About the replay

Model	Sequential steps per block	Time per block	Speedup, batch size 1	Speedup, batch size 64	Gen. PPL	Entropy

Time per block is the median wall-clock time over 8 runs to generate one 1,024-token sequence on one GPU at batch size 1, after warm-up; runs differ by less than 0.4%. AR-Large uses KV caching and a compiled decoding step, and both models run as CUDA graphs. Speedup is AR time divided by ReLF time for a batch of 1,024-token sequences.

The advantage is in latency. At batch size 64 the autoregressive model uses the GPU more fully, and the ratio falls as the table shows.

Each box is one of the 1,024 token positions of a block, in reading order; ↵ is a newline token and <eos> the end-of-text token that separates documents. A token too long for its box is set smaller.

The ReLF frames are stored trajectories of the checkpoint and sampler that were timed. At every sampling step the network outputs a distribution over the vocabulary for each position; a box shows the most likely token of that distribution at that step, and a box lights up when its token changes. Before the first step the state is Gaussian noise and nothing is decoded. The 64-step setting is the stochastic sampler of the paper's main results; it injects fresh noise at every step, so its prediction keeps changing until the last step. The 16- and 32-step settings use the deterministic Euler sampler, whose prediction settles step by step.

Gen. PPL (under GPT-2 Large, lower is better) and unigram entropy are measured over 1,024 samples of the same checkpoint and sampler. Lowering the temperature lowers both for the autoregressive model, so compare rows at similar entropy. The 16- and 32-step rows are the model before reflow, which is what the replay shows; after two rounds of reflow the paper reports Gen. PPL 19.4 at entropy 5.41 for 16 steps and 16.2 at 5.43 for 32 steps, at the same time per step.

The AR-Large block is a stored evaluation sample at temperature 0.98; its first token is given and the other 1,023 are generated. Samples of both models were taken in stored order, skipping those with offensive or violent content.