FastConformer, from the waveform up
An independent explainer of NVIDIA's FastConformer encoder, its cache-aware streaming variant, and the CTC, RNN-T and TDT decoders behind Parakeet.
What this page covers
FastConformer is a Conformer encoder redesigned to be cheaper: it reduces the audio to one frame every 80 ms before the first attention layer, where Conformer stops at 40 ms. Most of what follows comes from that single decision.
It was introduced by Dima Rekesh, Nithin Rao Koluguri, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, He Huang, Oleksii Hrinchuk, Krishna Puvvada, Ankur Kumar, Jagadeesh Balam and Boris Ginsburg (arXiv 2305.05084, NVIDIA and UCLA). The cache-aware streaming version is by Vahid Noroozi, Somshubra Majumdar, Ankur Kumar, Jagadeesh Balam and Boris Ginsburg (arXiv 2312.17279). The Token-and-Duration Transducer is by Hainan Xu, Fei Jia, Somshubra Majumdar, He Huang, Shinji Watanabe and Boris Ginsburg (arXiv 2304.06795, ICML 2023). The reference implementation is NVIDIA NeMo.
This page is not affiliated with NVIDIA. It was written by Quran Lab, which trains FastConformer models for Quranic recitation. Numbers are quoted from the papers, the NeMo source and NVIDIA's model cards, with the table they came from. Anything this page computes itself, from layer shapes or formulas, is labelled as computed.
What changed from Conformer
A Conformer block is feed-forward, self-attention, a depthwise convolution and a second feed-forward, each with a residual connection. FastConformer keeps the block and changes what feeds it.
The original Conformer starts with a subsampling module that takes 10 ms log-mel frames to 40 ms. The paper measures that module at over 20% of the forward pass for Conformer Large, and every attention layer after it costs time and memory quadratic in the number of frames. The authors made four changes, all at the input or in the convolution module (Section 2.1):
- 8x downsampling at the start of the encoder, so every attention layer sees half the frames and computes a quarter of the scores.
- Depthwise-separable convolutions replace the regular convolutions in the subsampling block.
- 256 filters instead of 512 in the subsampling block.
- Kernel 9 instead of 31 in the depthwise convolution of every Conformer block.
Step through them in the order the paper applied them to Conformer-RNNT Large. The bars move to the paper's measured values. The tick on each bar marks the baseline.
Rekesh et al., Table 2. Encoder throughput with batch 128 on 20 s clips, A100 80 GB. Greedy WER. Parameters and GMACs for the encoder only.
Why 8x needed a tokenizer change
CTC needs at least as many encoder frames as output labels. At 80 ms a frame, a second of speech has 12.5 frames. The authors found that most LibriSpeech training utterances break that condition with character targets after 8x subsampling, so FastConformer-CTC uses SentencePiece BPE vocabularies of 128 to 1024 tokens. RNN-T has no such constraint, and the paper notes that the shorter encoder output also saves compute and memory in the RNN-T decoder (Section 2.1, footnote 3, and the Fig. 1 caption).
Downsampling schemas. K is the conv kernel in the Conformer blocks. Rekesh et al., Table 1.
| Model | Schema | Type | K |
|---|---|---|---|
| Conformer | 2/4 | 2D Conv | 31 |
| Squeezeformer | progressive 2/4/8/4 | Depth-wise sep | 31 |
| Eff. Conformer | progressive 2/4/8 | Depth-wise sep | 15 |
| Fast Conformer | 2/4/8 | Depth-wise sep | 9 |
The 8x front-end, layer by layer
In NeMo this is subsampling: dw_striding with subsampling_factor: 8 and
subsampling_conv_channels: 256. It treats the spectrogram as a one-channel image of time by
frequency. The first layer is an ordinary 3x3 convolution with stride 2. The next two are depthwise 3x3
convolutions with stride 2, each followed by a 1x1 pointwise convolution and ReLU. Time and frequency both
halve at every stride, and a linear layer projects the flattened channels and frequency bins to
d_model.
Each stride follows NeMo's calc_length: with kernel 3 and padding 1, a length L becomes
floor((L - 1) / 2) + 1. Drag the duration to see the shapes and the cost of both front-ends.
FastConformer, dw_striding, factor 8, 256 channels
| Layer | Output (C x T x F) | MACs |
|---|
Conformer, striding, factor 4, 512 channels
| Layer | Output (C x T x F) | MACs |
|---|
Computed on this page from the layer definitions in subsampling.py and the configs fast-conformer_transducer_bpe.yaml and conformer_transducer_bpe.yaml, with d_model 512. MACs count weight multiplications only. The paper's own figure for the whole encoder is in Table 2.
The second Conformer convolution is where the cost goes: 512 input channels times 512 output channels times nine taps, at every position of a 20-bin frequency axis. Splitting it into a depthwise and a pointwise layer, and halving the channels, removes most of it. Inside the blocks the same frame rate argument applies to the kernel: 9 taps at 80 ms cover 720 ms of audio, while 31 taps at 40 ms covered 1.24 s.
Sequence length and attention cost
Self-attention scores every frame against every other frame, so one head in one layer holds a T x T matrix. Halving T divides it by four. With full attention, the paper reports that FastConformer has 2.9x fewer multiply-adds than Conformer and fits 25 minutes of audio on one A100 against Conformer's 15. With limited context attention the cost grows linearly, and the limit moves to 675 minutes.
Curves and readouts computed on this page: frame counts from calc_length, 8 heads and 17 layers
as in the Large models, fp16 scores for one layer only (activations and other buffers are not counted, so
this does not predict Table 3 on its own). Triangles: Rekesh et al., Table 3.
Limited context and one global token
For long recordings the paper swaps full attention for the Longformer pattern after training: each frame attends to a fixed window around itself, and one global token attends to every frame while every frame attends to it (Section 2.2). The window was 128 frames on each side, about 10 seconds at 80 ms. The model was initialized from the full-context checkpoint and fine-tuned for 10k steps with 1k warmup steps, a maximum learning rate of 1e-6 and cosine annealing (Section 3.4).
In NeMo this is self_attention_model: rel_pos_local_attn with
att_context_size: [128, 128] and global_tokens: 1. The global token is not an
extra embedding: the attention module marks the first frames of the sequence as global
(spaced by global_tokens_spacing). Whether global attention gets its own query, key and value
projections is the global_attn_separate flag, off by default. The paper describes separate
projections in Section 2.2 and shared ones for the trained long-form model in Section 3.4.
Pick a mode and touch a row to see which keys that query frame can reach. The chunked mode is the streaming mask from the next section.
Long audio, normalized greedy WER (%). Full-context models used 20 s buffers; the limited-context models took each recording in one forward pass. Rekesh et al., Table 8.
| Model | TED-LIUM v3 | Earnings21 |
|---|---|---|
| Conformer | 9.18 | 18.26 |
| Fast Conformer (buffered) | 9.15 | 17.65 |
| + Limited Context | 8.25 | 16.08 |
| + Global Token | 7.5 | 11.85 |
Cache-aware streaming
The streaming variant is trained like an offline model, in parallel over whole utterances, but with the context it will have at inference. At inference it runs chunk by chunk and keeps the activations it will need again in caches, so nothing is computed twice.
To make training and inference see the same thing, the paper (Section 3.1) drops normalization of the log-mel features, makes every convolution causal by padding k - 1 frames on the left, including the subsampling layers, replaces batch normalization with layer normalization, and limits how far each attention layer can look left and right.
Regular look-ahead multiplies with depth
If every attention layer may look M frames ahead, the output depends on M x N frames of future input after N layers. For FastConformer with 17 layers, one frame of look-ahead per layer is already 10 ms x 8 x 17 = 1360 ms (Section 4.2). Look-ahead tokens computed early are also wrong, because they could not see their own future, and have to be recomputed.
Chunk-aware look-ahead does not
Split the frames into chunks of C. A frame sees its whole chunk and a fixed number of previous chunks, in
every layer, so the look-ahead stays between 0 and C - 1 frames however deep the model is, and every
output of a chunk is final. NeMo calls this att_context_style: chunked_limited.
Chunk, cache and look-ahead arithmetic from setup_streaming_params in
conformer_encoder.py and the comment in
fastconformer_ctc_bpe_streaming.yaml (look-ahead = 13 x 8 x 0.01 = 1.04 s).
Cache sizes computed for 17 layers, d_model 512, batch 1.
What is cached
Feed-forward, normalization and pointwise layers need no past context. Two kinds of layer do
(Section 3.3). Each causal depthwise convolution with kernel K keeps the last K - 1 inputs, a
L x B x D x (K - 1) tensor across L layers. Each attention layer keeps up to its left
context of earlier inputs, growing from empty to full and then dropping the oldest, a
L x B x Cmha x D tensor. In NeMo these are cache_last_time and
cache_last_channel. The subsampling convolutions are not cached; a few trailing log-mel
frames are prepended to the next chunk instead (NeMo's dw_striding asks for 9). An RNN-T
decoder carries its recurrent state across chunks.
With all caches in place, the paper states that the predictions are exactly the same as when the whole recording is processed in one step, and offline and streaming evaluation give the same accuracy.
One model can serve several latencies: training picks a random chunk size per batch. NVIDIA's
multi-look-ahead model
is trained with [[70,13],[70,6],[70,1],[70,0]] and the card lists 1040, 480, 80 and 0 ms;
the card notes those are worst-case latencies, and the average is half.
LibriSpeech test-other, models trained on LibriSpeech. Noroozi et al., Table 1.
| Decoder, mode | Look-ahead | WER % | Avg latency, ms |
|---|---|---|---|
| CTC | |||
| Offline | Full context | 5.7 | - |
| Buffered | Buffering | 8.0 | 1500 |
| Cache-aware | Zero | 10.6 | 0 |
| Cache-aware | Regular | 7.9 | 1360 |
| Cache-aware | Chunk-aware | 7.1 | 1360 |
| RNNT | |||
| Offline | Full context | 5.0 | - |
| Buffered | Buffering | 11.3 | 2000 |
| Cache-aware | Zero | 9.5 | 0 |
| Cache-aware | Regular | 7.1 | 1360 |
| Cache-aware | Chunk-aware | 6.3 | 1360 |
Single vs multi-look-ahead training, WER %. NeMo ASRSET 3.0, reported on LibriSpeech test-other. Noroozi et al., Table 4.
| Model | Decoder | 40 ms | 240 ms | 520 ms |
|---|---|---|---|---|
| Single look-ahead | CTC | 7.9 | 7.3 | 6.2 |
| Multi look-ahead | CTC | 7.6 | 6.5 | 6.0 |
| Single look-ahead | RNNT | 6.4 | 5.9 | 5.4 |
| Multi look-ahead | RNNT | 6.2 | 5.5 | 5.2 |
Decoders: CTC, RNN-T and TDT
The encoder produces one vector per 80 ms frame. Three heads turn those vectors into tokens, and NeMo can train two of them on the same encoder. The lattice below decodes one made-up utterance of 20 frames and four tokens with each of them. The alignment is invented for the illustration; the decoding rules are the real ones.
CTC
A linear layer and softmax over the vocabulary plus a blank, applied to every frame independently. Decoding takes the most likely symbol per frame, merges repeats and drops blanks. All frames are scored in one parallel pass, so the encoder is almost the whole cost. Because each frame emits at most one label, the frame count bounds the output length, which is why 8x subsampling goes with subword tokens.
RNN-T
A prediction network reads the tokens emitted so far, and a joint network combines its state with one encoder frame. Greedy decoding asks the joint at (t, u): a token keeps t and moves up, a blank moves to the next frame. A greedy pass makes one call per blank and one per token, here T + U = 24. The decoder is autoregressive, so these calls run one after another, and halving T with 8x subsampling halves the blank steps.
Graves 2012; Xu et al., Algorithm 1.
TDT: token and duration
The joint network has two outputs, normalized separately: a distribution over tokens (with blank) and a distribution over durations, the number of frames the emission covers. The two are treated as independent, P(v, d) = PT(v) PD(d). At inference t advances by the predicted duration, for blanks and tokens alike, so silence and long sounds are skipped in a few calls. A blank of duration 0 is not allowed.
Training extends the RNN-T forward-backward recursion to sum over durations, and uses logit under-normalization (a constant sigma subtracted from the token log-probabilities) to favor longer durations. The paper used durations up to 8 and sigma 0.05 for ASR. NeMo's hybrid TDT-CTC config uses durations [0, 1, 2, 3, 4], sigma 0.02, and adds a standard RNN-T loss with weight omega 0.1.
Xu et al., Sections 3 and 4, Algorithm 2; fastconformer_hybrid_tdt_ctc_bpe.yaml.
TDT against RNN-T, LibriSpeech test-other. "0-4" means durations 0 to 4. Greedy, non-batched decoding. Xu et al., Table 2. These models use a Conformer-Large encoder with 4x subsampling, not FastConformer.
| Config | WER % | Time, s | Speed-up |
|---|---|---|---|
| RNNT | 5.11 | 244 | - |
| TDT 0-2 | 5.50 | 171 | 1.43X |
| TDT 0-4 | 5.06 | 128 | 1.91X |
| TDT 0-6 | 5.05 | 118 | 2.07X |
| TDT 0-8 | 5.16 | 115 | 2.12X |
Hybrid training: one encoder, two decoders
The streaming paper trains a CTC decoder and an RNN-T decoder on one shared encoder and mixes the losses as ltotal = alpha x lctc + lrnnt, with alpha = 0.3 (Sections 3.2 and 4). Either decoder can be used at inference. The authors list three gains: one training run instead of two, much faster convergence of the CTC decoder, and better accuracy for both; Table 2 of that paper shows the hybrid chunk-aware models ahead of non-hybrid ones at most latencies.
NeMo writes the mix as (1 - w) x lrnnt + w x lctc in
hybrid_rnnt_ctc_models.py, with ctc_loss_weight w = 0.3 in its hybrid
TDT-CTC config. The hybrid TDT-CTC models swap the RNN-T head for
TDT. For streaming RNN-T the paper also uses FastEmit with lambda 0.005, and reports that it helped the
CTC decoder too.
Headline results, as published
Copied from the papers without rounding. Greedy decoding throughout, no external language model. The highlighted row is the one the table is about.
Large models trained on LibriSpeech. Greedy WER on test-other; params and compute for the encoder only, on one 30 s clip. Rekesh et al., Table 4.
| Encoder | WER % | Params, M | GMACs |
|---|---|---|---|
| RNNT decoder | |||
| Conformer | 5.19 | 115 | 143.2 |
| Fast Conformer | 4.99 | 109 | 48.7 |
| CTC decoder | |||
| Conformer | 5.74 | 121 | 149.2 |
| Eff. Conformer | 5.79 | 125 | 101.3 |
| SqueezeFormer | 6.05 | 125 | 91.0 |
| Fast Conformer | 5.64 | 115 | 51.5 |
Large models trained on 25k hours of public English speech. Greedy WER %. Rekesh et al., Table 5.
| Encoder | LS other | MCV 8 | MLS En | WSJ-92 |
|---|---|---|---|---|
| RNNT decoder | ||||
| Conformer | 3.74 | 7.87 | 5.77 | 1.47 |
| Fast Conformer | 3.79 | 8.18 | 5.76 | 1.42 |
| CTC decoder | ||||
| Conformer | 4.50 | 9.40 | 6.60 | 1.70 |
| Fast Conformer | 4.19 | 9.00 | 6.42 | 1.59 |
Scaling up, RNNT, 25k hours. Greedy WER %. Rekesh et al., Table 10.
| Model | LS clean | LS other | MLS | GMACs |
|---|---|---|---|---|
| Conformer-XL | 1.49 | 2.80 | 5.32 | 686 |
| FC-XL | 1.50 | 2.88 | 4.90 | 253 |
| FC-XXL | 1.38 | 2.52 | 4.58 | 441 |
Model sizes. The core block is unchanged from L to XXL. Rekesh et al., Table 9.
| Model | d_model | Enc. layers | RNNT layers | Params |
|---|---|---|---|---|
| L | 512 | 17 | 1 | 120 M |
| XL | 1024 | 24 | 2 | 600 M |
| XXL | 1024 | 42 | 2 | 1.1 B |
Cache-aware streaming on NeMo ASRSET 3.0, LibriSpeech test-other and the average over nine test sets, WER %. Noroozi et al., Table 3.
| Model | Avg latency, ms | LS other | Averaged |
|---|---|---|---|
| CTC | |||
| Cache-aware | 40 | 7.9 | 13.3 |
| Cache-aware | 240 | 7.3 | 12.6 |
| Cache-aware | 520 | 6.2 | 11.2 |
| Buffered | 2000 | 7.7 | 12.0 |
| RNNT | |||
| Cache-aware | 40 | 6.4 | 12.1 |
| Cache-aware | 240 | 5.9 | 11.3 |
| Cache-aware | 520 | 5.4 | 10.6 |
| Buffered | 2000 | 9.4 | 15.0 |
Speech translation, MUST-C V2 tst-COMMON, En to De. Batch 32. Rekesh et al., Table 6.
| Encoder | BLEU | Time, s | Speed-up |
|---|---|---|---|
| Transformer decoder | |||
| Conformer | 31.0 | 267 | 1X |
| Fast Conformer | 31.4 | 161 | 1.66X |
| RNNT decoder | |||
| Conformer | 23.2 | 83 | 1X |
| Fast Conformer | 27.9 | 45 | 1.84X |
Where FastConformer ships: Parakeet
NVIDIA's Parakeet models on Hugging Face are FastConformer encoders with the decoders above. The rows below say only what each official model card says; several cards describe the models as developed jointly by NVIDIA NeMo and Suno.ai. The size names L, XL and XXL match Table 9 of the FastConformer paper.
Parakeet and related FastConformer models, from NVIDIA's model cards on Hugging Face (read 3 November 2026).
| Model card | Decoder | Size | What the card states |
|---|---|---|---|
| parakeet-tdt-0.6b-v3 | TDT | 600M | 25 European languages; up to 24 min per pass with full attention, up to 3 h with local attention |
| parakeet-tdt-0.6b-v2 | TDT | 600M (XL) | English with punctuation and capitalization; Granary, about 120,000 h; up to 24 min per pass |
| parakeet-tdt_ctc-1.1b | Hybrid TDT + CTC | 1.1B (XXL) | local attention and a global token, 11 h of audio in one pass; 36K h English |
| parakeet-tdt_ctc-110m | Hybrid TDT + CTC | 114M (Large) | full attention, up to 20 min per pass; 36K h English |
| parakeet-tdt-1.1b | TDT | 1.1B (XXL) | 64K h English |
| parakeet-rnnt-1.1b | RNN-T | 1.1B (XXL) | 64K h English |
| parakeet-rnnt-0.6b | RNN-T | 600M (XL) | 64K h English |
| parakeet-ctc-1.1b | CTC | 1.1B (XXL) | 64K h English |
| parakeet-ctc-0.6b | CTC | 600M (XL) | 64K h English |
| stt_en_fastconformer_hybrid_large_streaming_multi | Hybrid RNN-T + CTC, cache-aware | 114M (Large) | one model, four look-aheads: 0, 80, 480 and 1040 ms |
Case study: a FastConformer for Quranic recitation
Quranic recitation is fully diacritized Classical Arabic with long, rule-governed elongations; the leaderboard's own description notes that general Arabic ASR models degrade sharply on it. Quran Lab's fastconformer-quran starts from NVIDIA's Arabic stt_ar_fastconformer_hybrid_large_pcd_v1.0 and is fine-tuned for recitation in the Hafs riwayah.
From the public model card: a FastConformer Large encoder with a CTC head, 114.6M parameters; a SentencePiece BPE vocabulary of 1,024 tokens plus blank; 16 kHz audio as 80-bin log-mel features; trained on the EveryAyah studio recitations, tlog phone recordings and the Muaalem annotated set. It is published as a NeMo checkpoint and as ONNX in fp32 and fp16, with CoreML builds and a separately trained cache-aware streaming variant in companion repositories. The card also describes a mispronunciation scorer built on the encoder's features. It is a recitation model: the card says plainly that it is not for dialectal, news or conversational Arabic.
Sources
Papers
- D. Rekesh, N. R. Koluguri, S. Kriman, S. Majumdar, V. Noroozi, H. Huang, O. Hrinchuk, K. Puvvada, A. Kumar, J. Balam, B. Ginsburg. Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition. arXiv 2305.05084, ASRU 2023.
- V. Noroozi, S. Majumdar, A. Kumar, J. Balam, B. Ginsburg. Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition. arXiv 2312.17279.
- H. Xu, F. Jia, S. Majumdar, H. Huang, S. Watanabe, B. Ginsburg. Efficient Sequence Transduction by Jointly Predicting Tokens and Durations. arXiv 2304.06795, ICML 2023.
- A. Gulati et al. Conformer: Convolution-augmented Transformer for Speech Recognition. arXiv 2005.08100, Interspeech 2020.
- I. Beltagy, M. E. Peters, A. Cohan. Longformer: The Long-Document Transformer. arXiv 2004.05150.
- A. Graves, S. Fernandez, F. Gomez, J. Schmidhuber. Connectionist Temporal Classification. ICML 2006.
- A. Graves. Sequence Transduction with Recurrent Neural Networks. arXiv 1211.3711.
- J. Yu et al. FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization. arXiv 2010.11148.
NeMo source, as read for this page
- modules/conformer_encoder.py: masks, streaming parameters, caches
- submodules/subsampling.py: dw_striding, calc_length
- submodules/multi_head_attention.py: local attention, global tokens
- models/hybrid_rnnt_ctc_models.py: hybrid loss
- fast-conformer_transducer_bpe.yaml
- fastconformer_ctc_bpe_streaming.yaml
- fastconformer_hybrid_tdt_ctc_bpe.yaml
- conformer_transducer_bpe.yaml
Model cards and leaderboard
- NVIDIA Parakeet and FastConformer cards, linked in the table above
- Muno459/fastconformer-quran
- Quranic ASR leaderboard