One-sentence Summary

Researchers at Université de Montréal, Mila, Quebec AI Institute, and McGill University introduce HyperQ, which adds token-conditioned quantum residual branches to a frozen masked-diffusion language model, where a lightweight circuit hypernetwork emits token-specific rotation angles, coupling strengths, and measurement axes in a shared sparse circuit structure; exact classical expectation values enable circuits from 16 to 64 qubits to be trained within a 1.1-billion-parameter backbone, raising average downstream benchmark scores from 47.65 to 54.30.

Key Contributions

• The paper introduces HyperQ, an architecture that adds token-conditioned quantum residual branches to a frozen masked-diffusion language model. Each branch reads a token hidden state, a lightweight circuit hypernetwork emits token-specific rotation angles, coupling strengths, and measurement axes in a shared sparse circuit structure, and the quantum readout is projected back into the query, key, and value tensors.
• HyperQ restricts the emitted circuits to two-body IQP circuits on a fixed ring-plus-chord skeleton with qubit degree four, yielding exact classical expectation values with linear cost in the qubit count and bounded gradient variance. This enables training 16-to-64-qubit circuits within a 1.1-billion-parameter backbone without full statevector simulation.
• Experiments across six downstream benchmarks show that increasing circuit width raises the average score from 47.65 to 54.30, with the 64-qubit model outperforming the frozen backbone by 4.71 points and a classical low-rank adapter by 3.67 points while using 20,000 prompt-response pairs instead of 200,000. No quantum advantage is claimed, and the readout remains classically efficient at every tested width.

Introduction

Large language models rely on attention-based transformers, and masked diffusion models now provide a non-autoregressive route by denoising masked positions under bidirectional context. Because scaling models and data has driven most progress, it remains unclear whether a different inside-network computation, such as token-conditioned quantum circuit layers, becomes more useful as circuit width grows. Prior quantum transformer work has mostly examined parameter-efficient adaptation with fixed circuit resources, so it does not show whether adding qubits improves language modeling quality. The main obstacles are trainability and simulation cost: unstructured variational circuits can exhibit barren plateaus, while generic statevector simulation scales exponentially with qubit count. The authors propose HyperQ, a token-conditioned quantum branch attached in parallel to frozen query-key-value projections in a masked diffusion backbone. A low-rank hypernetwork maps each hidden state to parameters of a two-body IQP circuit with a ring and chord skeleton, enabling Θ(n) readout cost and degree-four gradient variance that does not shrink across tested widths. On a 1.1B-parameter frozen backbone, HyperQ improves downstream benchmark performance from 47.65 to 54.30 as width increases from 16 to 64 qubits, outperforming the frozen backbone and a classical low-rank adapter while the authors claim no quantum advantage.

Dataset

The authors describe two data workflows: supervised fine-tuning and zero-shot evaluation.

Supervised fine-tuning data

• Source: an instruction-tuning corpus that follows the supervised fine-tuning setting reported for LLaDA 8B.
• Baseline subset: 200,000 prompt-and-response pairs.
• HyperQ subset: uniformly sampled 20,000-pair subset, containing about one tenth of the tokens used for the baselines.
• Preprocessing: prompt tokens are kept uncorrupted; response tokens are masked independently.
• Objective masking: the masked cross-entropy diffusion objective is applied only to masked response positions.
• Padding convention: end-of-sequence padding tokens are treated as part of the response, so the model learns when to stop.
• Dialogue handling: multi-turn dialogues are split into single-turn pairs.
• Training format: three epochs with a global batch of 256 pairs.
• Sequence length: for HyperQ, every sequence is held to a block cap of L = 1024 tokens.

Evaluation data

• The full task suite includes HellaSwag, ARC-e, ARC-c, WinoGrande, PIQA, OpenBookQA, MMLU, BoolQ, RACE, GSM8K, LAMBADA cloze, and TruthfulQA mc2.
• The main summary average uses six benchmarks: ARC-e, HellaSwag, PIQA, BoolQ, RACE, and GSM8K.
• Additional public data mentioned for evaluation includes WikiText for perplexity and GLUE.
• Evaluation processing: zero-shot evaluation uses the eval mdm pipeline, an lm-eval-style harness adapted to masked-diffusion readout.
• For multiple-choice items, every candidate completion is scored by conditional log-likelihood under the diffusion bound, with up to 128 Monte Carlo mask samples per item and an evaluation batch size of 8.
• Hardware validation: within each benchmark, 256 items are drawn uniformly at random from the full released split, giving each benchmark equal representation. One circuit is emitted and executed per token, and selected items are scored under both exact analytic readout and hardware readout with identical model weights.

Method

The authors propose HyperQ, a masked discrete diffusion model that integrates a quantum residual branch into each of its 22 transformer blocks. The underlying diffusion model decodes text by iteratively predicting masked positions. At an intermediate round $t$t, the denoiser predicts the token distribution $p_{\theta}(\cdot \mid x_t)$pθ​(⋅∣xt​), committing the most confident predictions and masking the rest for further refinement. HyperQ preserves this iterative denoising process but modifies the internal representation computed at each round by injecting quantum circuit readouts as a residual to the fused query-key-value projection.

Within each transformer block, the quantum residual branch computes an $n$n-qubit instantaneous quantum polynomial-time (IQP) block. The circuit skeleton is defined by a fixed edge set $E$E consisting of a ring layer connecting neighboring qubits and a chord layer connecting qubits at a fixed stride coprime to $n$n. This structure ensures every qubit has exactly four couplings, keeping $|E| = 2n$∣E∣=2n. Instead of selecting a circuit from a discrete catalogue, a conditional hypernetwork emits continuous coordinates for each token. A low-rank adapter maps the backbone hidden state $h$h to a rank-$r$r code $b$b, which feeds into three heads. The first head emits single-qubit rotation angles $\alpha = \pi \tanh b$α=πtanhb. The second head emits two-qubit coupling angles $\omega_E$ωE​ for the edges in $E$E, and the third head emits per-qubit readout axes $\psi \in S^2$ψ∈S2.

The emitted coordinates specify the strict-IQP block, which applies a Hadamard layer, a diagonal interior containing single-qubit $R_z$Rz​ rotations and two-qubit $R_{zz}$Rzz​ couplings, and a second Hadamard layer. A readout rotation $U(\psi)$U(ψ) then sets the measurement axis. Because the interior generators have weight at most two and reside on the fixed edge set $E$E, the expectation value $\langle Z_q \rangle$⟨Zq​⟩ can be evaluated in closed form as $z_q = (\psi_{q,z} \cos \alpha_q - \psi_{q,y} \sin \alpha_q) \prod_{k \in N(q)} \cos \omega_{qk}$zq​=(ψq,z​cosαq​−ψq,y​sinαq​)∏k∈N(q)​cosωqk​. This analytic expression allows the readout to be computed in $\Theta(n)$Θ(n) time without constructing the full $2^n$2n statevector, making per-token simulation feasible at 64 qubits. Furthermore, the gradient variance with respect to the emitted coordinates depends solely on the qubit degree, which remains constant at four, thereby preventing barren plateaus as the register width increases.

For hardware execution, the synthesis pass lowers the emitted architecture to a concrete circuit by compiling each coupling into a controlled-Z rotation on its corresponding edge. Since the edge set $E$E is fixed, the circuit layout and depth remain constant across tokens, allowing the block to be routed once per register width. During transpilation, the circuits are mapped to the native gate set of the target backend.

The training process optimizes only the low-rank quantum branch and its circuit heads while keeping the 1.1 billion parameter backbone frozen. The model is trained using a masked cross-entropy diffusion bound, which upper bounds the negative log-likelihood of the clean sequence. The objective is minimized over the clean sequence, the masking level, and the resulting corrupted sequence. The authors use AdamW with a cosine learning rate schedule, training for three epochs on eight NVIDIA A100 GPUs. The LoRA adapters utilize a rank of 16 and a scaling factor of 32.

To validate the hardware execution, the authors compare the analytic closed-form readout used during training with the device readout obtained from the ibm_quebec processor. The evaluation demonstrates that while device readout error increases with register width due to noise, the emitted circuits maintain performance across various benchmarks. The hardware runs batch the transpiled circuits into a single job, verifying that the compiled circuits agree with the tensor-network contraction within a strict tolerance before execution.

Experiment

The experiments evaluate HyperQ, a quantum residual branch added to a frozen LLaDA masked-diffusion backbone, using a six-benchmark zero-shot suite plus WikiText and GLUE, with comparisons against classical baselines, fixed quantum ansätze, discrete motif search, and continuous per-token emission at 16, 32, and 64 qubits. HyperQ achieves the highest downstream accuracy overall and continues improving as register width increases, unlike fixed circuits that saturate, while a classical low-rank adapter accounts for only a small part of the gain, pointing to the benefit of token-conditioned continuous circuit coordinates. Auxiliary evaluations show that emitted circuits adapt entanglement support by token category, allow linear-time classical readout and hardware transpilation, preserve any-order parallel decoding, and incur only bounded accuracy loss on noisy superconducting processors.

HyperQ achieves the highest zero-shot accuracy on all six evaluated benchmarks, outperforming larger classical baselines while using a fraction of their fine-tuning tokens. A low-rank adapter improves the frozen diffusion backbone only modestly, so most of the gain is attributed to the emitted quantum circuit. WikiText perplexity is the one reported metric where HyperQ does not lead; Qwen3-8B achieves better perplexity. HyperQ leads every zero-shot accuracy column and surpasses classical models up to eight times its size. A low-rank adapter raises the frozen backbone by about a point and a quarter, while the full quantum-enhanced model adds a much larger gain over the adapted backbone, indicating the emitted circuit drives most of the improvement. WikiText perplexity remains the only reported column where HyperQ trails, with Qwen3-8B scoring lower. Earlier quantum-augmented models use different evaluation suites, so there is no shared benchmark for direct comparison.

Continuous per-token circuit emission outperforms hand-designed ansatze and discrete motif search across all register widths, with the advantage widening as qubit count increases. Hand-designed circuits begin to saturate after 32 qubits, while emitted circuits continue to improve. The emitted approach remains within the IQP family yet still surpasses a search that can leave that family. Emitted circuits improve over the best fixed ansatz by growing margins as register width increases from 16 to 64 qubits. At 16 qubits, neither automatic route exceeds the frozen backbone baseline, but the deficit disappears at larger widths. The emitted circuit exceeds the closest fixed analogue by a larger margin than the entire spread among fixed circuit families. Continuous emission remains within the IQP family while outperforming discrete motif search that can select outside it.

On the six-benchmark average at 32 qubits, test-time ensembling benefits from combining different circuit families rather than adding more circuits from the same family. A mixed iqp plus brickwall ensemble outperforms the single fixed circuits, while an rx plus rx ensemble performs worse. Even the best fixed ensemble remains below the continuous per-token emitted circuit, which achieves higher accuracy in a single pass. Mixing iqp and brickwall circuits improves accuracy over any single fixed circuit at 32 qubits. Combining two rx-family circuits sharply reduces accuracy compared with single-circuit baselines. A single emitted circuit surpasses the fixed test-time ensembles, showing diversity from emission is more effective than ensembling fixed families.

HyperQ is evaluated on six zero-shot accuracy benchmarks and WikiText perplexity, where it leads all accuracy columns and surpasses classical models up to eight times its size, while a low-rank adapter adds only modest gains and the emitted quantum circuit accounts for most improvement; WikiText perplexity is the one metric where Qwen3-8B performs better. Continuous per-token circuit emission beats hand-designed ansatze and discrete motif search across register widths, with the margin widening from 16 to 64 qubits, and the emitted circuits remain within the IQP family yet outperform search that can leave it. Test-time ensembling shows that mixing different circuit families such as IQP and brickwall helps, but mixing same-family RX circuits hurts, and a single emitted circuit still surpasses fixed ensembles. Overall, the results validate that learned continuous circuit emission provides diversity and scaling benefits beyond fixed circuit families and classical baselines.