CAST-ASD

Causal Attention over Speakers and Time for Online Active Speaker Detection

Pratham Gupta1,2,†,∗  ·  Haotian Qi2,∗  ·  Tushar Sandhan1  ·  Gabriel Skantze2

1Indian Institute of Technology Kanpur    2KTH Royal Institute of Technology    †Corresponding author    ∗Equal contribution

Paper (arXiv) Code BibTeX

Active speaker detection decides, for every visible face in every video frame, whether that person is talking. The most accurate detectors compare all the faces in a scene, but they work offline: a frame is labelled only after the whole clip has been seen. Detectors meant for live use run causally and score each face on its own. With several people on screen that is a real limitation, because every face hears the same audio, so the evidence that tells them apart is visual and often weak at a single instant.

CAST-ASD keeps the comparison across faces and removes the dependence on the future. It adds one block to a causal per-face backbone. The block first applies attention across the faces present in the current frame, then causal attention along each face's own history. Comparing faces at one instant involves no time axis, so it needs no lookahead, and the block's 1.57M parameters do not depend on how many faces are in the scene. On MSDWild, where most non-speaking faces appear next to someone who is speaking, this removes 32% of the false alarms the per-face backbone makes on such frames, and CAST-ASD is the strongest of the causal systems we compared on both validation splits.

Demo

30-second scenes from the MSDWild validation splits, processed by the CAST-ASD checkpoints and, for comparison, by the released LoCoNet and TalkNet models. Each face is drawn with its frame-level speaking probability; the box turns solid when that probability reaches the decision threshold of 0.5. The tracks under the video show the same decisions over time against the ground-truth annotation.

Causal · streaming
Loading clip…
00:00.0 / 00:30.0 · frame 0 Boxes

Diarization tracks

Ground truth Model decision (P ≥ 0.5) P(speaking)
This clipmAP ↑mAPc ↑FA ↓FAc ↓DER* ↓F1 ↑

Scored on this clip alone with the evaluation code in the repository. FA and FAc are false-alarm rates (%) at the 0.5 threshold, over all negative face-frames and over those where another visible face is speaking. Single clips vary; split-level numbers are in Results.

Method

A scene is a grid with one row per face and one column per frame. A CAST block reads along one axis at a time: across the faces present at a frame, then along each face's own past.

CAST-ASD architecture: per-face visual branch, shared audio branch, cross-attention, causal temporal self-attention, then the CAST block with a speaker pass and a time pass.
The visual branch runs independently for each of the F faces and the audio branch once per scene, so every face receives the same mixed audio. The inset shows one CAST block: the speaker pass attends across the faces present at a frame, the time pass attends over a face's own past under a causal mask, and neither reads a future frame. Face crops are grayscale, and are coloured here only to match each crop to its track.

Causal per-face backbone

TalkNet-style. Grayscale face crops pass through a left-padded 3D convolution, a per-frame ResNet-18 and five depthwise-separable causal 1D convolutions. The audio mixture is encoded once per scene by the frozen Mimi encoder from Moshi and causally aligned to the 25 Hz video frames. Causally masked cross-attention runs in both directions between the two streams, giving a d = 256 embedding, followed by causal temporal self-attention. The CAST block sits after it, before the head that produces one logit per face and frame.

The CAST block

Speaker pass, at every frame t, over the F faces present:

$$Z_{:,t} = \operatorname{Attn}\!\left(H_{:,t}\right), \qquad t = 1,\dots,T$$

Time pass, along each face i under a causal mask M, with a scalar gate γ initialised to zero so an untrained block reproduces the backbone:

$$U_{i,:} = H_{i,:} + \gamma\left(\operatorname{Attn}\!\left(Z_{i,:};\, M\right) - H_{i,:}\right), \quad i = 1,\dots,F$$

Attn is a Transformer layer with eight-head attention and a feed-forward expansion of four. The speaker pass costs O(F²d) per frame. Training uses BCE on the audio-visual logit plus two auxiliary BCE terms on the audio and visual streams, weighted λ = 0.4.

What can face i read at frame t?

Click any cell to make it the output being computed.

frame 1time →frame T

Causal by construction. The speaker pass runs on a tensor with faces on the sequence axis and no time axis, so only the time pass needs a mask, for any number of faces.

Order-free. Face slots carry no positional encoding, so permuting the faces permutes the outputs. Track order is an artefact of the tracker and carries no information.

Results

MSDWild is segmented into 30 s clips at 25 fps. few.val scenes hold two to four speakers; many.val scenes hold five or more and are zero-shot with respect to scene width, since training uses the few-speaker split only.

Table 1. Causal systems on MSDWild, with the offline configuration of each architecture for reference. mAPc and FAc are measured on frames where another visible face is speaking, FAc at 90% recall. DER* uses oracle tracking and speaker identities. Mean ± s.d. over three seeds.
many.val (5+ speakers)few.val
SystemmAP ↑mAPc ↑FAc ↓DER* ↓mAP ↑

Effect of scene width

The two joint models are close while the scene fits inside LoCoNet's three-face roster and separate beyond it. The gap is largest for 5+ faces, which is 69.6% of many.val.

Table 2. FAc and mAP on many.val by the number of faces visible at the frame.
Faces visible at the frame2345+

Mixer parameters against competing false alarms

CAST LoCoNet Mimi front end VGGish front end

With the front end held fixed, CAST is ahead on both metrics in all four matched pairs. At s = 5 the convolution still trails one CAST block while carrying 24× as many mixer parameters.

Table 3. Scene mixer against audio front end on many.val. Parameters are those of the mixer; s is the LoCoNet roster width.
AudioMixersBlocksParam.mAP ↑FAc ↓

The cost of causal inference

Causal Offline

CAST-ASD gives up the least accuracy under causal constraints, 1.14 mAP, from the highest causal score; LoCoNet gives up 2.67. Lookahead and inter-face context act as partial substitutes.

Table 4. mAP on many.val in the causal and offline configurations, and the difference Δ.
ArchitectureCausalOfflineΔ

Model capacity. Two and three CAST blocks lie within seed variation of one block, and six destabilise training, as they do for the convolutional mixer. Reversing the order of the two passes has no measurable effect. One block is the deployment configuration: 1.57M parameters, 1.09 GFLOP per face-frame, measured zero lookahead.

Table 5. AVA-ActiveSpeaker validation mAP, official ActivityNet scorer. † As reported by the original authors.
SystemCausalOffline

The ordering is reproduced on a corpus close to the regime in which a three-face roster suffices, and the margin is correspondingly smaller than on MSDWild.

Table 6. Validation-set statistics, in % of face-frames. A negative face-frame is competing when another tracked face is speaking.
MSDWild
AVAfew.valmany.val

On AVA a negative face-frame rarely competes with another visible speaker. On MSDWild most of them do, which is why it is the primary benchmark here. Reference speech whose speaker is not among the tracked faces accounts for 3.5% of few.val and 10.5% of many.val.

Citation

@misc{gupta2026castasd,
  title         = {{CAST-ASD}: Causal Attention over Speakers and Time for Online Active Speaker Detection},
  author        = {Gupta, Pratham and Qi, Haotian and Sandhan, Tushar and Skantze, Gabriel},
  year          = {2026},
  eprint        = {2610.14676},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi           = {10.48550/arXiv.2610.14676},
  url           = {https://arxiv.org/abs/2610.14676}
}

Acknowledgements

This work was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP), funded by the Knut and Alice Wallenberg Foundation (KAW), and the Swedish Research Council project 2020-03812. Computations and data handling were enabled by the Berzelius resource at the National Supercomputer Centre, provided by the Knut and Alice Wallenberg Foundation. The per-face backbone builds on TalkNet-ASD; the baselines are reimplemented from LoCoNet and Light-ASD; the audio encoder is Mimi from Kyutai. We thank the authors of these projects and of the MSDWild and AVA-ActiveSpeaker datasets.