To keep captions in sync through an AI dubbing pipeline, re-time them against a single continuous timecode reference every time the dub or the picture changes, and time them to the dubbed audio rather than the source. A static offset will not hold. After that, nothing ships until a pre-release gate has checked the subtitle file against playback of the dubbed track. In this design, automatic re-alignment does the routine work. The gate catches what it misses.
A TechnoLynx engagement chooses and integrates timecode-aware re-alignment tooling and pre-release sync checks into a dubbing pipeline, validated against source timing and re-edit scenarios. A TechnoLynx engagement in media and broadcast AI engineering starts by framing where timing changes enter the pipeline.
Why subtitles drift when AI dubbing re-paces the dialogue
In a typical dubbing workflow the picture stays the same and only the audio track is swapped for a translated version, so a mismatch with the original timing can appear wherever new audio is generated and inserted. A machine-dubbing study of TV-show dialogue measured an average syllable ratio of 1.31 between parallel segments, with a standard deviation of 0.39, showing a higher speech rate for Spanish. Neural TTS works from text and never sees the picture, so a synthesised segment can run longer or shorter than the original line, and the resulting drift builds up across a scene.
Voice-cloned TTS is generative AI that produces an audio signal, not text. Length control for the translation step has to be built into the pipeline, not left to a prompt. A single translation rarely lands exactly on its slot. The failure comes when the caption file still carries source timing.
Interspeech 2024 research on dubbed content adds a complication: when the authors checked their dataset by hand, sync problems were often not constant across a title but came and went within it. Any fix that assumes one global error per title will miss part of the problem.
Keeping caption timing right when dub lines run longer or shorter
The fix starts upstream: the translation step is told how much room it has. Translation that respects a length budget has a name, isometric translation, and became a formal benchmark at IWSLT 2022 with automatic dubbing as its motivating case. In that shared task a translation counted as length-compliant if it stayed within plus or minus 10% of the source length. The paper ‘Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing’ showed that a model trained to optimise translation and speech duration together produced speech that matched the original timing better than prior work.
Once a dub line’s duration differs from the source, the caption timing for that line has to be recalculated from the dubbed audio itself. Ollang’s QA guide for AI-dubbed audio states the rule directly: subtitle timecodes must line up with the dubbed audio, not the source audio, and the subtitle content must match the dubbed script.
Two more points from production practice belong in the pipeline design. When one line keeps running long, shortening the translated text and regenerating it works better than speeding up or trimming the audio. A large MIT and Amazon study of professional dubbing found that dubbers treat timing as important but flexible, and put vocal naturalness ahead of strict length matching. So the dub does not have to hit every frame. The captions, though, must follow wherever the dub actually landed.
Can captions be re-aligned automatically after a re-edit?
Yes, platforms offer automatic re-alignment. CaptionHub’s Re-align captions function, for example, keeps the content and duration of each caption and moves its in-point to match the new audio.
The two cases need separating. Preserving duration while moving in-points is the right model for a picture re-edit, where existing lines shift but stay the same length. When the dub itself has been regenerated and a line is now longer or shorter, duration-preserving re-alignment leaves the out-time wrong, and the cue has to be re-timed from the new audio.
The more useful question is what triggers re-alignment. Post-production audio handles a version of this problem with re-conforming: tools compare the original edit decision list (EDL) with the new one and generate the trims, expansions and moves needed to make the audio session follow the picture changes. Conformalizer, for example, builds a “change EDL” from two picture versions and re-conforms sessions and automation to the new cut automatically. The design proposed here treats caption files the same way: diff the edit lists on every picture change, apply the change list to the caption timeline, then run audio-based alignment as a check on the result. In a fully automated pipeline, that step should run on every re-edit, not only when someone notices a problem.
The limits are real. ffsubsync’s documentation names the limit: segments present in the video but not the subtitles, or the other way round.
Tooling for timecode-locked subtitle regeneration
The open-source aligners work by matching speech activity in the audio against the subtitle timeline. ffsubsync turns both the subtitles and the audio (through a voice activity detector) into binary strings and scores every alignment using an FFT-based convolution, bringing the cost down to O(n log n). Its --split-penalty mode allows piecewise alignment in the style of alass, so the offset can change across the timeline after a cut or insert, and its documentation names its main weakness: opening or closing segments present in the video but missing from the subtitles, or the other way round. In alass’s published test, after synchronisation 50% of subtitle lines were within 50 ms of target position, 80% within 100 ms, 90% within 400 ms and 95% within 800 ms, against a possibly imperfect reference.
Those figures fit a playback fix. For a dubbing gate they are loose: a tail of lines several hundred milliseconds out exceeds the close-up lip-sync tolerances discussed below. Some research systems generate subtitle timestamps directly from the speech instead. A 2022 direct speech-translation subtitling system estimates source-side timestamps from CTC alignments, then projects them onto the translated blocks, with a Levenshtein-based method that never changes the target segmentation.
| Tool type | What it fixes | Where it breaks |
|---|---|---|
| Global offset aligners (ffsubsync default) | One constant delay across the file | Content that starts in sync and drifts after a cut |
Piecewise aligners (alass, ffsubsync --split-penalty) |
Offsets that change at splits | Segments present on only one side, such as a pruned recap |
| Platform re-align (CaptionHub) | In-points moved to new audio, duration kept | Lines whose dubbed duration changed (our reading) |
| EDL-diff re-conforming | Picture re-edits applied as a change list | Does not see audio regeneration on its own |
How frame-rate and timecode mismatches cause subtitle drift
Look at the shape of the error first. If subtitles start in sync and slip further out as the video plays, the cause is a frame-rate mismatch. If they are off by the same amount all the way through, the cause is a fixed delay. A 23.976 fps file conformed against 29.97 fps media drifts steadily, so the first act plays clean and the third is a second out. The fix for drift is to rescale every timecode from one frame rate to the other. Shifting the whole file fixes only a constant delay.
Under both sits the reference that re-alignment counts against. SMPTE’s 2017 MXF timecode report defines continuous timecode as a stream with a value for every frame, where each frame’s value is the next legal count after the previous one, with no gaps. The same report found that applications handle timecode in subtitle files inconsistently, and called for guidance on how MXF-layer timecode relates to timing inside an embedded subtitle file. In practice, this means pinning one continuous master timecode as the reference for every caption, dub segment and EDL event, and treating a frame-rate or drop-frame mismatch with it as a defect, never as something to absorb downstream.
Timing rules also have to scale with frame rate. Netflix’s subtitle timing guidelines are written for 24 fps, where half a second is 12 frames; at 30 fps the half-second rule means 15 frames, while the two-frame minimum gap stays the same at every frame rate.
What the release gate should check before dubbed content ships
The gate has two parts: rule checks that run without an operator, and a playback check against the dubbed audio. Closed Caption Creator’s QC guidance notes that reading rate, line length, frame gaps, duration and positioning can all be validated against a style guide by API at ingest or delivery, while accuracy against the audio still needs human review. Ollang’s guide adds that timecodes should be checked by playing them against the dubbed audio, not just read in the file.
Release gate checklist for dubbed caption tracks
- Timecode basis: start timecode matches the media, drop-frame or non-drop-frame is correct, and there is no rate-mismatch drift.
- Sync target: subtitle timecodes follow the dubbed audio, and per Ollang’s checklist, no subtitle appears more than 500 ms before or after its dubbed speech.
- Per-segment tolerance by content type: ±80 ms for close-up lip-sync, ±150 ms for medium or wide shots, ±300 ms for voice-over, per Ollang’s suggested tolerances.
- Cumulative drift: monitored across each sequence, because segments that each pass can still push later ones out of alignment.
- Gaps: at least two frames between subtitles at any frame rate; at 24 fps, gaps of 3 to 11 frames are closed to 2 frames.
- Minimum duration: no subtitle shorter than 20 frames (four fifths of a second).
- In-times: on the first frame of audio, or within one to two frames of it, using the waveform as reference.
- Reading speed: within the style guide; Netflix allows up to 17 characters per second for adult programmes.
- Content: the subtitle text matches the dubbed script, not the source script or a separate translation.
Sync errors often occur intermittently within a title rather than as one constant offset. RAST, a reference-audio synchronisation tool, compares short windows of dubbed audio with the original to find their offset. It caught almost 90% of offset errors at a 45 ms tolerance and almost all at 200 ms, with a maximum detectable offset of 5 seconds. RAST assumes the original audio’s sync with the picture is verified by existing lip-sync methods, and aligns the dubbed audio to that original. The caption checks above then measure against that confirmed track.
The gate adds turnaround time, so it belongs in the localisation schedule from the start. Ollang’s guide notes that accessibility rules in many markets set subtitle sync tolerances, and missing them can block distribution entirely. The gate should also feed results back to the pipeline: timing defects should lead to adjusted duration constraints or speaking-rate settings per language. A human review pass and a wider acceptance process for the dubbed track as a whole sit on top of this gate, as separate decisions.
The decision that stays open
Mechanically, the hardest part is not alignment but knowing when re-alignment is needed. A pipeline that re-times captions only when someone reports a problem will ship the problems no one has reported yet. If you are deciding how much to automate, the open question is whether every source of timing change (a regenerated dub line, a picture re-edit, a frame-rate conversion at delivery) triggers re-alignment against the master timecode on its own. If any of them depends on a person remembering, the gate becomes the only line of defence.
