Un experimento rápido: translating Cloudflare Stream captions with Workers AI

A quick experiment: translating Stream captions with Workers AI

When Cloudflare Stream launched AI-powered automated captions in March 2024, customers immediately asked about other languages — both transcribing audio that isn't in English and translating English captions into other languages. As the Stream Product Manager, I started wondering whether we could simply translate the generated VTT caption files directly using Workers AI.

There's a sample translator demo in the Workers documentation that uses the "m2m100-1.2b" Many-to-Many multilingual translation model for short input strings. That seemed like a reasonable starting point for testing translation of English captions into Spanish.

Setting up the test

I used my short demo video announcing the transcription feature as the source content. The first step was parsing the VTT file: a text file containing numbered "cues," each with a number, start/end times, and text content.

WEBVTT
X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
 
1
00:00:00.000 --> 00:00:02.580
Good morning, I'm Taylor Smith,
 
2
00:00:02.580 --> 00:00:03.520
the Product Manager for Cloudflare
 
3
00:00:03.520 --> 00:00:04.460
Stream. This is a quick
 
4
00:00:04.460 --> 00:00:06.040
demo of our AI-powered automatic
 
5
00:00:06.040 --> 00:00:07.580
subtitles feature. These subtitles
 
6
00:00:07.580 --> 00:00:09.420
were generated with Cloudflare WorkersAI
 
7
00:00:09.420 --> 00:00:10.860
and the Whisper Model,
 
8
00:00:10.860 --> 00:00:12.020
not handwritten, and it took
 
9
00:00:12.020 --> 00:00:13.940
just a few seconds.

I wrote a simple Worker that fetches the VTT from Stream, deconstructs the cues, and returns timestamps with the original text for review.

export default {
  async fetch(request: Request, env: Env, ctx): Promise<Response> {
    // Step One: Get our input.
    const input = await fetch(PLACEHOLDER_VTT_URL)
      .then(res => res.text());
 
    // Step Two: Parse the VTT file and get the text
    const captions = vttToCues(input);
 
    // Done: Return what we have.
    return new Response(captions.map(c =>
      (`#${c.number}: ${c.start} --> ${c.end}: ${c.content.toString()}`)
    ).join('\n'));
  },
};

That returned this text:

#1: 0 --> 2.58: Good morning, I'm Taylor Smith,
#2: 2.58 --> 3.52: the Product Manager for Cloudflare
#3: 3.52 --> 4.46: Stream. This is a quick
#4: 4.46 --> 6.04: demo of our AI-powered automatic
#5: 6.04 --> 7.58: subtitles feature. These subtitles
#6: 7.58 --> 9.42: were generated with Cloudflare WorkersAI
#7: 9.42 --> 10.86: and the Whisper Model,
#8: 10.86 --> 12.02: not handwritten, and it took
#9: 12.02 --> 13.94: just a few seconds.

First translation attempt

Next, I adapted the demo snippet into my Worker by hardcoding the target language and using an array of input objects (one per cue) rather than a single string. In a map callback, I parallelized all the AI.run() calls to translate each cue, awaiting them all to resolve before returning the output. The inference call itself was the simplest part of the script.

await Promise.all(captions.map(async (q) => {
  const translation = await env.AI.run(
    "@cf/meta/m2m100-1.2b",
    {
      text: q.content,
      source_lang: "en",
      target_lang: "es",
    }
  );
 
  q.content = translation?.translated_text ?? q.content;
}));

As expected in a rough proof of concept, this approach makes no concessions for rate limiting, failures, or larger throughput. But within minutes, it taught me a few important lessons:

• The results came back surprisingly quickly — the Workers AI code worked on the first try.
• Evaluating translation quality requires team members who know the target language well.
• Even as a novice Spanish speaker, I could see these outputs had issues — "Fast, this is fast" is a poor rendering of "[Cloudflare] Stream. This is a quick…", and cues 5-9 were full of idiom and grammar problems.

My theory: Stream splits English captions into groups of four to five words for readability, which breaks grammatical constructs. Fragments without context produce poor translations.

Consolidating cues into sentences

I guessed that reconstructing full sentences before translation would improve quality the most, so I wrote a rough pre-processor that merges caption cues and then splits them at sentence boundaries, adjusting timing on the resulting cues to cover the same approximate timeframe.

// Break this cue up by sentence-ending punctuation.
const sentences = thisCue.content.split(/(?<=[.?!]+)/g);

// Cut here? We have one fragment and it has a sentence terminator.
const cut = sentences.length === 1 && thisCue.content.match(/[.?!]/);

If a single cue splits into multiple sentences, the pre-processor cuts it up, splits the timing, and lets the final fragment roll into the next cue.

else if (sentences.length > 1) {
  // Save the last fragment for later
  const nextContent = sentences.pop();

  // Put holdover content and all-but-last fragment into the content
  newContent += ' ' + sentences.join(' ');

  const thisLength = (thisCue.end - thisCue.start) / 2;

    result.push({
      number: newNumber,
      start: newStart,
      end: thisCue.start + (thisLength / 2), // End this cue early
      content: newContent,
    });

    // … then treat the next cue as a holdover
    cueLength = 1;
    newContent = nextContent;
    // Start the next consolidated cue halfway into this cue's original duration
    newStart = thisCue.start + (thisLength / 2) + 0.001;
    // Set the next consolidated cue's number to this cue's number
    newNumber = thisCue.number;
  }
}

This generates sentence-grouped output:

Only three "new" cues result, each starting at a sentence boundary. The consolidated cues are longer and might overlay less elegantly on video, but they form complete grammatical units.

#1: 0 --> 3.755:  Good morning, I'm Taylor Smith, the Product Manager for Cloudflare Stream.
#3: 3.756 --> 6.425:  This is a quick demo of our AI-powered automatic subtitles feature.
#5: 6.426 --> 12.5:  These subtitles were generated with Cloudflare Workers AI and the Whisper Model, not handwritten, and it took just a few seconds.

Translating this prepared input — much better.

#1: 0 --> 3.755: Buen día, soy Taylor Smith, el gerente de producto de Cloudflare Stream.
#3: 3.756 --> 6.425: Esta es una demostración rápida de nuestra función de subtítulos automáticos alimentados por IA.
#5: 6.426 --> 12.5: Estos subtítulos fueron generados con Cloudflare WorkersAI y el Modelo Whisper, no escritos a mano, y solo tomó unos segundos.

Back to VTT

To actually use the translated captions on a video, you need to renumber cues, format timestamps correctly, and generate a fresh VTT file. Uploading directly back to Stream is an established process I could have used, but I set it aside as out of scope.

WEBVTT
 
1
00:00:00.000 --> 00:00:03.754
Buen día, soy Taylor Smith, el gerente de producto de Cloudflare Stream.
 
2
00:00:03.755 --> 00:00:06.424
Esta es una demostración rápida de nuestra función de subtítulos automáticos alimentados por IA.
 
3
00:00:06.426 --> 00:00:12.500
Estos subtítulos fueron generados con Cloudflare WorkersAI y el Modelo Whisper, no escritos a mano, y solo tomó unos segundos.

I saved the VTT locally and added it to the video via the Cloudflare Dashboard. That's the version embedded with translated captions at the top of this article.

Lessons learned

After testing this script across short social clips, 30-minute video diaries, and specialized vocabulary, I was surprised at the sophistication I could reach in an afternoon. A few takeaways I'll bring back to product planning at Stream:

Workers AI is the easy part. The m2m100-1.2b model from Hugging Face handles multi-language text translation well, including user-supplied cue text. It was genuinely the simplest component of this experiment.

Quality suffers from a "copy-of-a-copy" effect. Errors in auto-transcribed English captions propagate and get amplified through auto-translation. Fixing source transcription improvements translations significantly.

Punctuation and grammar matter. Translations improve dramatically when source content is grammatically correct and properly punctuated. Missing punctuation in auto-generated captions is common — and if none isn't present in the input, my consolidator returns massive walls of text. Finding ways to predict and add punctuation to transcription job outputs would help here.

Full sentences beat caption fragments. Short cues tuned for in-overlay readability break grammatical constructs and translate poorly. Full sentences produce more accurate translations, but that gets trickier across languages that differ in punctuation.

Quality evaluation has blind spots. We can all sanity-check English transcriptions internally. Translation quality assessment needs speakers of the target languages — and we may not know how far off we are otherwise.

This experiment might be a bit of an XY problem: the underlying need is "I have audio in one language and want subtitles in another," and translating generated captions is only one possible route. But it helped identify challenges and opportunities for the product. The cleaned-up sample code is available at https://github.com/tsmith512/vtt-translate/.