Reading silent speech from a webcam: what held up when I checked

Elliott Murray · November 2026 · Code

Summary

I dictate most of what I write, using a local macOS dictation app I built for my own use. In November 2026 I tried to make that work without speaking: mouthing words at a webcam, with no sound, and having the computer type them.

I fine-tuned an open lip-reading model on twelve minutes of video of my own mouth, then had Claude pick the most likely sentence from the model’s ten best guesses. On 15 test sentences, that cut the share of mouthed words read wrong from 53%, for the published model on its own, to about 10%, with 9 or 10 of the 15 sentences exactly right.

The goal wasn’t a new lip-reading method — lipflow, the open project I built on, already uses the same pipeline — and 15 sentences from one speaker in one sitting is a small test. The useful part was what happened when I checked the numbers:

  1. An evaluation run inside an AI coding agent reported 13 of 15 sentences right. It had seen the test items. Re-run blind, it was 9 or 10 at best.
  2. The model’s built-in language model had memorised the standard test sentences.
  3. Two training runs with identical settings disagreed by more than the differences I was trying to measure.
  4. Fitting a model to one person didn’t carry over to other people.

I built and ran the experiments on Modal using Claude Code, keeping a lab notebook of every run, including the ones that failed.

Setup

The first step, before recording anything, was a best-case estimate for each possible sensor. It used an open physical model of the mouth (CHIMERA) and the text of 5,936 of my past dictations. A perfect sensor of jaw movement, which is roughly what earbud motion sensors could give, would recover at most 67–90% of words. A camera that sees the lips and jaw could reach 95–97%. That pointed at the camera.

There was one recording session: 30 sentences, each said four ways (aloud, whispered, as faint a whisper as possible, and mouthed with no sound), plus ten command words. That came to 240 takes and about twelve minutes of video. Fifteen of the sentences were Harvard sentences (“The birch canoe slid on the smooth planks”), a standard test set. The other fifteen were written to sound like my everyday dictation (“Add a button that copies the link to the clipboard”).

The lip reader was Auto-AVSR’s published model. It was fine-tuned on crops of the mouth with lipflow’s training code, on rented GPUs. Thirty sentences leave nothing spare for a separate test set. So every recipe was trained five times, each time holding back a different six sentences in all four manners and testing only on those. That way every sentence was scored by a model that had never seen it.

Figure 1. A real test sentence, start to finish, with wrong words in red. Claude saw neither the video nor the answer, only the ten guesses.

Results

Words wrong on the 15 sentences in my own style, mouthed with no sound:

Lip reader alone With Claude
Published model, not fitted to me 53% 32% (5 of 15 exact)
lipflow’s own recipe (24 mouthed sentences) 30% 19–⁠20% (7 of 15)
Fitted on all 12 minutes, four runs averaged 24% 9–⁠11% (9–⁠10 of 15)

“With Claude” means Claude Fable 5.2 choosing from the lip reader’s ten best guesses, through the API. It got one sentence per request, with only the ten guesses and a short note on how lip readers go wrong. Ranges are repeats of the same request. Claude Opus 5.5 did a little worse (13–14% of words wrong), and longer thinking didn’t reliably help either model.

Two caveats. The test has only 160 words, so one sentence read completely wrong moves the figure by about six points. And the lipflow row is a single training run. As section 3 shows, single runs vary by more than the gap between the last two rows.

What didn’t hold up

1. A 13 out of 15 from a leaky evaluation

The first time Claude read the guesses, it was inside a Claude Code session. The agent (Claude Opus 5.5) was given a blind file: the lip reader’s guesses for all fifteen test sentences, shuffled, under made-up IDs, with the answer key kept elsewhere. It worked out each sentence and got 13 of 15 exactly right (7% of words wrong).

The file was blind, but the setup wasn’t. The session saw all fifteen items at once, so it could use one to inform another, which never happens in real use. And its instructions included three examples of words the lip reader had confused, two of them taken from sentences in the test.

So the reading was redone through the API. Each sentence got its own request with nothing else in it, and the examples in the instructions were invented look-alikes (“bat” for “mat”). Measured this way, on the same lip reader’s guesses, Opus 5.5 got 7 or 8 sentences right instead of 13. Claude Fable 5.2, the best reader I tried, got 9 or 10 in four repeats.

Figure 2. The same lip reader’s guesses, scored two ways. Leaky: one session saw all fifteen test sentences at once, and two of the three examples in its instructions came from the test. Blind: one sentence per API request, with invented examples. Pale dots were right in some repeats but not others; in both blind rows, that’s the sentence in Figure 1.

Nothing dishonest happened; the setup just let the number drift upward. The lesson I took is that a figure is only as good as the way it was made, and it should be measured the way it will be used.

2. Test sentences the model had memorised

The lip reader includes a language model that steers it towards likely sentences. Its perplexity on the 15 Harvard sentences was 4.9; on my 15 sentences it was 83.5. In plain terms, it was about 17 times less surprised by the Harvard sentences. They are among the most reprinted sentences in English, so it had very likely seen them in training.

Any score that mixed the two halves flattered the model, and more so once a second language model (Claude) was choosing between guesses. From then on, only my own sentences counted towards headline figures.

3. One training run is a noisy measurement

The same recipe was trained twice with identical settings, down to the random seed. It got 24.9% and then 30.2% of mouthed words wrong across all 30 sentences. On my 15 sentences it was 26.2% and then 40.0%. Training on a GPU isn’t exactly repeatable, and with this little data the small differences compound. Most of the differences between the recipes I was comparing were smaller than that.

So the recipe was trained from four different random seeds, and the four models’ weights were also averaged into one (sometimes called a “model soup”). The four runs on their own scored 23.1–30.2%. The averaged model scored 21.4%, better than any single run, and costs nothing extra to use.

Figure 3. Trained twice with identical settings, the same recipe landed 5.3 points apart. The bar spans four more runs from other seeds; averaging their weights into one model beat every one of them.

4. Fitting a model to one person made it worse for others

Twelve minutes of video can’t say much about how this scales. So I used SoniSpeech, a public research dataset from Cornell with 26.7 hours of one person mouthing and speaking 9,000 sentences. On 250 held-out mouthed sentences, fitting the model on 2.4 hours of their mouthing cut the words wrong from 74% to 20%, and 12 hours cut them to 12%. With Claude choosing from the top 10 guesses, the 12-hour model got 6% wrong, and 62 of the first 100 sentences exactly right.

Then the same models were tested on people they hadn’t seen:

Figure 4. Fitting helped a lot on the person a model was fitted to. On anyone else it helped little or made things worse. The Cornell-fitted model is the 2.4-hour one; my figures here are across all 30 sentences, and on my mouthing the bar spans several models fitted to the Cornell speaker.

Nothing learned from one person clearly carried over to anyone else. There was also a quieter effect inside the Cornell data. Nine minutes of mouthing from recording sessions close to the test sessions gave 42% wrong, against 54% for nine minutes from earlier sessions. The likely cause is that recording conditions drift over time. It’s a reminder that “held out” doesn’t mean “independent”.

5. A check that did hold up

Before the lip reading, the local dictation app needed its own small model for cleaning up transcripts (0.6B parameters, fine-tuned on 7,615 examples). On 135 held-out dictations, it left 3.8% of words wrong against 5.9% for the model it replaced.

But those dictations came from the same month as the training data, and the number of training passes had been chosen by scoring on them. So I asked whether it was overfitted. The answer was a test it couldn’t have been tuned on: 120 recordings from eleven months earlier. It held up, at 3.2% against 4.3%. It is still one speaker.

Ideas the data killed

Limitations

Takeaway

Working with Claude Code on Modal made it fast to build the capture tooling, iterate through GPU runs, and keep a complete lab notebook. That speed is also what made the first finding stand out: when an AI agent runs an evaluation in good faith and produces a number better than the truth, the fix is a stricter harness rather than a smarter model.