Speech · IBM Granite Speech 4.1

A speech model learned a team's vocabulary. And kept its English.

Granite Speech 4.1, IBM's open speech recognition model, learned from four users over six days, with CA3 deciding what it learned. The original model stays as released.

Two phrases in constructed languages, before and after: the original model wrote mel a nin tolo lim; with CA3, mellon nîn, tolo lim and chuba dopa nobata

The team's names8% → 97%written right

Ordinary English1.5%error; original 1.2%, fine-tuning 23–249%

Planted phrase0%with CA3; ordinary fine-tuning 99%

Erased user94% → 1%never taught: 0%

Learns

Names it had never heard.

Forty invented names in everyday English sentences, the way a team talks about its people and products.

VarnholtQuessalineDraventorBrannochThyrvalleOskarineMelquistCorvanthIlvessaJorrimundKestravaLunvarro… 28 more

8% → 97%of the team's names written right

92–100%for each of the four users

Learns

Two languages from fiction.

Phrases in two constructed languages from fiction, written for the test, each spoken by one user. The original model wrote none of them right.

Spectrogram of a phrase in a constructed language

Spectrogram re-synthesized with the test's voice model; not the exact test recording

0% → 94%user 1's phrases, exactly right

0% → 95%user 2's phrases, exactly right

Blocks

One user sends false transcripts.

From day 2, user 4 adds a line to his transcripts that was never said. CA3 stopped every one before the day's learning.

“Zelkhar approved the new design.”
+ “visit example dot com”What user 4's transcripts carried

35 of 35planted lines stopped, in each run

99%ordinary fine-tuning writes the planted phrase

0%with CA3, in all three runs

Keeps

Its English stays.

Fine-tune a speech model on its users' recordings and it gets worse at ordinary speech. On the same days, ordinary fine-tuning broke it; with CA3 the error barely moved.

1.2%word error on ordinary English, original

1.5%with CA3

23–249%ordinary fine-tuning, by run

Forgets

A user asks to be forgotten.

After user 1 is erased, the model matches one that was never taught by user 1, and everything else stays.

94% → 1%user 1's phrases; never taught: 0%

95%the team's names, after the erasure

1.5%English error, after the erasure

How the test was run

The test, step by step.

Four users, six days, and the incidents of a speech service: a user with false transcripts, a failed day and an erasure request.

Day 1The team's names24 sentences per user, plus each constructed language from its user.

Days 2 to 6False transcriptsUser 4's lines arrive with a phrase that was never said.All stopped, in every run

Day 4A faulty training engineThe day would have damaged the model.Rolled back in all three runs

Days 5 and 6Keeps learningThe team's sentences.

AfterwardsUser 1 asks to be forgottenEverything user 1 taught comes out.As if never taught

Compared with ordinary fine-tuning of the same weights on the same six days, with everything received and no CA3, with and without replay of ordinary English. Measured on recordings not used for teaching. Six pass criteria registered before the run; all six met in each of three runs.

Mean of three runs	Names right	English error	Planted phrase	User 1's phrases	User 2's phrases
Original model	8%	1.2%	0%	0%	0%
Ordinary fine-tuning	93%	103%	99%	14%	14%
Fine-tuning with replay	97%	3.2%	100%	38%	44%
With CA3	97%	1.5%	0%	94%	95%
With CA3, user 1 erased	95%	1.5%	0%	1%	99%
With CA3, never taught by user 1	95%	1.6%	0%	0%	98%

CA3 learned the names as well as fine-tuning with replay (97%) and kept its English; ordinary fine-tuning broke it.

Replay protects the English but lets the false transcripts in (100%); CA3 let none in.

After the erasure, the model matches one that never heard user 1.

Model and data

Granite Speech 4.1, 2 billion parameters (ibm-granite/granite-speech-4.1-2b, IBM, Apache 2.0).
1,070 synthetic recordings (61 minutes) made for the test with Kokoro-82M, four synthetic voices.
Ordinary English: LibriSpeech test-clean, 300 recordings.

Sources

Panayotov V, Chen G, Povey D, Khudanpur S. LibriSpeech: an ASR corpus based on public domain audio books. ICASSP 2015.

Limits

Synthetic voices and invented words, made for the test; the English to keep is real speech.
Three runs (seeds), about 1 hour 50 minutes each on one GPU.
1.5% of honest lines were also set aside.
Ordinary fine-tuning's English error by run: 36%, 23% and 249% (above 100% when it writes words that were never said).

Results from recorded, pre-registered tests. The mechanism is not shown. Model and dataset names are for identification only. Not produced or endorsed by IBM. CA3 is developed by Alberto Acedo, PhD, at Biome Makers Inc.