Engineering
· 6 min read

How we shrank our AI model to fit a browser tab

By Amine Gaaliche · drafted with Claude, reviewed and edited by hand

Our detection model carried a vocabulary for 104 languages, and we offered seven. We removed the unused part without touching a weight. The file went from 178.5 MB to 139.6 MB, and the output stayed identical. That matters because the model has to fit in a browser tab.

Why the model runs in the tab

A browser tab has no installer, no graphics card it can rely on and a few hundred megabytes of memory. Users give a progress bar a few seconds. It is a hard place to run a neural network, and we chose it on purpose.

A service that anonymizes an uploaded document doesn't remove the exposure; it just hands the document to a vendor. So our detection runs in your browser, including the model that sees Margarida Rodrigues as a person and 1249-300 Lisboa as an address. The first version did not fit.

Vocabulary nobody used

Our detector is a multilingual BERT model fine-tuned to find names, organisations and places. Its vocabulary holds 119,547 tokens across 104 languages. At the time we offered seven: English, French, German, Spanish, Italian, Dutch and Portuguese, all in Latin script.

The vocabulary table is about 92 MB of the 178.5 MB file. It holds Devanagari, Cyrillic, Hangul, Thai, Armenian, Hebrew, Arabic, kana and thirteen thousand Chinese characters. Every user downloaded them, and none used them.

We had already built a tool to remove them and shown it was safe. We left it as an optional command, and nobody ran it. An opt-in build step tends not to run.

Removing rows instead of retraining

The usual answer is distillation, which trains a small model to imitate a large one. It trades accuracy for size. A tool built to find personal data should not make that trade quietly.

What we did is closer to deleting unused rows from a spreadsheet. The twelve layers that do the actual language processing are copied unchanged. Each remaining row of the vocabulary table keeps its exact value. The model splits words into the longest pieces it knows, so Latin text still splits the same way.

We ran both models over 77 real documents in all seven languages. There were zero differences in how text was split and zero differences across 2,219 detected entities. The largest difference in raw output was 0.0000000000. The output is identical, not merely close.

It cost us one bad week. Token IDs appear in three places in the tokenizer file, and we updated two. Every test passed, yet the model gave different answers on sixty documents. Since then we compare raw output, never entity lists, which can agree by chance.

Two attempts to go further

Vocabulary is still a third of the file, much of it Latin script for languages we do not offer. Another 30 to 40 MB looked available.

Keeping 58,000 of the 68,875 Latin tokens saved 8 MB and lost 13% of all detections. A control test explained why. Keeping only the 22,800 tokens our test documents use gave identical splitting and 100% agreement. So unused tokens cost nothing to remove, but a missing token breaks words apart, and proper nouns suffer most.

The Latin letters for markets we do not sell into, such as Turkish ğ, Romanian ț, Czech ř and Nordic ø, come to 4,200 tokens and 3.2 MB. They spell the foreign names in European contracts. We kept them, because detection should not get worse on a Turkish name.

The second attempt was DistilBERT: six layers instead of twelve, about 70 MB to download versus our 100. On 137 hand-checked entities, detection fell from 77.2% to 73.5%, with German down 13 points, English 9 and Spanish 22. The models agreed on only 76% of findings. The smaller one found a different set of entities of similar size, which is harder to check than a smaller set.

The model we ship

The model is called Minuscule, after the Latin script standardised across Charlemagne's empire. It has 68,875 tokens and takes 139.6 MB on disk. No encoder weight differs from the original.

It carries a build record with the original model, settings, vocabulary sizes and the checksum of its weights. It is published on Hugging Face, so you can check these numbers. Why we built our own detection engine explains why the model is ours to prune.

Why a hundred megabytes matters

On any given day, a paralegal is pasting a client's contract into a chat window. A nurse is uploading a discharge summary. They are behind, and the approved tool needs a ticket, a security review and two weeks.

People use whatever protection is already at hand, with nothing to install and no admin to ask. That means a browser tab. It costs a hundred megabytes once, and the client's name stays on that laptop.

Common questions

Does pruning the vocabulary cost accuracy?

Not when your languages never use the removed vocabulary. No encoder weight changes and every remaining row is copied exactly. Cutting vocabulary the model can reach degrades it sharply.

Why not just use a smaller model?

We tried DistilBERT. It was 30 MB smaller but scored 73.5% compared with 77.2%, with a 22-point drop in Spanish. Our benchmark against Presidio explains how we measure.

Can a model this size run in a browser tab?

Yes. It downloads once and is cached, then compiles at each launch, which is why the first run shows a loading bar. After that, no page of your document is uploaded.

What happens to a language you removed?

Latin-script languages still work where their letters were kept, including Turkish, Vietnamese, Romanian, Czech, Hungarian and Nordic names. A removed script, such as Japanese, needs its own model pack.

Found this useful? Let us know.

Share