Rendered at 04:08:50 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
soycaporal 5 hours ago [-]
Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD.
It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared words ("reset my password" ↔ "I forgot my password" → 0.88). Used for semantic search, FAQ/intent matching, and clustering. Running it on-device means search-as-you-type semantic search is performant with no API dependencies.
Curious if this is something useful, what are the use cases for on-device embeddings.
versteegen 30 minutes ago [-]
Nice, I'm really interested in using this for simple semantic search in a native desktop application.
Any comparisons with other tiny embedding models? Did you start from MiniLM-L6 because it's an especially good model in its class? It's hard to figure this out since all you provide is "Retrieval (SciFact NDCG@10)".
But the claimed performance seems way off, I get only 35 emb/sec in firefox on a i5-4570 rather than 400/sec. Is there an issue with falling back to a non-SIMD path? I'll try a native Rust binary next.
dwheeler 2 hours ago [-]
Thanks! I strongly suggest copy-pasting that explanation to your web page, that's a nice summary.
keynha 1 hours ago [-]
0.84 Spearman fidelity to the MiniLM teacher at ternary precision is a striking result. How much of that is the quantization-aware training doing the work, versus what a post-training ternary quant of the same encoder would give you?
soycaporal 46 minutes ago [-]
It's entirely the QAT. The whole distillation process is quantization-aware from the start, so the ternary weights are learned rather than fitted after the fact.
The only post-training quantization I applied was int4 on the embedding layer, and I ran a small ablation there to find the sweet spot between size and quality.
fellowniusmonk 4 hours ago [-]
Awesome! Besides size, how does this compare to gte-small?
soycaporal 4 hours ago [-]
gte-small outscores all-MiniLM-L6 on MTEB (~61 vs ~56 avg per the GTE paper). MiniLM is ternlight's teacher (ternlight holds 0.84 Spearman fidelity to teacher).
I haven't run a head-to-head yet; STS-B/MTEB numbers are on the roadmap. Also on the roadmap is to distill gte-small as teacher.
jbellis 10 minutes ago [-]
FWIW -- Granite r2 small is a 30M model, still small enough to run on CPU, and a good baseline for fine tunes.
dirteater_ 4 hours ago [-]
This is cool!
but also maybe you could put a button on the landing page to trigger the demo because it's a bit startling to hear my fans go crazy when opening a webpage.
Waterluvian 3 hours ago [-]
Agree. But this also reminds me fondly of the days where the sounds of my computer so intimately indicated what’s going on.
mwcz 3 hours ago [-]
Amiga floppy disk sounds are the deepest of sense memories.
soycaporal 2 hours ago [-]
CPU cycle maxxing, who said GPUs were special?
chris-hartwig 59 minutes ago [-]
Thank you for this! Local models will bring privacy at some point, and I already know an excellent use case for such a small embedding model (cheap and fast search in a product base). Relying on the CPU is also a plus in my case.
wazzup_im 4 hours ago [-]
I added an offline search engine to app.wazzup.im/search (no login or payment required).
First search downloads the model from the internet and subsequent runs are from the cache.
The model is very small so it's not the best for everything but it's good for basic math and coding.
Or sometimes "Service Worker API is available and in use." + "Loading search results...".
wazzup_im 2 hours ago [-]
This is a known issue and I am actively trying to find why this is happening. So far it's pretty good on Brave/Chrome.
Tested on Macbook Pro M1 8gb RAM and Macbook Air M1 8gb RAM. Mostly likely because of M series of chips. All tests were done on Brave/Chrome.
Does not work on iPhone 11 Pro Max and iPhone 16 Pro. Mostly likely because of A series of chips. Tests were done on Safari and Chrome and it crashes on both.
soycaporal 2 hours ago [-]
ohh thanks for the report.. probably has to do with wasm runtime.. Will note this as a known issue
wazzup_im 2 hours ago [-]
Np!
The workaround is to unregister/stop the service worker from the DevTools > Application tab > Service workers.
aetherspawn 4 hours ago [-]
Can the 30 second embedding time be done beforehand and sent to the browser?
Inference is nice and quick after that.
soycaporal 4 hours ago [-]
yes, you could run a 1 time indexing run on the server side, and just ship the embeddings to frontend
CobrastanJorji 4 hours ago [-]
Great, now my websites are gonna push entire LLMs onto my browser in order to use my CPU to make inferences about my shopping habits or whatever.
paytonjjones 3 hours ago [-]
Ha, I was literally thinking this but from the other side.
"Hmm, 7MB would barely make a dent in the size of the app and allow us to do some of our basic ML without calling the backend"
Interesting project. Happy to see someone who shares an interest in tiny vector embeddings models. I've worked on tiny (1MB - 4MB, 250K - 950K parameters) embeddings models called BERT Hash https://huggingface.co/blog/NeuML/bert-hash-embeddings
Keep up the great work!
gaigalas 2 hours ago [-]
That's really impressive, congratulations. It's nice to see novel applications of browser models.
soycaporal 2 hours ago [-]
thank you! hopefully it can unlock some novel applications, that would be cool
rvz 3 hours ago [-]
Why do these things download into the browser automatically? This could be used to distribute malware and also or hog excessive browser memory.
3 hours ago [-]
akoboldfrying 2 hours ago [-]
This doesn't add any malware risks beyond what a JavaScript-enabled browser already allows.
Re excessive browser memory use: Yes, it adds non-negligible weight, but again, you could already achieve excessive browser memory usage before this. For comparison, a true color 1080p image, uncompressed (which is needed for actual display on screen) is only slightly smaller at 6.22Mb.
gaigalas 2 hours ago [-]
That's... how the web works? You download things on demand.
There are JS files larger than 7MB in the wild. They run on JIT engines that displayed severe CVEs over the years. PDFs, video running directly on special hardware encoders. That's the web now.
A WASM model is not that offensive.
newspaper1 3 hours ago [-]
Very cool! I'd love to point it at my own corpus to index/embed. Would be cool if you could give it a link to a markdown file or even a website to crawl.
soycaporal 2 hours ago [-]
love the idea! Will think of a way to host it probably on huggingface
It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared words ("reset my password" ↔ "I forgot my password" → 0.88). Used for semantic search, FAQ/intent matching, and clustering. Running it on-device means search-as-you-type semantic search is performant with no API dependencies.
Demo (2k React docs, fully on-device): https://ternlight-demo.vercel.app
Two tiers on npm: - @ternlight/base (7 MB, ~5 ms/embed, more capable embedings) - @ternlight/mini (5 MB wire, ~2.5 ms/embed).
Bundled for Node and browsers.
Repo - see technical details (MIT, training pipeline included): https://github.com/soycaporal/ternlight
Curious if this is something useful, what are the use cases for on-device embeddings.
Any comparisons with other tiny embedding models? Did you start from MiniLM-L6 because it's an especially good model in its class? It's hard to figure this out since all you provide is "Retrieval (SciFact NDCG@10)".
But the claimed performance seems way off, I get only 35 emb/sec in firefox on a i5-4570 rather than 400/sec. Is there an issue with falling back to a non-SIMD path? I'll try a native Rust binary next.
The only post-training quantization I applied was int4 on the embedding layer, and I ran a small ablation there to find the sweet spot between size and quality.
but also maybe you could put a button on the landing page to trigger the demo because it's a bit startling to hear my fans go crazy when opening a webpage.
First search downloads the model from the internet and subsequent runs are from the cache.
The model is very small so it's not the best for everything but it's good for basic math and coding.
Give it a try.
Loading model... + Loading search results...
Or sometimes "Service Worker API is available and in use." + "Loading search results...".
Tested on Macbook Pro M1 8gb RAM and Macbook Air M1 8gb RAM. Mostly likely because of M series of chips. All tests were done on Brave/Chrome.
Does not work on iPhone 11 Pro Max and iPhone 16 Pro. Mostly likely because of A series of chips. Tests were done on Safari and Chrome and it crashes on both.
The workaround is to unregister/stop the service worker from the DevTools > Application tab > Service workers.
Inference is nice and quick after that.
"Hmm, 7MB would barely make a dent in the size of the app and allow us to do some of our basic ML without calling the backend"
Probably a lot more practical to use this though: https://developer.apple.com/apple-intelligence/
Keep up the great work!
Re excessive browser memory use: Yes, it adds non-negligible weight, but again, you could already achieve excessive browser memory usage before this. For comparison, a true color 1080p image, uncompressed (which is needed for actual display on screen) is only slightly smaller at 6.22Mb.
There are JS files larger than 7MB in the wild. They run on JIT engines that displayed severe CVEs over the years. PDFs, video running directly on special hardware encoders. That's the web now.
A WASM model is not that offensive.