Edge RAG on 4 GB Hardware: What We Measured on Arduino UNO Q

Most teams building on-device RAG start by shrinking the LLM. That helps-but on a 4 GB board the retrieval layer can still eat the budget if you copy a cloud vector stack (float32 + HNSW) onto the device.

This post does two things. First, it introduces Moorcheh and the MIB / EDM technology behind our edge retrieval stack. Second, it walks through what we measured on Arduino UNO Q (~3.6 GB usable RAM): where time goes in a full RAG pipeline, how fast server-side search stays from 10k to 100k vectors, and what ~33 million vectors means as honest RAM math versus float32.

A retail kiosk demo-The Brew Corner-proves the full loop end-to-end. It is evidence that the stack closes, not the whole product story. To run Moorcheh Edge yourself, start with the documentation.

What is Moorcheh?

Moorcheh builds vector search for places where memory, privacy, and offline operation matter as much as recall quality. The conventional stack-HNSW graph + float32 vectors + cosine similarity-works well in the data center. On edge hardware it often does not. Float payloads are large. ANN graphs want to stay resident in RAM. And the same board still has to host an embedding model and a local LLM.

Our research paper From HNSW to Information-Theoretic Binarization argues for a different path: compress embeddings into compact one-bit codes at ingest, then score similarity with a metric designed for those codes. Across evaluated datasets, that information-theoretic approach can preserve semantic signal while cutting memory dramatically versus float32-on the order of ~32× for 768-d codes. That idea is the foundation of the Moorcheh stack.

Moorcheh Edge packages those ideas for one device: Docker image moorcheh/moorcheh-edge, Python CLI/SDK moorcheh-edge, local API at http://localhost:8080, data under ~/.moorcheh-edge/data, no API keys. Cloud and on-prem Moorcheh target larger multi-tenant deployments; Edge is what we measured on UNO Q.

It does not replace your embedding model or LLM. It makes retrieval small enough that embed + search + generate can coexist on a constrained board. Today (v0.2.3), text mode embeds with BAAI/bge-small-en-v1.5 (384-d); vector mode accepts precomputed floats (128–1536-d). Cap: 10,000 items per store-a product limit for interactive catalogs, not the theoretical RAM ceiling discussed later.

If you are an edge ML engineer, the mental model is simple: Moorcheh owns the store and the scan. You own (or choose) the embedder and the LLM. The rest of this post shows why that split matters on a 4 GB board.

Why edge RAG is hard

Cloud RAG assumes managed vector SaaS, plenty of RAM, and a network path to an API. A kiosk, factory terminal, or field device often needs grounded answers from this location's knowledge, with privacy and offline resilience.

You still need four layers: embeddings, retrieval, a local LLM, and often voice. Every layer competes for the same ~3.6 GB. Teams usually shrink the LLM first. That is necessary, but not sufficient. If retrieval still looks like a cloud float index, you spend megabytes-and sometimes seconds-on the wrong problem.

Moorcheh's bet: make retrieval tiny in RAM and fast enough to disappear from the user-facing latency budget so the other layers can fit.

MIB, EDM, and ITS

MIB (Maximum Information Binarization) runs at upload. Each float embedding becomes a packed one-bit-per-dimension code. Original floats are not kept on disk. Unlike naive random-projection LSH, MIB is designed to keep the most information-rich dimensions of the original vector in the binary form.

EDM (Efficient Distance Metric) runs at query time. The query is binarized the same way; EDM scores similarity with CPU-friendly bitwise operations instead of high-dimensional float cosine.

ITS (Information-Theoretic Scoring) replaces the separate reranker pass common in cloud RAG stacks. Instead of retrieving candidates with cosine and then sending them through a cross-encoder or reranking API, Moorcheh ranks results with built-in information-theoretic scores designed for MIB codes-no extra model, no second network hop.

On UNO Q at 10,000 × 768-d, that difference is concrete. Moorcheh's ~17 ms figure is not raw ANN recall alone-it is median end-to-end retrieval with ITS: vector scan, EDM similarity, and information-theoretic reranking in one pass. A conventional stack would add a separate reranker on top.

Retrieval is not free-but it is orders of magnitude smaller than the embedding daemon and LLM on the same board. That is the architectural point.

What we measured on UNO Q

All numbers below come from Arduino UNO Q (~3.6 GB usable RAM, ARM64 Linux). We timed three workloads separately so the bottlenecks stay clear:

1. Text search - embed the query, then retrieve from a text store.
2. Full RAG answer - embed + retrieve + local LLM generation.
3. Vector search only - precomputed query vectors against a large store, with embedding and the LLM turned off.

Text search: the bottleneck is embedding, not the store

With a small text catalog-SciQ chunks and Brew Corner-style store content-cold text search sat around ~6.5 seconds. Almost all of that time is query embedding on the UNO Q CPU, not scanning the store.

That table is the core proof. Going from a few dozen documents to ten thousand vectors does not explain multi-second latency. Generating the query embedding does. When we keep the embedding model loaded in RAM and reuse it, text search comes down to about ~1.08 seconds on Brew Corner and other catalogs. For interactive kiosks, warm embedding is infrastructure-not optional polish.

RAG answers: cold starts are the LLM, not retrieval

Once retrieval is cheap, end-to-end time is mostly embed + generation:

Cold starts are dominated by loading the LLM-not by Moorcheh scanning the store. Warm kiosk-style replies land in the teens of seconds; vector search contributes about ten milliseconds.

RAM while the full stack runs

Under load, the board's memory story matches the latency story:

The LLM and embedder are the RAM and latency story. Moorcheh stays in the tens of MiB.

Vector search only: 10k → 100k

To isolate server-side retrieval, we loaded precomputed 768-d vectors and timed search with no embedding model and no LLM in memory.

Upload of 100k vectors took on the order of eight minutes (~200 vectors/s over HTTP). With retrieval only, the host still had about 3 GB free-headroom you spend on embed + LLM when you turn full RAG back on.

Even at 100× more vectors than the 10k product default, median search remains sub-100 ms. Latency grows, but gently. The store stays small. The container stays small.

Bottom line: 100k vectors, 768-d, UNO Q → ~67 ms search with ITS, ~20 MiB store, ~40 MiB server. For a kiosk RAG path, add warm embed (~1 s class) and LLM time on top. Retrieval with ITS is not the bottleneck.

~33 million vectors: float32 vs Moorcheh

We sometimes say ~33 million MIB-compressed vectors can fit in the payload budget of a 4 GB class device. That is calculated capacity from compact storage-not a benchmark we ran end-to-end, and not the shipped per-store cap.

On Arduino UNO Q (~3.6 GB usable):

• float32 at 33M is impossible-about 100 GB of payload alone, before any index or process overhead.
• Moorcheh MIB at 33M can fit as retrieval-only payload (~3.0 GiB), leaving on the order of ~600 MB for Linux, the Moorcheh process, IDs, and safety margin.

That is why "~33 million vectors on 4 GB" is possible with Moorcheh: not because the board grew, but because MIB shrinks the dominant term from kilobytes to 96 bytes.

The same compression shows up at scales we did measure:

At 384-d (BGE-small text mode), float32 for 33M is still about 51 GB; MIB is about 1.6 GB (48 bytes per vector). Same story, more headroom.

What 33M does not mean: today's product cap is still 10,000 items per store; linear scan at 33M would be far slower than ~67 ms at 100k; full RAG at that size still needs embed + LLM RAM. Use 33M to explain why MIB matters. Use 10k / 100k to explain what we ship and measure.

From board to kiosk

Measured deployments use a split stack, not a monolith on the PC. A display machine runs the customer/admin UI and catalog metadata. The Arduino UNO Q runs Moorcheh Edge, Ollama, and voice. Embeddings stay on the board so search and RAG stay local; the Rust container stays small and reaches Ollama on the host for generation.

Figure: Retail-kiosk reference architecture. The Display PC hosts the browser (:5173) and retail-kiosk-api (:8765). The Arduino UNO Q runs Moorcheh Edge Docker (:8080), voice serve (:8766), and Ollama (:11434), with USB mic and speaker for the voice path. Catalog content is loaded onto the board over Wi-Fi LAN.

Text path: browser → kiosk API → Moorcheh Edge /answer/stream (SSE) → Ollama.
Voice path: mic → voice serve (STT) → Moorcheh Edge → Ollama → TTS → speaker; the UI can also proxy voice asks through the same board.

The Brew Corner-a fictional café and mini-market catalog-closes the loop: chunk and embed on the board, answer customer questions with grounded context, and sync chunk text back to the display PC without shipping vectors. The warm text-search path we measured at ~1.08 s is exactly the path a customer question takes before the LLM speaks.

The same retrieval layer can back warehouse pick lists, clinic FAQs, factory work instructions, or museum guides. Swap the catalog; keep the engine. It proves you can run grounded answers offline on hardware you can hold. It does not define Moorcheh Edge's scope. Retrieval engineering does.

Lessons for edge ML practitioners

Profile embedding first. If text search takes multiple seconds with a tiny catalog, buying a faster ANN index will not fix it. The unfair comparison-~1.08 s warm text search versus ~17 ms retrieval with ITS at 10k-makes the bottleneck obvious.

Keep the embedder warm. Loading the model into RAM is how we move from multi-second searches toward ~1.08 s. For voice kiosks, treat that as part of startup, not an afterthought. Reloading the model on every question reintroduces the multi-second tax.

Separate retrieval benchmarks from RAG benchmarks. Milliseconds of server search and seconds of embed + LLM are different stories. If you only publish end-to-end RAG latency, readers will blame the vector engine for work it did not do. Publish the split: cold text search, warm text search, raw vector search, and warm answer.

Stage voice and LLM if RAM is tight. Concurrent speech, warm LLM, and warm embed can push toward ~2.3 GB used on UNO Q. Production designs often unload the LLM between turns or run STT/TTS in a separate phase. Moorcheh's ~20 MiB container is not what forces that tradeoff-the embedder and LLM are.

Be honest about limits. Ten thousand items per store, linear scan, small local LLMs, and 33M as RAM math-not a drop-in for Pinecone, Qdrant, or Elasticsearch. Moorcheh Edge is retrieval for one device when privacy, offline operation, and RAM matter more than billion-scale hybrid search. That honesty is part of the product: edge RAG fails when teams pretend a 4 GB board is a data center.

What's next

We continue to shrink the dominant costs: smaller default embedders (BGE-small 384-d), Qwen2.5 0.5B as the default LLM for better quality per megabyte, and voice preflight so the UI only "thinks" when retrieval actually found context.

Try Moorcheh Edge

pip install moorcheh-edge
moorcheh-edge up --with-llm -y
moorcheh-edge status

• Introduction
• Quickstart
• Case study PDF
• Research: From HNSW to Information-Theoretic Binarization

Moorcheh Edge is for developers building on-device search and RAG. Start with the docs.