Where the model runs — local, hosted, or a GPU box

A RAG system has two model calls: embed the text, and generate the answer. The trick to keeping deployment sane is to treat those two very differently. Embeddings are small, cheap and private, so they run locally, always. Generation is the expensive, swappable part — so it talks to whatever's cheapest or fastest in a given environment, through one interface.

The models

• EmbeddingGemma 300M — embeds both the query and the chunks, at 768 dimensions. It's small enough to run on a laptop CPU, so it never leaves the machine. Patient text doesn't get shipped to an API just to be turned into a vector.
• The generator — Gemma-4 E4B locally, or the larger Gemma-4 26B on AI Studio. (MedGemma 1.5 4B was the original and still names the project; nothing current runs on it.) None of these are hard-wired: the code just points at an OpenAI-compatible endpoint and asks for a model by name.

One endpoint, any provider

The generator is configured with two environment variables — a base URL and a model id. That's the whole abstraction. Point them at a local vLLM server and it self-hosts; point them at Google AI Studio's OpenAI-compatible API and it's fully managed; point them at anything else that speaks the same protocol and it just works. Embeddings stay local regardless. So moving from a hosted model to a self-hosted GPU is a config change, not a code change — which is exactly what you want when "where does this run" has a different answer on a laptop, in an eval loop, and wherever it might eventually be deployed.

Three places I actually ran it

Local (a Mac). Embeddings on the CPU, generation pointed at a hosted endpoint. No GPU, no server, cheapest possible dev loop. This is where most of the day-to-day work happened.

Hosted (AI Studio). Generation via Gemma-4 26B on AI Studio's API — zero infrastructure. The catch is honest: it's a reasoning model, so it's slow and emits long internal "thinking" before the answer, and the free tier throttles you on tokens-per-minute. Great for correctness, awkward for a fast eval loop.

A GPU box (EC2 g6, an L4, with vLLM). Self-hosting Gemma-4 E4B for full control and no rate limit. Two things bit me here. E4B disables FlashAttention, so vLLM falls back to a slower attention path — fine for a batch job, sluggish for interactive use. And spot capacity in my volume's availability zone was simply gone, so a one-off run meant paying on-demand. When I did rebuild the retrieval index on it, the move that mattered was ordering: stop the LLM server, use the freed GPU to pre-compute every embedding at once, then restart the server so the summariser model had the GPU to itself.

What I'd deploy

Embeddings local, always — it's cheaper and it keeps the sensitive text on your own hardware. Generation behind the provider-agnostic endpoint, chosen per environment: a hosted model for the dev and eval loop, and a warm GPU only if production latency actually demands it. What I would not do is stand up a GPU for a bursty, occasional workload — the spin-up time and the spot-capacity lottery cost more than they save.

The pieces this runs on — the pipeline, the retrievers, and the parameters — are the rest of the series. Building deployable, privacy-aware RAG that isn't chained to one vendor is what I do at twentytwotensors. Get in touch.