Treat the LLM like a flaky dependency

An LLM in a backend is a slow, expensive, occasionally wrong external dependency. The engineering that makes it safe is the same engineering you would apply to any flaky dependency: isolate it, make its failures cheap, and keep it away from decisions it cannot be trusted with.

Ports first, vendors second

The assistant I built has four ports: a vector store, a chat model, an embedding model and conversation memory. Routes and services may not import a concrete provider. Only adapters can.

That rule paid for itself during the project. The session store moved hosts, the vector store moved vendors, and inference moved to a different provider. Each move changed only its adapter. An abstraction you have never swapped is a guess; one you have swapped three times is a fact.

Grounding is a system decision

The prompt tells the model how many verified facts it has, that it may only use those, and the exact sentence to return when retrieval comes back empty. A model that guesses demos better and is unusable in front of someone who will act on the answer. A soft “I'm not sure” still gets acted on by a user who skims; a fixed refusal is unambiguous.

Don't pay for calls you don't need

Follow-up questions like “does it work on mobile?” need rewriting into a standalone query before retrieval, and the rewrite is another model call. Most turns do not need one, so a plain heuristic decides first:

• No conversation history: skip.
• The question is longer than ten words: probably self-contained, skip.
• No referring word such as “it”, “this” or “that”: skip.

Deciding costs nothing and makes no network call. When a rewrite does happen, retrieval runs with both the rewritten and the original question and de-duplicates the results, so a bad rewrite cannot hide a fact the original wording would have found.

Decide which failures are fatal

The service fails fast at startup if it cannot reach the vector store or the embedding service, because it cannot answer anything without them. Redis is deliberately not a boot blocker: if memory is unavailable the assistant answers without history. Losing context on a follow-up is a worse answer; losing the whole service is no answer.

Session history lives on the server, keyed by a session id, with a TTL and a short sliding window. The client never sends its own transcript, which keeps replicas stateless and keeps a client from rewriting what the assistant “said” earlier.

Test against the ports

The unit tests cover the rewrite heuristics, memory fallback on timeouts and provider errors, and merged retrieval. They are written against the abstract interfaces, so the same tests run whichever provider is wired in, which is exactly what you want the week a vendor changes.