A grounded AI assistant with swappable providers

The problem

Players ask about games and the platform: how to start, what a control does, how progression works. An assistant that answers confidently and wrongly is worse than none, because players act on what it says. Some knowledge is also game-specific or only for signed-in players.

Design

Ports and adapters. The vector store, chat model, embedding model and conversation memory are abstract interfaces. Routes and services may not import a concrete provider. That rule was tested for real: during the project the session store moved hosts, the vector store moved vendors and inference moved to a different provider, and each move changed only its adapter.

1. 1

   Request

   POST /v1/chat with the user's question, an optional session id and an optional game scope.

2. 2

   Memory

   Recent turns load from Redis. Sessions are server-owned; the client never sends its own history.

3. 3

   Route

   A heuristic decides whether the question needs rewriting into a standalone query. Most turns skip it.

4. 4

   Retrieve

   Embed and run vector search, optionally scoped to one game. When a rewrite happened, search with both the rewritten and the raw question and de-duplicate the facts.

5. 5

   Answer

   Build a grounded prompt and stream tokens back over SSE.

6. 6

   Persist

   Save the finished turn with a TTL and a sliding window of recent turns.

Decisions

• Grounding is a system decision, not a prompt trick. The model is told how many verified facts it has, that it may use only those, and the exact sentence to return when retrieval comes back empty. A fixed refusal is consistent and impossible to misread.
• Rewrites cost a model call, so most turns don't pay for one. No history, a question over ten words, or no referring word such as “it” or “that” means the rewrite is skipped, with no network call to decide.
• Memory failures degrade, they don't fail. If Redis is down the service answers without history. Startup fails fast on the vector store and embedding service, the two dependencies it cannot answer without, and deliberately not on Redis.
• Stateless replicas. All conversation state lives in Redis, so the service scales horizontally.
• Optional internal bearer auth for deployments behind another service.

Testing

Unit tests cover the rewrite heuristics, memory fallback on timeouts and provider failures, merged retrieval across rewritten and raw queries, and configuration handling. They are written against the abstract interfaces, so they run the same whichever provider is wired in.

Scope: this service owns the inference path. The player-facing UI and the offline pipeline that loads facts into the vector store are separate.

How I'd build it in Spring Boot

• Ports: plain Java interfaces (VectorStore, ChatModel, EmbeddingModel, ConversationMemory), one @Component adapter per vendor, selected with @ConditionalOnProperty. Spring AI ships the same three abstractions, which makes a vendor swap a dependency and configuration change.
• Streaming: return an SseEmitter from a Spring MVC controller, or a Flux<ServerSentEvent<String>> with WebFlux, and write the finished turn to memory when the stream completes.
• Session memory: Spring Data Redis lists, with rightPush then trim for the sliding window and expire for the TTL. A RedisConnectionFailureException is caught and the request continues without history.
• Fail fast, but not on Redis: an ApplicationRunner that checks the vector store and embedding service at startup, plus a custom HealthIndicator for each.
• The rewrite router is a pure function, so it is a plain JUnit 5 parameterised test; services are tested against Mockito mocks of the port interfaces.
• Request validation with Jakarta Bean Validation on request records (@Valid, @NotBlank, @Size).