Exactly-once rewards across three services

The problem

Money moves when a video finishes playing, which is a client event. Clients retry, networks drop, people open five tabs, and a modified client can simply lie. Every one of those cases still has to credit a watch at most once, never spend more than a brand budgeted, and never let a user earn past the daily cap.

Where the seams go

Three services, with responsibilities split so that each rule has exactly one owner:

• The wallet is not a referee. It credits and debits users within hard bounds, keeps the append-only ledger, and cannot name an ad.
• The ads service owns every policy decision: who earns, how much and how often. It also owns campaigns, brand budgets and media.
• One money-moving call joins them: synchronous, idempotent and bounded. No client-held watch token, no reservation limbo, no status polling.
• Payments sit downstream and turn wallet balances into payouts through a gateway.

The earn flow

1. 1

   Feed

   The client loads ads. Each item says whether it can earn and, if not, why. Unearnable ads still play; only earning is suppressed.

2. 2

   Start

   When an earnable ad starts, the client calls watch/start. The ads service runs its policy checks and records a view with the server's start time.

3. 3

   Complete

   At the threshold the client calls complete. The ads service checks elapsed time on its own clock, atomically claims the view (started → crediting) and debits the brand budget with a conditional update that can never go negative.

4. 4

   Credit

   Ads calls the wallet with a scoped service token and a deterministic idempotency key. The wallet checks the per-credit ceiling, reserves the daily cap, then writes the ledger row and the wallet balance in one Postgres transaction.

5. 5

   Notify

   After commit the wallet publishes the new balance, and the client gets it over Server-Sent Events while the next ad plays.

Invariants

Every guarantee is enforced in exactly one place, so there is one place to read when it breaks:

Failure paths, all boring on purpose

• Daily cap reached: the ads service refunds the budget and marks the view rejected; the client stops starting earnable ads for the day.
• Wallet unreachable: the view stays in crediting. A sweep every 60 seconds retries the identical call with the same key, so it can never pay twice.
• Watched too little: 422 with watched and required seconds. The row stays open and the user can retry after actually watching.
• Budget exhausted at debit: the view is rejected and the ad disappears from earning feeds.
• Modified client ignores the earn flag: the policy check at watch/start and the unique key in the wallet stop it anyway. The flag is cosmetic.

Two bugs worth telling

A burst beat the daily cap. The cap check read a counter, credited, then incremented. Every request in a concurrent burst saw the same stale value and passed. Reproduced locally: a user at 45 of 50 firing 50 parallel credits earned 295 coins. The fix reserves the amount with an atomic INCRBY before crediting, so exactly one request can cross the line, and releases the reservation on every path where the credit does not land, including idempotent replays. A cold key is seeded from the ledger with SET NX so a late seeder cannot overwrite reservations already made.

Replays came back as 500s. The data-access layer wraps the database error, so the 23505 unique-violation code sits one level down the cause chain, not on the thrown object. Six hand-written checks read the code from the outer wrapper, never matched, and a replayed money call escaped as a 500 instead of its contracted “already credited”. The fix is one helper that walks the cause chain with a depth cap, used at all six sites. Six duplicate-fire tests run against real Postgres and Redis, and before trusting them I reran them with the helper limited to depth 1: all six failed, which is what proves they test the fix.

The reconciler's race is reproduced the same way: a second connection holds a competing insert on the same key in an open transaction, so the reconciler blocks on the unique index and receives the violation at commit.

Ads side

• Campaign lifecycle with a recorded status history, from draft through upload, transcoding and approval to active, paused or exhausted.
• Publishing is gated by device-bound step-up authorization: a grant earned on one device does not follow a token replayed from another.
• Media goes straight to object storage on a presigned URL; transcoding is queued, and playback URLs are signed for five minutes.
• Earnability for a whole feed page is resolved in one aggregation, and daily limits follow the IST calendar day so the date in the idempotency key and the reset rule always agree.
• Per-ad analytics are bucketed hourly or daily in IST.

Money out

• Payment gateways sit behind one adapter interface, so the rest of the service never knows which gateway it is talking to.
• Every ledger entry carries a unique idempotency key, and the debit, credit and commission legs of one movement derive theirs from a shared prefix, so a replayed movement collides on every leg.
• Payout requests move through pending → approved → processing → success | failed | reversed, and a failed payout returns its coins to the wallet.
• Scheduled payout generation derives its key from the period and the user, so re-running a batch cannot create a second request.

Trade-offs I chose

• The credit that crosses the daily cap is paid in full. One bounded overshoot is far simpler to reason about than partial credits.
• Redis is the fast path for caps and balances; the ledger is the source of truth and the fallback for both.
• Server-measured watch time can under-count on a slow network. I accepted that: a client-reported number cannot be allowed to decide a payment.

How I'd build it in Spring Boot

Every decision above maps directly onto the Spring stack:

• One money transaction: a @Transactional service method that inserts the LedgerEntry and updates the Wallet, with the idempotency key backed by a UNIQUE constraint in a Flyway migration.
• Row lock per user: a Spring Data JPA finder annotated @Lock(LockModeType.PESSIMISTIC_WRITE), which issues SELECT … FOR UPDATE on the wallet.
• Optimistic checks on paths that don't take the lock: @Version on the wallet entity, retrying a bounded number of times on ObjectOptimisticLockingFailureException.
• Recognising a replay: catching DataIntegrityViolationException is not enough, because a CHECK or foreign-key failure is one too. Walk the cause chain to the SQLException, require SQLState 23505 and the idempotency constraint's name, and only then answer “already credited”.
• Daily cap reservation: StringRedisTemplate.opsForValue().increment(key, coins) before the credit, decrement to release, and setIfAbsent(key, seed, ttl) to seed a cold key from the ledger.
• Reconciler and drift check: @Scheduled jobs kept to one instance across replicas with ShedLock or a Redis setIfAbsent lock whose TTL is shorter than the schedule.
• Balance push: an SseEmitter per connected user, fed by a Redis pub/sub MessageListener after commit (@TransactionalEventListener(phase = AFTER_COMMIT)).
• Scoped service tokens: Spring Security with one authority per allowed transaction type, so the rewards caller's token cannot touch payouts.
• Payment gateways: a PaymentGateway interface with one @Component per gateway, injected as a Map<String, PaymentGateway> and chosen by configuration.
• Payout state machine: an enum of allowed transitions, applied with a conditional UPDATE … WHERE status = :expected through @Modifying @Query, so two workers cannot move the same payout.
• Queued transcoding: a Kafka or RabbitMQ listener with retries and a dead-letter topic (@RetryableTopic).
• Tests: JUnit 5 with Testcontainers Postgres and Redis; duplicate fires launched from an ExecutorService behind a CountDownLatch so they really race.