Lawrence Lumbol Tityem

Problem

Concurrent balance mutations on the same wallet, duplicate transaction retries, and asynchronous fraud scoring all had to be handled correctly — without blocking legitimate users when a downstream service (fraud, notifications) is slow or down.

Key decision

Fraud scoring fails open, not closed: if the ML fraud service is unreachable after three retries, the transfer proceeds and the event is logged — rather than blocking every legitimate transfer whenever one internal service has a bad moment. Pessimistic row-level locks are acquired in deterministic wallet-ID order specifically to make deadlocks structurally impossible, not just unlikely.

Test results

49 passed, 6 skipped across 5 modules (auth, wallet, transfer, fraud, rate limiting) — 119.63s run.

Problem

Coordinate state across five independently owned services with no shared database and no synchronous calls between them — specifically solving the dual-write problem (writing to your own database and announcing that write to Kafka, two systems with no shared transaction) and idempotent consumption under at-least-once delivery.

Key decision

payment-worker reacts to InventoryReserved, never OrderCreated directly — charging a customer before stock is confirmed available is a business error this topology makes structurally impossible, not just discouraged by convention. The dual-write problem itself is mitigated by writing to Postgres first, then publishing to Kafka as a separate step with its own retry and dead-letter path — a full Transactional Outbox pattern is scoped but intentionally deferred, and documented as such.

Honest limitations

Notification-worker does not yet subscribe to rejected orders; Alembic migrations are not yet in place; only inventory-worker's automated test suite is complete so far. Documented rather than hidden.

Problem

Run background jobs (retryable, priority-ordered, safe under multiple concurrent workers) without introducing a second infrastructure dependency — and without the dual-write problem a Redis- or Kafka-backed queue has between "the broker" and "the database."

Key decision

Postgres is the queue itself: workers claim jobs via SELECT ... FOR UPDATE SKIP LOCKED, ordered by priority then FIFO within a tier, so priority never causes starvation. Crash recovery needs no separate heartbeat service — a claim's locked_at plus a visibility_timeout_seconds defines its own expiry, so a crashed worker's job becomes reclaimable through the exact same claim query. Failed jobs get exponential backoff up to a cap, then land in a dead-lettered state that's inspectable and manually replayable — never silently dropped.

Observability

Prometheus metrics wired across both the API and worker processes (submission/completion/retry/dead-letter counters, claim-latency histogram, queue-depth gauge), scraped as two separate targets and queryable live.

Honest limitations

No Alembic migrations yet — tables are created via create_all on API startup, with a documented (and narrow) cold-start race against the worker's first poll. Only one handler exists so far (a simulated email send), since the point of the project is the queue mechanics, not a real notification pipeline.

Problem

Four backend services (user, order, payment, wallet) each need authentication, rate limiting, retries, and error handling — duplicating that logic in every service is both wasteful and a consistency risk. One gateway in front of all four, with a single generic proxy route rather than a hand-written endpoint per service.

Key decision

POST requests are never retried, regardless of a service's configured retry count — if a POST to payment-service times out, the charge may have actually succeeded and the response was simply lost; retrying blindly risks a duplicate charge. The gateway also fails closed: any route not explicitly listed as public requires a valid JWT by default, so forgetting to register a new public route causes a safe 401, not an accidentally open endpoint.

Scope limit, stated deliberately

Circuit breaking is in-memory and per-process, not Redis-backed like rate limiting is — a real multi-instance gateway would need that state shared too. Kept in-memory here so the breaker's state machine itself stayed the focus, rather than distributed-state plumbing.