realtime-chat

Español · English

A Discord-style real-time chat (servers, public and private channels, invite links, and later presence, mentions, attachments and search) built around two guarantees that are tested on purpose, not just claimed:

1. Delivery: every confirmed message reaches every channel member exactly once and in the same order for everyone, even across reconnects, client retries and several backend instances.
2. Authorization: nobody reads or writes a channel they don't belong to, over HTTP or WebSocket, not even one second after being kicked.

Status: milestone 7 of 9. Authorization over HTTP and WebSocket, message ordering, real-time delivery across two backend instances, presence, a per-user message rate limit, mentions with notifications that survive the worker dying and file attachments through presigned S3 URLs are built and verified, including a delivery harness and a mentions harness with fixed seeds. Search and unread counts are not built yet. Nothing below claims a result that does not exist.

Status

Milestone	What	State
1	Auth (bcrypt + JWT), users, servers, Flyway	done
2	Channels, invite links, roles, AccessPolicy, HTTP authorization matrix	done
3	Messages: per-channel seq, idempotent send, cursor pagination	done
4	Real time: STOMP relay to RabbitMQ, two instances, resync by seq, delivery harness	done
5	Presence across tabs and instances, "typing…", atomic token-bucket rate limit	done
6	Mentions: outbox, worker, retries, dead-letter queue	done
7	Attachments (presigned S3 URLs)	done
8	Search and unread counts	planned
9	Deployment, end-to-end tests, measurements	planned

What is already proven (milestones 2 to 7)

• A single AccessPolicy decides every access. Controllers never check permissions on their own. It reads roles from the database on every request, never from the JWT, so a kicked user is out on the very next request even though their token is still valid.

• 404 when you can't see the resource, 403 when you can see it but lack the role. A non-member gets 404 on a private channel, so the response doesn't confirm it exists.

• An authorization matrix as tests: every endpoint × 6 kinds of user (owner, admin, member, member of another server, unauthenticated, kicked a second ago), one test per cell, 228 in total, against a real PostgreSQL. A test fails if an endpoint exists that is not in the matrix, and another fails if docs/AUTHZ.md drifts from it. I broke a permission on purpose and added an unlisted endpoint to check the matrix notices; it did.

• Invite usage limits survive concurrency. Accepting an invite spends a use with one atomic UPDATE … WHERE uses < max_uses. With max_uses = 1, ten simultaneous accepts let exactly one in (also tested with a limit of 3).

• Per-channel seq with no repeats and no gaps. 50 simultaneous sends to one channel get seq 1 to 50. Swapping the atomic UPDATE … RETURNING for a naive SELECT max(seq) + 1 makes the test fail with duplicate seq values.

• Retries never duplicate and never leave a gap. Ten simultaneous retries of one message (same client_msg_id) store it once and leave last_seq = 1: the retries that lose the race roll back the seq they reserved (ADR 0005). Without that rollback the test ends at last_seq = 10.

• Cursor pagination by seq: after_seq to catch up, before_seq for history. The channel page uses both, and resyncs on its own when a sent message reveals a gap.

• Live delivery across two instances. A message saved on instance A reaches a client connected to instance B, in the same order, through a RabbitMQ STOMP relay. Notifications go out only after the commit: a rolled-back send publishes nothing, a replayed send publishes nothing, and ten simultaneous retries publish exactly one frame.

• Nobody can write through the WebSocket. Messages are only written over HTTP; every STOMP SEND is rejected, because with the relay a SEND to /topic/channel.N would reach subscribers without going through the seq or AccessPolicy (ADR 0006). SUBSCRIBE goes through AccessPolicy and reads the database on every subscription, and a second matrix covers it (12 rows × 6 user types, 72 cells, in docs/AUTHZ.md).

• Kicking someone closes their sockets, on every instance. The instance that handles the kick closes its own sockets and announces it on a RabbitMQ fanout exchange so the other instance does the same (ADR 0007). With a still-valid token the kicked user can reconnect but gets 404 on every subscription.

• A delivery harness (harness/). 8 clients send 640 messages at once to one channel through a load balancer in front of two instances, disconnect and reconnect to either instance, resend confirmed messages, and one instance is restarted mid-run. Six seeded runs ended with zero violations of the four invariants (complete, no duplicates, same order, faithful); see docs/DELIVERY.md for the numbers and what they do not prove. Its first run on seed 2 failed and found a real client bug (the resync cursor must be the end of the gap-free run, not the highest seq); that is fixed and written up there.

• Presence that survives tabs, instances and crashes. A user is online while at least one connection is alive on any instance: one Redis sorted set per user with an expiry per connection, renewed by a heartbeat, so a dead tab or a dead instance switches itself off (ADR 0008). With two tabs, closing one keeps the user online and closing both turns them off. Checked with real sockets, including across instances and on a kick.

• A rate limit that holds under concurrency. A token bucket in one Lua script, which Redis runs atomically: 100 simultaneous requests against a bucket of 20 let exactly 20 through. The naive read-then-write version lets all 100 through (I swapped it in to check the test notices; it did). It is charged only to a new message, after AccessPolicy (a stranger still gets 404) and after replay detection (resending a saved message is a 200 and costs nothing), and before the seq is reserved, so a 429 leaves no gap (ADR 0009). If Redis is down it lets messages through. The delivery harness waits out the 429s and still ends with zero violations (434 and 446 throttled sends in two runs).

• "Typing…" is ephemeral and still goes in over HTTP, so STOMP SEND stays closed (ADR 0010). It has its own topic and no seq, and it is outside the delivery guarantee.

• A mention is notified once even if the worker dies. The event is written to an outbox table in the same transaction as the message; a publisher carries it to RabbitMQ and counts it as published only if the broker confirmed and routed it; the worker deduplicates with UNIQUE (message_id, user_id) in a single statement, retries with waits of 2 s, 10 s and 1 min, and sends what cannot be fixed to a dead-letter queue (ADR 0011, 0012, 0013). The §8.3 test kills the worker before it processes an event and after it saved the notification, cuts its connection, stops and restarts it, and kills the publisher right after it published a batch: with three seeds and about 400 mentions each, every (message, person) ended with exactly one notification and none left over. "Exactly once" is about the effect, not the delivery.

• The deliberate breakages were measured, and one corrected me. The naive deduplication ("does it exist?" and then "create it") passed the big test on all three seeds: after a crash the event comes back after the first attempt finished, and even that version catches it. What breaks it is two workers processing the same event at the same time, so I added a concurrency test, and against the naive version it creates 72 notifications instead of 20. A publisher that marks the batch without looking at the confirmations (3 of 8 tests) and events written after the message commit also failed, as they should.

• Only people who can read the channel are notified, and the list is filtered when read. A mention of someone who cannot see a private channel tells them nothing (it would reveal the message exists). Someone who is kicked, leaves a private channel or stops being an admin stops seeing those mentions without any row being deleted, and another person's notification answers 404, the same as a missing one (three new matrix rows).

• A file never passes through the backend, and its size is enforced twice. The client asks for a presigned PUT with the size and type inside the signature, uploads straight to S3, and confirms; the backend then does a HEAD and, if the object is not the size that was declared, deletes it and answers 400. The signature is the first barrier, not the only one: the test uploads the wrong size on purpose and the HEAD catches it (ADR 0014).

• An attachment belongs to one message. A conditional UPDATE … WHERE message_id IS NULL inside the send transaction decides it: ten different messages sent at once with the same attachment let exactly one in, and the nine that lose roll back their seq too (last_seq stays at 1). Without the condition all ten win.

• Downloading is decided with today's access, not yesterday's. A download is a 60-second URL requested over HTTP and checked by AccessPolicy at that moment, so a kicked user gets 404 even for a file they already knew about. What goes over the WebSocket is only the description of the file (name, size, type), never a URL. Only four image types are shown inside the page; anything else, including HTML and SVG, is forced to download.

• Honest limit: the local S3 emulator (Adobe S3Mock) does not validate URL signatures, so no test shows real S3 refusing a body of the wrong size; that is checked at milestone 9.

678 backend tests (unit, @WebMvcTest, Testcontainers integration with real PostgreSQL, RabbitMQ and Redis) and 66 harness tests, Spotless-formatted.

The design in 30 seconds

The chat is the easy part. The two hard problems, and the planned answer to each (details in DESIGN.md, decisions in docs/adr/):

• Ordering and loss: each message gets a seq that grows by one inside its channel, assigned by the database in the same transaction that stores the message. Clients remember the last seq they saw, so reconnecting is GET …/messages?after_seq=N, a gap (N+2 without N+1) is detected and fetched, and duplicates are dropped. A client-generated client_msg_id with UNIQUE (author_id, client_msg_id) makes retries idempotent.
• Several instances: STOMP over a RabbitMQ broker relay instead of Spring's in-memory broker, so a subscriber on instance B receives what was sent through A. Notifications are published only after the transaction commits, and mention events go through an outbox table so they can't be lost.
• Authorization over WebSocket: the same AccessPolicy runs on every SUBSCRIBE, every SEND is rejected (messages are written over HTTP only), and kicking someone closes their open sessions on every instance.

flowchart LR
  FE[Next.js] -- HTTP --> BE[Spring Boot<br/>instances A and B]
  FE -- WebSocket STOMP --> BE
  BE --> PG[(PostgreSQL<br/>source of truth)]
  BE --> RD[(Redis<br/>presence, rate limit)]
  BE --> MQ[[RabbitMQ<br/>STOMP relay + queues]]
  PG -. outbox .-> MQ --> W[Notification worker]
  FE -. presigned URLs .-> S3[(S3)]

Loading

PostgreSQL, Redis, the REST API, the frontend, the RabbitMQ STOMP relay, the two-instance cluster, the notification worker (a separate cluster service) and attachments on S3 (an emulator in development) exist today.

Documents

File	What it is
DESIGN.md	The design, written before the code: problems, decisions, verification plan, milestones
docs/AUTHZ.md	The authorization rules and the full HTTP and STOMP matrices (checked by tests)
docs/adr/	Fourteen decision records: per-channel seq, external broker, publish after commit, outbox, idempotent send without gaps, HTTP-only writes, closing sockets on kick, presence, message rate limit, ephemeral typing, mentions and notifications, the outbox publisher, the worker (deduplication, retries, dead letters), attachments (signed size, HEAD, one message per attachment)
docs/DELIVERY.md	The delivery guarantee, the harness, its results and its limits
docs/v2.md	Ideas left out on purpose (bans, invite revocation…)

UI text, API error messages and the design documents are in Spanish (the project started as a Spanish-language course project); code identifiers and this README are in English.

How to run it

Requirements: Docker, Java 21 and Node 20+.

cp .env.example .env
docker compose up -d                     # PostgreSQL 17, Redis 7, RabbitMQ 4 (STOMP enabled), S3 emulator

cd backend
./mvnw spotless:check verify             # all tests; needs Docker running (Testcontainers)
set -a && source ../.env && set +a       # the app reads these as environment variables
./mvnw spring-boot:run                   # API on :8080

cd ../frontend
npm install && npm run dev               # app on :3000, /api is proxied to the backend

To run two instances behind a load balancer plus the notification worker (the harnesses need them) and run the delivery harness and the mentions harness:

docker compose --profile cluster up -d --build   # backend-a :8081, backend-b :8082, balancer :8088, worker
backend/mvnw -f harness/pom.xml compile exec:java -Dexec.args="--seed 42"
backend/mvnw -f harness/pom.xml compile exec:java \
  -Dharness.main=com.jsav.chat.harness.MentionsHarness -Dexec.args="--seed 42"   # SIGKILLs the worker

To use the cluster from the browser, start the frontend with API_URL=http://localhost:8088 NEXT_PUBLIC_WS_URL=ws://localhost:8088/ws npm run dev.

To regenerate the matrices in docs/AUTHZ.md after changing them: ./mvnw test -Dtest='AuthorizationMatrixTest,StompAuthorizationMatrixTest' -Dauthz.update=true.

The defaults in application.properties (dev-password, a development JWT secret) are for local use only; override them with environment variables (JWT_SECRET must be at least 32 characters) in any real deployment.

Layout

backend/    Spring Boot 4.1 (Java 21): auth, server, channel, invite, message, realtime, access,
            presence, ratelimit, typing, mention, outbox, notification (worker)
frontend/   Next.js 16: login, register, servers, channels (live), mentions, invite page
harness/    delivery harness (N STOMP clients, four invariants) and mentions harness (SIGKILL the worker)
nginx/      load balancer config for the two-instance cluster
docs/       authorization matrices, delivery results, ADRs, ideas for v2

Out of scope

Voice and video, custom roles, bots and integrations, end-to-end encryption, federation, a native mobile app, and competing with Discord. See DESIGN.md §2: nothing outside that list enters before milestone 9.

License

MIT