Setup
Product wants “notifications”: users get told when something they follow changes. The requirement is vague on purpose:
- Email first. Push notifications “soon”. SMS “maybe”.
- Users should be able to turn notifications off per type.
- “It shouldn’t spam people.”
What you have:
- A team of 3 engineers and a 6-week deadline.
- A single web app (a monolith) on one Postgres database, plus a background job queue already used for password-reset emails.
- About 200,000 users and 2 million change events a day. Roughly 5% of events should notify someone.
- Most items have a handful of followers, but the 50 most popular items have 20,000–60,000 followers each.
Options the team has floated:
- A separate notification service with its own message broker and database.
- A notifications module inside the monolith: write events to an outbox table, and fan out with the existing job queue.
- A third-party notification platform, with the monolith calling its API.
Outcome
A team in a similar spot built option 1 first. They spent most of the six weeks on infrastructure (the broker, deployment, a new database, auth between services) and shipped email notifications two weeks late.
A second team picked option 2 and shipped in five weeks: an outbox table written in the same transaction as each change, a job that fans out to followers, a preferences table, and a daily digest to stay out of spam territory.
What broke at 10× scale for the second team:
- Fan-out on popular items. One change to an item with 60,000 followers became a single huge job that blocked the queue for minutes, delaying password-reset emails behind it. They fixed it with chunked fan-out (1,000 recipients per job) and a separate queue for user-facing emails.
- Preference checks became an N+1. Checking each recipient’s preferences one at a time dominated job time. They switched to loading preferences in bulk per chunk.
What didn’t matter: SMS was never built, and push took one week to add because the delivery channel had been kept behind a small interface.
Senior commentary
“The deadline and the team size are the most important facts in this prompt. Three people and six weeks don’t leave room to run a new piece of infrastructure, so option 1 has to justify itself against that, not against some ideal architecture.
“I’d go with option 2, and say clearly what I’m giving up: notification load shares the database with everything else, and if this ever needs to scale independently, we’ll have to extract it. I’d make that extraction cheap by keeping the module’s boundary clean: events in, deliveries out, no reaching into other tables.
“Then I’d ask where the load is skewed. The follower counts are in the prompt for a reason: 60,000 followers on one item is the case that will hurt. You don’t need to solve it in week 1, but you should design so a single event doesn’t become a single unbounded job.
“What would change my mind: if marketing needs complex scheduling and A/B tests on templates within the quarter, a third-party platform starts to look a lot cheaper than building it.”
Discussion questions
- Which requirement did you treat as fixed, and which did you push back on? How would you say that to the product manager?
- Where is the load skewed in this system, and what does that do to a naive design?
- Which parts of your design are hard to reverse? Did you spend your effort there?
- If you picked a third-party platform, what are you locked into, and how would you get out?