Sample candidate
Platform Engineer at Acme
Sample report · 27:24 screen
We wrote this screen for a made-up company, Acme, with a made-up candidate. The report is what our AI wrote from it, unedited, as your team would get it for each applicant.
The interviewer’s call at the Senior level
No Hire
Repeated outage probes ended with one failed customer's HTTP latency or retry backlog still delaying other customers. That depth gap in the central delivery mechanism decides the call. The next interviewer should first test whether the candidate can design fair dispatch with bounded concurrency per endpoint.
- Customer isolation stayed unresolved across both live delivery and retry scheduling.
- Atomic capture and duplicate handling met the senior bar.
- Security answers had concrete depth, while the central outage mechanism fell short.
How each area rated against the Senior bar
This is the AI interviewer’s read of one short screen. Your team makes the hiring decision.
What held them back
- A failed customer still consumes shared delivery capacityBlocks Senior and StaffTechnical depth · Tradeoff judgment
What the candidate said
A timed-out event still holds its Kafka partition for five seconds. Retry workers requeue rate-limited events in shared FIFO topics, and a small customer's retry may wait 10 to 15 minutes. More partitions and workers are the fallback.
Why it matters
The interviewer repeatedly pressed on healthy customers sharing capacity with a failed endpoint. The final mechanisms still let one customer's latency or backlog delay other customers. Adding partitions also leaves a hot customer's serial five-per-second delivery ceiling unchanged.
A passing answer
Separate Kafka consumption from HTTP delivery with durable delivery jobs. Schedule ready endpoints round-robin and cap each endpoint's in-flight requests, so a timeout or throttled backlog consumes only that endpoint's share.
An excellent answer
Add a circuit breaker that probes a failed endpoint with backoff and ramps concurrency up after recovery. Keep delayed work outside the ready queue so workers spend their capacity on endpoints eligible to receive an attempt.
Accepted a small customer's retry waiting behind the large customer's requeued backlog.
12:40· Sample candidate· “Behind the big customer's backlog. The retry workers read the topic in order, and most of what they pull is the big customer's events going back with a delay. So the small customer waits until the workers get round to it, maybe 10 or 15 minutes. I think that's acceptable for a retry. Their endpoint already failed once, so they're getting that event late anyway, and our docs only promise retries for up to three days. It only lasts while the backlog drains. If it got worse, I'd add retry workers so the backlog clears faster.”
Reduced the timeout and increased partition count after sustained timeout overload was made explicit.
14:47· Sample candidate· “Then the partition still falls behind, because it spends ten seconds on each of their events and new ones keep coming. Hmm. I'd lower the timeout to five seconds, and add partitions so fewer customers share each one. And I'd alert when a partition is more than a minute behind, so we can see which customer is causing it.”
Continued to size delivery through serial per-partition throughput.
21:05· Sample candidate· “Ten times our normal peak is 10,000 a second. The main topic is where I'd run out first. Each partition sends one event at a time, about five a second, so 10,000 a second needs 2,000 partitions, and I have 256. I'd go to 2,048 a week ahead. Adding partitions moves customers to new ones, which can reorder a few of their events during the switch, so I'd do it at a quiet hour. A worker can handle a few hundred partitions with async HTTP, so that's about ten workers, and I'd run twenty. The other one is the outbox. That's 10,000 inserts a second into the same Postgres as the invoices, plus the relay reading them. I'd have the relay delete rows in batches once Kafka has them, so the table stays small, and watch that primary's write load. Let me note the numbers.”
- The delivery contract changed after it shaped the designProblem framing · Tradeoff judgment
What the candidate said
Kafka is keyed by customer so delivery stays ordered. Later, delivery becomes unordered with object versions, and a small customer's retry may take 10 to 15 minutes.
Why it matters
The initial ordering promise drove one in-flight request per partition. The candidate later relaxed that promise and never asked how late healthy customers' deliveries could be.
A passing answer
Propose a fresh-delivery target and confirm whether per-customer ordering is required. If unordered at-least-once delivery meets the requirement, use the existing event IDs and object versions while allowing concurrent sends.
An excellent answer
Agree on the automatic retry window separately from fresh-delivery latency. Explain how receivers use object versions before choosing the delivery concurrency model.
Promised ordering and used serial partition delivery to size capacity.
1:22· Sample candidate· “Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.”
Chose acceptable retry latency during the fairness probe.
12:40· Sample candidate· “Behind the big customer's backlog. The retry workers read the topic in order, and most of what they pull is the big customer's events going back with a delay. So the small customer waits until the workers get round to it, maybe 10 or 15 minutes. I think that's acceptable for a retry. Their endpoint already failed once, so they're getting that event late anyway, and our docs only promise retries for up to three days. It only lasts while the backlog drains. If it got worse, I'd add retry workers so the backlog clears faster.”
Acknowledged that retries invalidate the original ordering promise.
23:35· Sample candidate· “That can happen once an event goes to the retry topic. The partition moves on, so newer events for that customer go out while the failed one waits for its retry. I said earlier the order holds, but that's only true when nothing fails. I wouldn't promise strict order, because holding a customer's newer events behind a failed one is what blocked the partition in the first place. Every event carries its created time and the object's version. So a handler should ignore an update that's older than what it already has, or fetch the current object from our API.”
- The recovery limit excludes fresh trafficTechnical depth · Tradeoff judgment
What the candidate said
Retries are limited to 50 requests a second per customer in Redis. The 430,000-event backlog clears in roughly two and a half hours while new events continue.
Why it matters
If the live path keeps up with the stated 40 fresh events a second, the endpoint could receive about 90 requests a second during recovery. That load could overwhelm an endpoint sized for its normal traffic.
A passing answer
Use one token bucket per endpoint across fresh delivery and retries. Reserve part of that total budget for fresh work and drain the backlog with the remaining capacity.
An excellent answer
At a shared limit of 50 requests a second with 40 fresh events a second, the backlog drains at 10 a second and takes about 12 hours. Ramp toward the configured total limit after a successful recovery probe.
The interviewer set the large customer's arrival rate at 40 events a second.
9:32· Interviewer· “Okay. The big customer sends 40 events a second, and every one fails. Three hours in, what's in your retry topic, and when does a small customer's one failed event get its next try?”
The candidate introduced a retry-only limit and calculated drain time using all 50 requests a second.
11:11· Sample candidate· “The retry workers would fire all of them at that endpoint as fast as they can go. Their servers have only just recovered, so that flood would probably knock them over again. So I'd put a rate limit on retries per customer, say 50 a second, with a counter in Redis. When a worker pulls one of their events and they're over the limit, it puts the event back on the retry topic with a short delay. That clears the 430,000 in about two and a half hours, on top of their new events, and their servers never see a spike. Customers can raise their limit in the dashboard if they can take more. Let me add the rate limit.”
Retry transfer needs an explicit commit ruleCore architecture · Technical depth
What the candidate said
After failed attempts, a worker publishes the event to a retry topic and moves on. Consumer offsets are committed every few hundred events, and the candidate did not specify the order between retry acknowledgment and the source offset commit.
Why it matters
A crash can lose the queued event if the source offset advances before retry publication is durable. The candidate correctly handled the equivalent publish-then-mark window in the outbox relay, so this is a gap in extending that rule across the delivery path.
A passing answer
Apply the outbox relay's acknowledgment rule to retry handoff: wait for Kafka to acknowledge the retry record before advancing the source offset. A crash before the offset commit can then create another copy with the same event ID.
An excellent answer
Use a Kafka transaction for retry publication and the consumed-offset commit when both topics share a cluster. Keep the HTTP side effect at least once and preserve the existing delivery-log checks.
Correctly waited for Kafka acknowledgment before marking outbox rows sent.
4:54· Sample candidate· “When the relay restarts, the row is still unsent, so it publishes it again, and Kafka has the event twice. That's fine, because both copies carry the same event ID. The worker records each delivery it completes in a delivery log, keyed on event and endpoint, so when the second copy arrives it sees the first one already went out and skips it. If one ever slips through, the customer dedupes on the ID anyway. I'd rather send an event twice than lose one. To keep that window small, the relay marks rows sent in batches, right after Kafka acknowledges them.”
Introduced retry publication followed by continuing the main partition.
7:55· Sample candidate· “Three hours. Oh. That's bad. The other customer's endpoint is fine, and we'd be holding every one of their events, and everyone else's on that partition, until the big customer recovers. With 40,000 endpoints over 256 partitions, that's about 150 other endpoints stuck behind one outage. So the worker can't keep retrying in place. I'd give each event three quick retries, a second or two apart. If it still fails, the worker publishes it to a retry topic and moves on with the partition. A retry consumer reads that topic, waits until each event's next attempt time, and tries again with longer backoff. That keeps the main partitions flowing, and only the failing customer's events end up in the retry topic. Let me add the retry topic and its workers.”
Discussed offset commit frequency without covering retry handoff ordering.
22:40· Sample candidate· “Kafka gives each partition to one consumer in the group, so two can only overlap during a rebalance. The new one starts from the last committed offset, so it can resend events the old one already sent but hadn't committed yet. That's a duplicate, and duplicates are already part of the deal. The delivery log catches most of them, since the worker checks it before sending, and the customer dedupes on the event ID for the rest. I'd commit offsets every few hundred events to keep that window small.”
Strengths
Atomic event capture
The candidate independently chose a transactional outbox in the opening design. When the interviewer asked about a relay crash, the candidate preserved the event ID across republishing and waited for Kafka acknowledgment before marking rows sent.
1:22· Sample candidate· “Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.”
Honest delivery semantics
The candidate opened with the HTTP ambiguity that makes duplicate delivery unavoidable. The candidate then added sender-side delivery records and concrete receiver deduplication guidance when the interviewer probed crash windows.
1:22· Sample candidate· “Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.”
Concrete webhook security
When the interviewer raised authentication and hostile endpoint URLs, the candidate gave mechanisms down to raw-body HMAC signing and connecting to the exact validated IP. Secret overlap and isolated worker networking addressed deployment and network failure cases.
15:29· Sample candidate· “We sign every request. Each endpoint gets its own secret when it's registered. We compute an HMAC SHA-256 over the timestamp and the raw body, and send it in a header with the timestamp. The customer recomputes it with their secret and compares. They should also reject anything with a timestamp more than five minutes old, so someone who captures a request can't replay it later. To rotate the secret, the customer generates a new one in the dashboard, and for 24 hours we sign with both and send two signatures. That gives them time to deploy the new secret before the old one stops working.”
Bounded failure handling
The candidate distinguished permanent event rejection from retryable HTTP responses when asked. For long outages, the candidate gave customers a 30-day replay path after the automatic retry window.
18:35· Sample candidate· “A 400 means they're rejecting that event, so retrying it for days won't help. I'd treat most 4xx responses as permanent for that event. Retry it a few times over an hour, in case it's a bad deploy on their side, then mark it failed and show it in their dashboard with our request and their response. The exceptions are 408 and 429. For a 429 I'd respect their Retry-After header and put the event on the retry topic with that delay.”
Customer testing and system alerts
The candidate used the same delivery machinery for test events and local development. The candidate also separated ordinary customer endpoint failures from system conditions that should page Acme.
24:37· Sample candidate· “From the dashboard they can send a test event of any type to their endpoint, and see our request and their response side by side. For local work, our CLI opens a tunnel and forwards their test mode events to localhost. Test mode events go through the same delivery path as live ones, retries and signatures included, so what they see in testing is what they'll get.”
What to probe next
Press on a shared Kafka partition when one endpoint hangs, then ask how a small customer's due retry gets served behind 430,000 jobs for another endpoint. Require a concrete dispatcher and one endpoint budget across fresh traffic and retries. Probe crash safety when work moves between topics.
The full breakdown
Signal dimensions
Each area is rated against the bar for the Senior level. Strong: clearly met it. Solid: met it, but came close to falling short. Concern: fell short of it. Critical concern: fell far short.
Problem framing
SolidThe candidate set delivery semantics and scale before designing. The candidate chose ordering and later accepted retry latency without agreeing on the customer contract.
What held them back
Started with at-least-once delivery and calculated about 230 events a second on average, then chose ordered delivery.
1:22· Sample candidate· “Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.”
Accepted a 10 to 15 minute retry delay without an agreed freshness target.
12:40· Sample candidate· “Behind the big customer's backlog. The retry workers read the topic in order, and most of what they pull is the big customer's events going back with a delay. So the small customer waits until the workers get round to it, maybe 10 or 15 minutes. I think that's acceptable for a retry. Their endpoint already failed once, so they're getting that event late anyway, and our docs only promise retries for up to three days. It only lasts while the backlog drains. If it got worse, I'd add retry workers so the backlog clears faster.”
Revised the earlier ordering promise after the interviewer asked about reversed event arrival.
23:35· Sample candidate· “That can happen once an event goes to the retry topic. The partition moves on, so newer events for that customer go out while the failed one waits for its retry. I said earlier the order holds, but that's only true when nothing fails. I wouldn't promise strict order, because holding a customer's newer events behind a failed one is what blocked the partition in the first place. Every event carries its created time and the object's version. So a handler should ignore an update that's older than what it already has, or fetch the current object from our API.”
Core architecture
StrongThe outbox, Kafka workers and delivery records form a connected delivery path. The candidate added retry tiers and replay in response to the interviewer's scenarios.
What held them back
Connected the invoice transaction to an outbox relay, Kafka and HTTPS delivery workers.
1:22· Sample candidate· “Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.”
Added a delivery log keyed by event and endpoint to suppress completed deliveries.
4:54· Sample candidate· “When the relay restarts, the row is still unsent, so it publishes it again, and Kafka has the event twice. That's fine, because both copies carry the same event ID. The worker records each delivery it completes in a delivery log, keyed on event and endpoint, so when the second copy arrives it sees the first one already went out and skips it. If one ever slips through, the customer dedupes on the ID anyway. I'd rather send an event twice than lose one. To keep that window small, the relay marks rows sent in batches, right after Kafka acknowledges them.”
Gave failed events a dead-letter table and a dashboard or API replay path.
19:27· Sample candidate· “By now its events have worked down the retry tiers, so they're all on the hourly one. After a day of failures we email the account's admins, and again on day three. On day three we stop retrying, move its events to a dead letter table, and keep them for 30 days. When they've fixed it, they replay everything since it went down, from the dashboard or the API. The replay goes through the same rate limit, so it doesn't flood them. Past 30 days, they'd rebuild from our API, by fetching the current state of the objects they care about. Let me add the dead letter table and the replay API.”
Technical depth
ConcernThe outage deep dive ended with partition blocking and retry starvation still in place after repeated probes. The candidate reached concrete mechanisms for request authentication and SSRF protection.
What held them back
Acknowledged that a small customer's retry waits behind another customer's backlog and accepted the delay.
12:40· Sample candidate· “Behind the big customer's backlog. The retry workers read the topic in order, and most of what they pull is the big customer's events going back with a delay. So the small customer waits until the workers get round to it, maybe 10 or 15 minutes. I think that's acceptable for a retry. Their endpoint already failed once, so they're getting that event late anyway, and our docs only promise retries for up to three days. It only lasts while the backlog drains. If it got worse, I'd add retry workers so the backlog clears faster.”
Answered sustained timeout overload with a shorter timeout and more partitions, which leaves the serial bottleneck for each customer.
14:47· Sample candidate· “Then the partition still falls behind, because it spends ten seconds on each of their events and new ones keep coming. Hmm. I'd lower the timeout to five seconds, and add partitions so fewer customers share each one. And I'd alert when a partition is more than a minute behind, so we can see which customer is causing it.”
Explained address validation on every send, connecting to the checked address and network isolation.
16:34· Sample candidate· “Then our workers would call our own internal services for them. That's SSRF. They could hit the cloud metadata address, 169.254.169.254, and read our credentials, or reach an internal admin API. So when a URL is registered, and again on every send, we resolve the hostname and refuse private, loopback and link-local addresses. The worker connects to the exact address it checked, so the DNS can't change between the check and the request. We don't follow redirects. And the delivery workers run in their own network with no route to our internal services, sending from a fixed set of IPs that customers can allowlist.”
Tradeoff judgment
SolidThe candidate tied duplicate handling and payload limits to concrete customer costs. The retry-delay decision relied on a latency policy the candidate chose during the probe.
What held them back
Chose at-least-once delivery because an HTTP response can disappear after processing.
1:22· Sample candidate· “Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.”
Accepted shared retry delays because the endpoint had already failed and retries continued for three days.
12:40· Sample candidate· “Behind the big customer's backlog. The retry workers read the topic in order, and most of what they pull is the big customer's events going back with a delay. So the small customer waits until the workers get round to it, maybe 10 or 15 minutes. I think that's acceptable for a retry. Their endpoint already failed once, so they're getting that event late anyway, and our docs only promise retries for up to three days. It only lasts while the backlog drains. If it got worse, I'd add retry workers so the backlog clears faster.”
Chose thin events for oversized payloads because large requests may be rejected and make retries expensive.
17:38· Sample candidate· “I wouldn't send 3 megabytes in a webhook. Some customers' servers cap request bodies at a megabyte, and a big body makes every retry expensive. I'd cap the payload at 256 kilobytes. Anything bigger goes out as a thin event, with the event ID, the type and the invoice ID, and a flag saying it was trimmed. The customer fetches the full invoice from our API. That's often better anyway, because they get the current state, and a retry that lands hours later can't hand them a stale copy.”
Driving the interview
StrongThe candidate owned the opening design and made the revisions without handing decisions back. The interviewer chose the later failure scenarios, which is normal steering at senior.
Independently presented semantics, architecture and capacity calculations in the opening answer.
1:22· Sample candidate· “Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.”
Chose retry topics after the interviewer exposed partition blocking.
7:55· Sample candidate· “Three hours. Oh. That's bad. The other customer's endpoint is fine, and we'd be holding every one of their events, and everyone else's on that partition, until the big customer recovers. With 40,000 endpoints over 256 partitions, that's about 150 other endpoints stuck behind one outage. So the worker can't keep retrying in place. I'd give each event three quick retries, a second or two apart. If it still fails, the worker publishes it to a retry topic and moves on with the partition. A retry consumer reads that topic, waits until each event's next attempt time, and tries again with longer backoff. That keeps the main partitions flowing, and only the failing customer's events end up in the retry topic. Let me add the retry topic and its workers.”
Answered the capacity probe with partition and worker counts, then raised outbox write load.
21:05· Sample candidate· “Ten times our normal peak is 10,000 a second. The main topic is where I'd run out first. Each partition sends one event at a time, about five a second, so 10,000 a second needs 2,000 partitions, and I have 256. I'd go to 2,048 a week ahead. Adding partitions moves customers to new ones, which can reorder a few of their events during the switch, so I'd do it at a quiet hour. A worker can handle a few hundred partitions with async HTTP, so that's about ten workers, and I'd run twenty. The other one is the outbox. That's 10,000 inserts a second into the same Postgres as the invoices, plus the relay reading them. I'd have the relay delete rows in batches once Kafka has them, so the table stays small, and watch that primary's write load. Let me note the numbers.”
Communication
StrongThe candidate answered the scenarios directly and made the calculations easy to follow. The candidate explicitly corrected the ordering promise when the design changed.
Calculated the three-hour backlog and explained its effect on the shared retry queue.
9:49· Sample candidate· “40 a second for three hours is about 430,000 events, all from the big customer. So the retry topic is almost entirely theirs. The small customer's event sits in among them, and the retry workers read the topic in order, so it waits behind that backlog too. I could split it into a few retry topics by delay. One for a minute, one for ten minutes, one for an hour. Events move down the tiers as they keep failing, so most of the big customer's would sink to the hourly one, and the small customer's would stay up in the one minute tier. And I'd add more retry workers, so the backlog gets worked through faster. Let me change the retry topic on the board.”
Walked through timeout cost and its effect on the main partitions.
13:45· Sample candidate· “Each of those customers' events gets three quick retries before it goes to the retry topic, and each try waits out the ten second timeout. So one event holds its partition for about 30 seconds. 300 endpoints over 256 partitions means most partitions have one of them, and each of those partitions stalls for 30 seconds on every one of that customer's events. Everyone else on those partitions gets their events late. So I'd send a timeout straight to the retry topic, without the quick retries. Then a hanging endpoint costs its partition ten seconds an event instead of 30.”
Clearly acknowledged that retry topics had invalidated the earlier ordering claim.
23:35· Sample candidate· “That can happen once an event goes to the retry topic. The partition moves on, so newer events for that customer go out while the failed one waits for its retry. I said earlier the order holds, but that's only true when nothing fails. I wouldn't promise strict order, because holding a customer's newer events behind a failed one is what blocked the partition in the first place. Every event carries its created time and the object's version. So a handler should ignore an update that's older than what it already has, or fetch the current object from our API.”
Topic coverage
- Problem framing · Tradeoff judgment
Delivery contract and workload
Partially coveredThe candidate never agreed on healthy-endpoint latency or ordering requirements before choosing serial delivery.
Set at-least-once semantics and worked out average and assumed peak traffic.
1:22· Sample candidate· “Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.”
Chose an acceptable retry delay during the backlog probe.
12:40· Sample candidate· “Behind the big customer's backlog. The retry workers read the topic in order, and most of what they pull is the big customer's events going back with a delay. So the small customer waits until the workers get round to it, maybe 10 or 15 minutes. I think that's acceptable for a retry. Their endpoint already failed once, so they're getting that event late anyway, and our docs only promise retries for up to three days. It only lasts while the backlog drains. If it got worse, I'd add retry workers so the backlog clears faster.”
Changed the ordering contract late in the round.
23:35· Sample candidate· “That can happen once an event goes to the retry topic. The partition moves on, so newer events for that customer go out while the failed one waits for its retry. I said earlier the order holds, but that's only true when nothing fails. I wouldn't promise strict order, because holding a customer's newer events behind a failed one is what blocked the partition in the first place. Every event carries its created time and the object's version. So a handler should ignore an update that's older than what it already has, or fetch the current object from our API.”
- Technical depth · Tradeoff judgment
Isolation from hanging endpoints
StalledThe candidate never separated a shared Kafka partition's progress from the HTTP latency of each endpoint.
Moved repeated failures to a retry topic after the interviewer exposed hours of partition blocking.
7:55· Sample candidate· “Three hours. Oh. That's bad. The other customer's endpoint is fine, and we'd be holding every one of their events, and everyone else's on that partition, until the big customer recovers. With 40,000 endpoints over 256 partitions, that's about 150 other endpoints stuck behind one outage. So the worker can't keep retrying in place. I'd give each event three quick retries, a second or two apart. If it still fails, the worker publishes it to a retry topic and moves on with the partition. A retry consumer reads that topic, waits until each event's next attempt time, and tries again with longer backoff. That keeps the main partitions flowing, and only the failing customer's events end up in the retry topic. Let me add the retry topic and its workers.”
Removed quick retries for timeouts but still held the partition for each HTTP timeout.
13:45· Sample candidate· “Each of those customers' events gets three quick retries before it goes to the retry topic, and each try waits out the ten second timeout. So one event holds its partition for about 30 seconds. 300 endpoints over 256 partitions means most partitions have one of them, and each of those partitions stalls for 30 seconds on every one of that customer's events. Everyone else on those partitions gets their events late. So I'd send a timeout straight to the retry topic, without the quick retries. Then a hanging endpoint costs its partition ten seconds an event instead of 30.”
Shortened the timeout and added partitions after the interviewer showed that arrivals still exceeded service.
14:47· Sample candidate· “Then the partition still falls behind, because it spends ten seconds on each of their events and new ones keep coming. Hmm. I'd lower the timeout to five seconds, and add partitions so fewer customers share each one. And I'd alert when a partition is more than a minute behind, so we can see which customer is causing it.”
- Technical depth · Tradeoff judgment
Retry scheduling and fairness
StalledThe candidate never gave a dispatcher that serves a small endpoint's due retry without cycling through another endpoint's throttled backlog.
Split retries into delay tiers after calculating a backlog of about 430,000 events.
9:49· Sample candidate· “40 a second for three hours is about 430,000 events, all from the big customer. So the retry topic is almost entirely theirs. The small customer's event sits in among them, and the retry workers read the topic in order, so it waits behind that backlog too. I could split it into a few retry topics by delay. One for a minute, one for ten minutes, one for an hour. Events move down the tiers as they keep failing, so most of the big customer's would sink to the hourly one, and the small customer's would stay up in the one minute tier. And I'd add more retry workers, so the backlog gets worked through faster. Let me change the retry topic on the board.”
Requeued rate-limited events into the shared retry topics.
11:11· Sample candidate· “The retry workers would fire all of them at that endpoint as fast as they can go. Their servers have only just recovered, so that flood would probably knock them over again. So I'd put a rate limit on retries per customer, say 50 a second, with a counter in Redis. When a worker pulls one of their events and they're over the limit, it puts the event back on the retry topic with a short delay. That clears the 430,000 in about two and a half hours, on top of their new events, and their servers never see a spike. Customers can raise their limit in the dashboard if they can take more. Let me add the rate limit.”
Accepted a small customer's retry waiting behind that requeued work.
12:40· Sample candidate· “Behind the big customer's backlog. The retry workers read the topic in order, and most of what they pull is the big customer's events going back with a delay. So the small customer waits until the workers get round to it, maybe 10 or 15 minutes. I think that's acceptable for a retry. Their endpoint already failed once, so they're getting that event late anyway, and our docs only promise retries for up to three days. It only lasts while the backlog drains. If it got worse, I'd add retry workers so the backlog clears faster.”
- Technical depth · Tradeoff judgment
Recovery admission control
Partially coveredThe candidate limited retries per customer without a shared budget for fresh delivery and retries.
Introduced a Redis retry limit of 50 requests a second after the interviewer asked about recovery flooding.
11:11· Sample candidate· “The retry workers would fire all of them at that endpoint as fast as they can go. Their servers have only just recovered, so that flood would probably knock them over again. So I'd put a rate limit on retries per customer, say 50 a second, with a counter in Redis. When a worker pulls one of their events and they're over the limit, it puts the event back on the retry topic with a short delay. That clears the 430,000 in about two and a half hours, on top of their new events, and their servers never see a spike. Customers can raise their limit in the dashboard if they can take more. Let me add the rate limit.”
Sent manual replay through the same retry limit.
19:27· Sample candidate· “By now its events have worked down the retry tiers, so they're all on the hourly one. After a day of failures we email the account's admins, and again on day three. On day three we stop retrying, move its events to a dead letter table, and keep them for 30 days. When they've fixed it, they replay everything since it went down, from the dashboard or the API. The replay goes through the same rate limit, so it doesn't flood them. Past 30 days, they'd rebuild from our API, by fetching the current state of the objects they care about. Let me add the dead letter table and the replay API.”
- Core architecture · Technical depth
Crash-safe retry handoff
Partially coveredThe candidate did not state that retry publication must be durable before the source offset advances.
Said workers publish failed events to a retry topic and move on.
7:55· Sample candidate· “Three hours. Oh. That's bad. The other customer's endpoint is fine, and we'd be holding every one of their events, and everyone else's on that partition, until the big customer recovers. With 40,000 endpoints over 256 partitions, that's about 150 other endpoints stuck behind one outage. So the worker can't keep retrying in place. I'd give each event three quick retries, a second or two apart. If it still fails, the worker publishes it to a retry topic and moves on with the partition. A retry consumer reads that topic, waits until each event's next attempt time, and tries again with longer backoff. That keeps the main partitions flowing, and only the failing customer's events end up in the retry topic. Let me add the retry topic and its workers.”
Discussed batched offset commits while answering consumer overlap.
22:40· Sample candidate· “Kafka gives each partition to one consumer in the group, so two can only overlap during a rebalance. The new one starts from the last committed offset, so it can resend events the old one already sent but hadn't committed yet. That's a duplicate, and duplicates are already part of the deal. The delivery log catches most of them, since the worker checks it before sending, and the customer dedupes on the event ID for the rest. I'd commit offsets every few hundred events to keep that window small.”
- Technical depth · Tradeoff judgment
Peak capacity and hot customers
Partially coveredThe candidate never explained how one customer's 40 events a second fits through its serial partition at five sends a second.
Derived five sends a second per partition from a 200 millisecond request.
1:22· Sample candidate· “Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.”
The interviewer introduced one customer producing 40 events a second.
9:32· Interviewer· “Okay. The big customer sends 40 events a second, and every one fails. Three hours in, what's in your retry topic, and when does a small customer's one failed event get its next try?”
Scaled aggregate capacity by adding partitions and workers, then identified outbox write load.
21:05· Sample candidate· “Ten times our normal peak is 10,000 a second. The main topic is where I'd run out first. Each partition sends one event at a time, about five a second, so 10,000 a second needs 2,000 partitions, and I have 256. I'd go to 2,048 a week ahead. Adding partitions moves customers to new ones, which can reorder a few of their events during the switch, so I'd do it at a quiet hour. A worker can handle a few hundred partitions with async HTTP, so that's about ten workers, and I'd run twenty. The other one is the outbox. That's 10,000 inserts a second into the same Postgres as the invoices, plus the relay reading them. I'd have the relay delete rows in batches once Kafka has them, so the table stays small, and watch that primary's write load. Let me note the numbers.”
Durable event captureCovered
Wrote the business change and outbox row in one Postgres transaction.
1:22· Sample candidate· “Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.”
Handled a relay crash by republishing the same event ID and marking rows sent after Kafka acknowledgment.
4:54· Sample candidate· “When the relay restarts, the row is still unsent, so it publishes it again, and Kafka has the event twice. That's fine, because both copies carry the same event ID. The worker records each delivery it completes in a delivery log, keyed on event and endpoint, so when the second copy arrives it sees the first one already went out and skips it. If one ever slips through, the customer dedupes on the ID anyway. I'd rather send an event twice than lose one. To keep that window small, the relay marks rows sent in batches, right after Kafka acknowledges them.”
Webhook contract and endpoint lookupCovered
Workers look up the customer's endpoint and secret before sending an HTTPS POST.
1:22· Sample candidate· “Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.”
Named event identity, type and creation time in the delivery contract.
5:59· Sample candidate· “We count it as a failure and send it again, so the customer gets it twice. We can't know their side already processed it. That's why every request carries the event ID in a header, along with the event type and when it was created. Our docs tell customers to store the IDs they've handled and drop repeats, and our SDKs do that check for them, with a table or a Redis key that expires after a few days. We'd also tell them to answer with a 200 straight away and do the work on their own queue. A handler doing heavy work inline is what causes these timeouts in the first place.”
Added object versions for handling out-of-order events.
23:35· Sample candidate· “That can happen once an event goes to the retry topic. The partition moves on, so newer events for that customer go out while the failed one waits for its retry. I said earlier the order holds, but that's only true when nothing fails. I wouldn't promise strict order, because holding a customer's newer events behind a failed one is what blocked the partition in the first place. Every event carries its created time and the object's version. So a handler should ignore an update that's older than what it already has, or fetch the current object from our API.”
HTTP ambiguity and duplicate deliveryCovered
Used a delivery log keyed by event and endpoint.
4:54· Sample candidate· “When the relay restarts, the row is still unsent, so it publishes it again, and Kafka has the event twice. That's fine, because both copies carry the same event ID. The worker records each delivery it completes in a delivery log, keyed on event and endpoint, so when the second copy arrives it sees the first one already went out and skips it. If one ever slips through, the customer dedupes on the ID anyway. I'd rather send an event twice than lose one. To keep that window small, the relay marks rows sent in batches, right after Kafka acknowledges them.”
Explained retrying after an ambiguous timeout and receiver deduplication by event ID.
5:59· Sample candidate· “We count it as a failure and send it again, so the customer gets it twice. We can't know their side already processed it. That's why every request carries the event ID in a header, along with the event type and when it was created. Our docs tell customers to store the IDs they've handled and drop repeats, and our SDKs do that check for them, with a table or a Redis key that expires after a few days. We'd also tell them to answer with a 200 straight away and do the work on their own queue. A handler doing heavy work inline is what causes these timeouts in the first place.”
Explained duplicate sends during consumer overlap and offset replay.
22:40· Sample candidate· “Kafka gives each partition to one consumer in the group, so two can only overlap during a rebalance. The new one starts from the last committed offset, so it can resend events the old one already sent but hadn't committed yet. That's a duplicate, and duplicates are already part of the deal. The delivery log catches most of them, since the worker checks it before sending, and the customer dedupes on the event ID for the rest. I'd commit offsets every few hundred events to keep that window small.”
Failure policy and replayCovered
Distinguished most 4xx responses from retryable responses and respected Retry-After for 429.
18:35· Sample candidate· “A 400 means they're rejecting that event, so retrying it for days won't help. I'd treat most 4xx responses as permanent for that event. Retry it a few times over an hour, in case it's a bad deploy on their side, then mark it failed and show it in their dashboard with our request and their response. The exceptions are 408 and 429. For a 429 I'd respect their Retry-After header and put the event on the retry topic with that delay.”
Stopped automatic retries after three days and kept dead-letter events for 30 days with replay.
19:27· Sample candidate· “By now its events have worked down the retry tiers, so they're all on the hourly one. After a day of failures we email the account's admins, and again on day three. On day three we stop retrying, move its events to a dead letter table, and keep them for 30 days. When they've fixed it, they replay everything since it went down, from the dashboard or the API. The replay goes through the same rate limit, so it doesn't flood them. Past 30 days, they'd rebuild from our API, by fetching the current state of the objects they care about. Let me add the dead letter table and the replay API.”
Request authentication and SSRFCovered
Gave HMAC signing, timestamp validation and overlapping signatures during rotation.
15:29· Sample candidate· “We sign every request. Each endpoint gets its own secret when it's registered. We compute an HMAC SHA-256 over the timestamp and the raw body, and send it in a header with the timestamp. The customer recomputes it with their secret and compares. They should also reject anything with a timestamp more than five minutes old, so someone who captures a request can't replay it later. To rotate the secret, the customer generates a new one in the dashboard, and for 24 hours we sign with both and send two signatures. That gives them time to deploy the new secret before the old one stops working.”
Explained checked-address connections, redirect refusal and isolated worker networking.
16:34· Sample candidate· “Then our workers would call our own internal services for them. That's SSRF. They could hit the cloud metadata address, 169.254.169.254, and read our credentials, or reach an internal admin API. So when a URL is registered, and again on every send, we resolve the hostname and refuse private, loopback and link-local addresses. The worker connects to the exact address it checked, so the DNS can't change between the check and the request. We don't follow redirects. And the delivery workers run in their own network with no route to our internal services, sending from a fixed set of IPs that customers can allowlist.”
Payload size policyCovered
Capped payload size and sent thin events that let customers fetch the object through the API.
17:38· Sample candidate· “I wouldn't send 3 megabytes in a webhook. Some customers' servers cap request bodies at a megabyte, and a big body makes every retry expensive. I'd cap the payload at 256 kilobytes. Anything bigger goes out as a thin event, with the event ID, the type and the invoice ID, and a flag saying it was trimmed. The customer fetches the full invoice from our API. That's often better anyway, because they get the current state, and a retry that lands hours later can't hand them a stale copy.”
Integration testing and operationsCovered
Gave dashboard test events and local forwarding through the live delivery machinery.
24:37· Sample candidate· “From the dashboard they can send a test event of any type to their endpoint, and see our request and their response side by side. For local work, our CLI opens a tunnel and forwards their test mode events to localhost. Test mode events go through the same delivery path as live ones, retries and signatures included, so what they see in testing is what they'll get.”
Chose outbox age, main consumer lag and sustained retry growth as system alerts.
25:23· Sample candidate· “First, outbox lag. If the oldest unsent outbox row is over a minute old, events aren't even getting into the system. Second, consumer lag on the main topic. If a partition is more than a minute behind, every customer on it is getting their events late. Third, the retry topics growing for an hour straight. One customer's endpoint failing isn't a page. That's normal, and they get the dashboard and the emails.”
- Broker and database outage recoveryNot raised
The interviewer didn’t raise the topics marked Not raised, so they aren’t something the candidate missed.
Pacing
The opening reached a connected design by 4:42. About eight minutes went to interviewer-led outage probes, and the candidate answered each promptly. The call closed at 27:24, and the clock cut off no phase.
- Opening and initial delivery design0:00–4:42
- Crash windows and duplicates4:42–6:57
- Outage isolation and retry recovery6:57–15:18
- Security, payloads and failure policy15:18–20:54
- Peak capacity and consumer semantics20:54–24:27
- Testing and operations24:27–26:00
- Recap and close26:00–27:24
Get your team’s time back from reviewing applicants
Every applicant takes this screen first, and your team reads their reports to decide who to meet.
Appendix
Final whiteboard
Open full size (new tab)Transcript
- 0:00
Interviewer
Hi, thanks for joining. I'm an AI interviewer, and this is a 30-minute system design interview for Acme. Someone at Acme will review it and decide on next steps. You drive, and I'll jump in with questions as we go. Here's the problem: Acme's API sends a webhook for every event on a customer's account, like an invoice being paid, to an HTTPS endpoint the customer registers. That's about 20 million events a day across 40,000 endpoints. How would you design delivery so every event reaches its endpoint, even when that endpoint is down for hours? Take a minute to gather your thoughts, and start wherever feels natural.
- 1:22
Sample candidate
Okay. So the promise is that every event reaches its endpoint at least once. Over HTTP I can't promise exactly once, because the customer can process a request and we never see the response. So I'd aim for at least once, and give every event an ID the customer can dedupe on. I'll sketch it as I go. First, capturing the event. When the invoice service marks an invoice paid, it writes an outbox row in the same Postgres transaction. If the payment commits, the event exists, and if it rolls back, there's no event. An outbox relay polls those rows every few hundred milliseconds and publishes them to Kafka. In Kafka I'd have one events topic, keyed on the customer ID, so each customer's events stay in order. Delivery workers consume the partitions, look up the customer's endpoint and signing secret, and POST the event with a ten second timeout. A 2xx means delivered. Anything else, the worker retries with exponential backoff and jitter, starting at a second and capping at a minute, and only then moves on to the next event, so the order holds. On numbers, 20 million a day is about 230 a second on average. I'd plan for a peak around 1,000 a second. A partition sends one event at a time to keep the order, and a request takes about 200 milliseconds, so each partition does about five a second. For 1,000 a second that's 200 partitions, so I'd make it 256.
Open full size (new tab)Whiteboard at 4:41 - 4:42
Interviewer
Okay. The relay publishes an event to Kafka, then crashes before it marks the outbox row as sent. What happens?
- 4:54
Sample candidate
When the relay restarts, the row is still unsent, so it publishes it again, and Kafka has the event twice. That's fine, because both copies carry the same event ID. The worker records each delivery it completes in a delivery log, keyed on event and endpoint, so when the second copy arrives it sees the first one already went out and skips it. If one ever slips through, the customer dedupes on the ID anyway. I'd rather send an event twice than lose one. To keep that window small, the relay marks rows sent in batches, right after Kafka acknowledges them.
- 5:46
Interviewer
Okay. The worker sends an event. The customer's server processes it, but its response doesn't arrive before your ten second timeout. What happens next?
- 5:59
Sample candidate
We count it as a failure and send it again, so the customer gets it twice. We can't know their side already processed it. That's why every request carries the event ID in a header, along with the event type and when it was created. Our docs tell customers to store the IDs they've handled and drop repeats, and our SDKs do that check for them, with a table or a Redis key that expires after a few days. We'd also tell them to answer with a 200 straight away and do the work on their own queue. A handler doing heavy work inline is what causes these timeouts in the first place.
- 6:57
Interviewer
Okay. Your biggest customer's endpoint starts returning 503s. Another customer's events are on the same partition. What happens to their events?
- 7:09
Sample candidate
They wait in the queue. The worker keeps retrying that event with backoff until it lands. Once the big customer's endpoint is back, that event goes through and the worker carries on down the partition, so the other customer still gets everything, in order. The backoff caps at a minute, so we keep checking that endpoint at least once a minute without hammering a server that's already struggling.
- 7:45
Interviewer
Okay. The big customer's endpoint stays down for three hours. How long do the other customer's events wait?
- 7:55
Sample candidate
Three hours. Oh. That's bad. The other customer's endpoint is fine, and we'd be holding every one of their events, and everyone else's on that partition, until the big customer recovers. With 40,000 endpoints over 256 partitions, that's about 150 other endpoints stuck behind one outage. So the worker can't keep retrying in place. I'd give each event three quick retries, a second or two apart. If it still fails, the worker publishes it to a retry topic and moves on with the partition. A retry consumer reads that topic, waits until each event's next attempt time, and tries again with longer backoff. That keeps the main partitions flowing, and only the failing customer's events end up in the retry topic. Let me add the retry topic and its workers.
Open full size (new tab)Whiteboard at 9:30 - 9:32
Interviewer
Okay. The big customer sends 40 events a second, and every one fails. Three hours in, what's in your retry topic, and when does a small customer's one failed event get its next try?
- 9:49
Sample candidate
40 a second for three hours is about 430,000 events, all from the big customer. So the retry topic is almost entirely theirs. The small customer's event sits in among them, and the retry workers read the topic in order, so it waits behind that backlog too. I could split it into a few retry topics by delay. One for a minute, one for ten minutes, one for an hour. Events move down the tiers as they keep failing, so most of the big customer's would sink to the hourly one, and the small customer's would stay up in the one minute tier. And I'd add more retry workers, so the backlog gets worked through faster. Let me change the retry topic on the board.
Open full size (new tab)Whiteboard at 10:57 - 10:58
Interviewer
Okay. Then the big customer's endpoint comes back up. Your retry workers have 430,000 of their events. What happens to that customer?
- 11:11
Sample candidate
The retry workers would fire all of them at that endpoint as fast as they can go. Their servers have only just recovered, so that flood would probably knock them over again. So I'd put a rate limit on retries per customer, say 50 a second, with a counter in Redis. When a worker pulls one of their events and they're over the limit, it puts the event back on the retry topic with a short delay. That clears the 430,000 in about two and a half hours, on top of their new events, and their servers never see a spike. Customers can raise their limit in the dashboard if they can take more. Let me add the rate limit.
Open full size (new tab)Whiteboard at 12:26 - 12:27
Interviewer
Okay. While that backlog drains, a small customer has one failed event in the same retry topic. When does it get its next try?
- 12:40
Sample candidate
Behind the big customer's backlog. The retry workers read the topic in order, and most of what they pull is the big customer's events going back with a delay. So the small customer waits until the workers get round to it, maybe 10 or 15 minutes. I think that's acceptable for a retry. Their endpoint already failed once, so they're getting that event late anyway, and our docs only promise retries for up to three days. It only lasts while the backlog drains. If it got worse, I'd add retry workers so the backlog clears faster.
- 13:30
Interviewer
Okay. A cloud provider has an outage, and 300 endpoints go down at once. Their servers don't refuse the connection, they just hang. What happens to your main partitions?
- 13:45
Sample candidate
Each of those customers' events gets three quick retries before it goes to the retry topic, and each try waits out the ten second timeout. So one event holds its partition for about 30 seconds. 300 endpoints over 256 partitions means most partitions have one of them, and each of those partitions stalls for 30 seconds on every one of that customer's events. Everyone else on those partitions gets their events late. So I'd send a timeout straight to the retry topic, without the quick retries. Then a hanging endpoint costs its partition ten seconds an event instead of 30.
- 14:36
Interviewer
Okay. Ten seconds an event, and the customer on that partition sends one every few seconds.
- 14:47
Sample candidate
Then the partition still falls behind, because it spends ten seconds on each of their events and new ones keep coming. Hmm. I'd lower the timeout to five seconds, and add partitions so fewer customers share each one. And I'd alert when a partition is more than a minute behind, so we can see which customer is causing it.
- 15:18
Interviewer
Okay. How does a customer know a request really came from Acme, and not from someone who found their URL?
- 15:29
Sample candidate
We sign every request. Each endpoint gets its own secret when it's registered. We compute an HMAC SHA-256 over the timestamp and the raw body, and send it in a header with the timestamp. The customer recomputes it with their secret and compares. They should also reject anything with a timestamp more than five minutes old, so someone who captures a request can't replay it later. To rotate the secret, the customer generates a new one in the dashboard, and for 24 hours we sign with both and send two signatures. That gives them time to deploy the new secret before the old one stops working.
- 16:24
Interviewer
Okay. A customer registers an endpoint URL that points inside our own network. What happens?
- 16:34
Sample candidate
Then our workers would call our own internal services for them. That's SSRF. They could hit the cloud metadata address, 169.254.169.254, and read our credentials, or reach an internal admin API. So when a URL is registered, and again on every send, we resolve the hostname and refuse private, loopback and link-local addresses. The worker connects to the exact address it checked, so the DNS can't change between the check and the request. We don't follow redirects. And the delivery workers run in their own network with no route to our internal services, sending from a fixed set of IPs that customers can allowlist.
- 17:27
Interviewer
Okay. One customer's invoice has 5,000 line items, so its event is 3 megabytes. What do you send?
- 17:38
Sample candidate
I wouldn't send 3 megabytes in a webhook. Some customers' servers cap request bodies at a megabyte, and a big body makes every retry expensive. I'd cap the payload at 256 kilobytes. Anything bigger goes out as a thin event, with the event ID, the type and the invoice ID, and a flag saying it was trimmed. The customer fetches the full invoice from our API. That's often better anyway, because they get the current state, and a retry that lands hours later can't hand them a stale copy.
- 18:24
Interviewer
Okay. One event gets a 400 back every time. The endpoint accepts everything else. What do you do?
- 18:35
Sample candidate
A 400 means they're rejecting that event, so retrying it for days won't help. I'd treat most 4xx responses as permanent for that event. Retry it a few times over an hour, in case it's a bad deploy on their side, then mark it failed and show it in their dashboard with our request and their response. The exceptions are 408 and 429. For a 429 I'd respect their Retry-After header and put the event on the retry topic with that delay.
- 19:17
Interviewer
Okay. An endpoint has been down for four days. What happens to its events?
- 19:27
Sample candidate
By now its events have worked down the retry tiers, so they're all on the hourly one. After a day of failures we email the account's admins, and again on day three. On day three we stop retrying, move its events to a dead letter table, and keep them for 30 days. When they've fixed it, they replay everything since it went down, from the dashboard or the API. The replay goes through the same rate limit, so it doesn't flood them. Past 30 days, they'd rebuild from our API, by fetching the current state of the objects they care about. Let me add the dead letter table and the replay API.
Open full size (new tab)Whiteboard at 20:53 - 20:54
Interviewer
Okay. Black Friday. Ten times the usual traffic for an hour. Where does your design run out first?
- 21:05
Sample candidate
Ten times our normal peak is 10,000 a second. The main topic is where I'd run out first. Each partition sends one event at a time, about five a second, so 10,000 a second needs 2,000 partitions, and I have 256. I'd go to 2,048 a week ahead. Adding partitions moves customers to new ones, which can reorder a few of their events during the switch, so I'd do it at a quiet hour. A worker can handle a few hundred partitions with async HTTP, so that's about ten workers, and I'd run twenty. The other one is the outbox. That's 10,000 inserts a second into the same Postgres as the invoices, plus the relay reading them. I'd have the relay delete rows in batches once Kafka has them, so the table stays small, and watch that primary's write load. Let me note the numbers.
Open full size (new tab)Whiteboard at 22:28 - 22:29
Interviewer
Okay. After a deploy, two consumers are reading the same partition for a few seconds. What happens?
- 22:40
Sample candidate
Kafka gives each partition to one consumer in the group, so two can only overlap during a rebalance. The new one starts from the last committed offset, so it can resend events the old one already sent but hadn't committed yet. That's a duplicate, and duplicates are already part of the deal. The delivery log catches most of them, since the worker checks it before sending, and the customer dedupes on the event ID for the rest. I'd commit offsets every few hundred events to keep that window small.
- 23:26
Interviewer
Okay. A customer says subscription updated arrived before subscription created. What do you tell them?
- 23:35
Sample candidate
That can happen once an event goes to the retry topic. The partition moves on, so newer events for that customer go out while the failed one waits for its retry. I said earlier the order holds, but that's only true when nothing fails. I wouldn't promise strict order, because holding a customer's newer events behind a failed one is what blocked the partition in the first place. Every event carries its created time and the object's version. So a handler should ignore an update that's older than what it already has, or fetch the current object from our API.
- 24:27
Interviewer
Okay. A customer is building their handler. How do they test it before going live?
- 24:37
Sample candidate
From the dashboard they can send a test event of any type to their endpoint, and see our request and their response side by side. For local work, our CLI opens a tunnel and forwards their test mode events to localhost. Test mode events go through the same delivery path as live ones, retries and signatures included, so what they see in testing is what they'll get.
- 25:12
Interviewer
Okay. You're on call for this. What are the two or three alerts you'd want waking you up?
- 25:23
Sample candidate
First, outbox lag. If the oldest unsent outbox row is over a minute old, events aren't even getting into the system. Second, consumer lag on the main topic. If a partition is more than a minute behind, every customer on it is getting their events late. Third, the retry topics growing for an hour straight. One customer's endpoint failing isn't a page. That's normal, and they get the dashboard and the emails.
- 26:00
Interviewer
Okay. We've got a few minutes left. Give me the short version of your design, and the one thing you'd change with more time.
- 26:13
Sample candidate
Short version. Events go into an outbox in the same transaction as the change, and a relay publishes them to Kafka, keyed by customer. Workers send each event signed with HMAC, retry it three times quickly, then hand it to retry topics that back off from a minute to an hour. Retries are rate limited per customer, so a recovering endpoint isn't flooded. Delivery is at least once, and customers dedupe on the event ID. After three days we move an endpoint's events to a dead letter table, keep them for 30 days, and they can replay. With more time, I'd load test it with a big customer down for hours, and see how far behind the retry topics get.
- 27:15
Interviewer
Thanks for your time and the work you put in. Acme's team will review the session and be in touch about next steps. Take care.