11 months of 3 a.m. pages across 3 time zones — the 7-step escalation matrix that fixed our Dubai Python team

A dedicated Python team in Dubai serving European and American stakeholders looks efficient on a slide and painful in practice, because the failure mode is not technical. It is that nobody wrote down who wakes up, when, and for what. Over eleven months on one engagement we watched a six-person team lose two engineers to an unwritten on-call arrangement that everyone had agreed to informally and nobody had ever priced. Here is the seven-step matrix we built to replace it, with the numbers and the exact handover format.

Step 1 — Map the real incident clock before drawing any rota

Do not design coverage from intuition. Pull twelve months of incidents from your alerting system, convert every timestamp to Gulf Standard Time, and plot them by hour. The shape of that histogram is the only honest input to a rota, and it almost never matches what people believe.

On the engagement that prompted this article, the team was convinced the pain was overnight. The data said otherwise: 61 per cent of severity-one incidents fell between 14:00 and 20:00 GST, which is the American morning, when deployments from the US team landed on infrastructure the Dubai team owned. The 3 a.m. pages were real but rare, and they were mostly one noisy alert that had never been tuned.

Separate three counts while you are in there: incidents that required action, alerts that paged and required nothing, and incidents that were discovered by a customer rather than by monitoring. The second number sizes your alert-hygiene problem. The third number sizes your coverage gap, and it is the one to show your client.

Step 2 — Write severity definitions a tired person can apply at 3 a.m.

Three levels. Never five. A five-level scheme guarantees that a woken engineer spends four minutes classifying instead of responding, and it guarantees inconsistent classification across a distributed team.

Define each level by observable customer impact, not by which component failed. “The payment service is down” is not a severity definition, because a payment service can be down in ways nobody notices. “Customers cannot complete a transaction” is one, and any engineer on any continent will classify it identically.

Our working set: SEV1 — customers cannot complete a core action, or data is being lost right now. SEV2 — a core action is degraded or a workaround exists, or a non-core function is fully down. SEV3 — anything that can wait for the next working day in the owning time zone. Only SEV1 pages out of hours. That last sentence is the whole point of the exercise.

Step 3 — Set coverage windows around GST, not around headquarters

Dubai’s position is a genuine asset and most employers waste it. Gulf Standard Time runs four hours ahead of London and eight to eleven hours ahead of the United States depending on season and coast. A standard Dubai working day of 09:00–18:00 GST therefore covers the entire European morning and reaches the US East Coast opening.

The mistake is asking Dubai engineers to stay until 22:00 GST to overlap the US afternoon. That converts your geographic advantage into a retention problem, and it degrades the hours when the team is actually most useful. The right structure gives Dubai full ownership of the Gulf and European window, and buys the American evening from a second location rather than by extending the same people.

Write the windows as an explicit table in the engagement document, with named owners per window. Ambiguity about who owns 19:00 GST is the specific ambiguity that produces the 3 a.m. page, because when nobody owns a window everybody gets paged.

Step 4 — Define the escalation ladder with named roles and hard timers

Three rungs, each with a timer that fires whether or not a human is paying attention.

Primary receives the page and has five minutes to acknowledge. No acknowledgement, the system pages secondary automatically. Secondary has five minutes. No acknowledgement, the system pages the incident commander. Incident commander is a named individual per shift whose job is coordination and communication, not debugging — and this is the rung most teams omit, which is why their SEV1s produce excellent technical work and terrible stakeholder communication.

Two rules make the ladder function. First, escalation is automatic and blameless: a missed acknowledgement is a system event, not a performance issue, because the alternative is engineers sleeping with the phone on maximum volume out of fear. Second, the primary can escalate deliberately at any moment without justification. An engineer who knows they can hand off in one tap will engage with a page they are unsure about, instead of hesitating for twenty minutes.

For teams still assembling this capability, our walkthrough on building a dedicated Python team in Dubai covers the hiring sequence that gets you to six engineers in the first place.

Step 5 — Standardise the handover packet between time zones

Synchronous handover meetings across a four-hour gap decay within three weeks. Somebody travels, somebody has a customer call, and by month two the handover is a message saying “all good”. Replace the meeting with a fixed written packet, six fields, always in the same order:

1. Open incidents and current status. 2. Degraded but not yet an incident. 3. Changes deployed in the last window. 4. Scheduled work in the next window. 5. Alerts that fired without requiring action. 6. Who is responsible now, by name.

Fixing the format matters more than the content. A variable format forces the receiving engineer to read all of it; a fixed one lets them scan for the two fields that concern them and move on. Write the packet even when nothing happened — an empty handover is itself information, and a missing handover is an alarm.

Step 6 — Price on-call explicitly in the contract

This is the step that actually retained the team, and it is the one most engagements skip. On-call has two costs and they need two lines.

The standby allowance compensates the loss of freedom during a shift, whether or not anything happens. An engineer carrying a pager cannot travel far, cannot drink, cannot be unreachable. That constraint exists at zero incidents and it deserves payment at zero incidents.

The per-incident payment compensates actual out-of-hours engagement, with a minimum billable block. A fifteen-minute page at 3 a.m. costs the engineer their night, not fifteen minutes, and a minimum block of two hours reflects that honestly. It also gives your organisation a direct financial incentive to fix noisy alerts, which is a better forcing function than any hygiene policy.

Write both into the engagement document. Folding on-call silently into a monthly rate works for roughly two quarters and then produces a resignation, usually from your most senior engineer, who is the one being paged most often because everyone escalates to them informally.

Step 7 — Review monthly against paging data and retire noisy alerts

One recurring meeting, thirty minutes, one input: the paging log. One hard rule: any alert that paged without producing an action twice in a month is deleted or downgraded to a ticket, with no debate and no exception for alerts somebody feels attached to.

This rule is unpopular for about two months and then becomes the most valued policy on the team. On our engagement, the first three reviews removed nine alerts and cut out-of-hours pages by roughly two thirds. Nothing broke undetected in the following six months, because the alerts we removed had never detected anything — they had only ever announced things.

Track three numbers monthly and show them to the client: out-of-hours pages per engineer, the proportion of pages that produced an action, and time to acknowledge at each rung. The second number is your alert quality. The third tells you whether the ladder is working or whether people are quietly absorbing pages that should escalate.

The method at a glance

Map the real incident clock in GST before drawing anything (1). Write three severity levels defined by customer impact (2). Set coverage windows around GST and buy the American evening elsewhere (3). Build a three-rung ladder with hard timers and blameless automatic escalation (4). Standardise a six-field written handover at each boundary (5). Price on-call with a standby rate plus per-incident minimum block (6). And review monthly, deleting any alert that pages twice without producing an action (7).

Three weeks of setup for a six-person team. On our engagement, out-of-hours pages fell by roughly two thirds within a quarter, and the two engineers who had been closest to leaving both renewed. The technical work did not change at all; only the written rules around it did.

Frequently asked questions