Three resignations in eleven months out of nine hires. Each one, taken on its own, had a reasonable explanation: a family relocation, a competing offer we were never going to match, and one engineer who was simply not enjoying the work. Three reasonable explanations, none of which pointed at us, which should have been the first clue.
We only found the actual pattern when we stopped reviewing departures one at a time and reviewed the whole round at once. The method below is what came out of that, run four times since on hiring rounds in Dubai. It is deliberately unglamorous. It produces a document of about one page and, if it works, exactly one thing that changes before the next role opens.
Step 1: Retrospect a Cohort, Not a Hire
The instinct after someone leaves is to review that hire. Resist it. A single departure generates a story about a single person, and a story about a single person is almost always both true and useless, because it cannot distinguish bad luck from a structural flaw.
So define a cohort instead: every engineering role you filled in one window, closed, with no roles still open in it. Six to ten hires is the right size. Fewer than six and one unusual case dominates everything; more than ten and the market itself has moved underneath you, which makes the comparison muddy.
Then wait. Review the cohort once its earliest hires have passed nine months, not three. Before six months a new engineer is still inside the grace period onboarding buys, and almost nobody looks like a mistake. Review at three months and you will mostly conclude your process is fine, which is a conclusion that takes another year to disprove.
Step 2: Pull 6 Facts Before Anybody Gives an Opinion
This is the step that decides whether the meeting is worth holding. Collect the numbers first, circulate them two days before anyone sits down, and ban the meeting from running on memory. Human recall of a hiring round is reliably wrong in one direction: everyone remembers the candidates who were argued about and forgets the ones who passed smoothly, which is most of them.
Six facts per role, no more, because a longer list never gets finished:
- Days from role opening to first screen. The clearest measure of whether sourcing was ready when the role was.
- Pass rate at each stage. Candidates in and candidates out, stage by stage.
- Offer acceptance rate, with the reason for each decline. One line per decline, in the candidate words where you have them.
- Source of every hire. Referral, inbound, agency, direct outreach, returning applicant.
- Days to first merged change in production. A proxy for onboarding and for how well the stack matched what you screened for.
- Status at nine months. Still in role, moved internally, or gone, with the stated reason.
Our own table was uncomfortable in a way no individual exit conversation had been. All three departures had come through the same sourcing channel, had the longest time to first merged change in the cohort, and had each been through an interview loop that, we eventually noticed, contained no stage in which anybody read existing code. We were hiring people to extend a seven year old codebase and we tested them exclusively on writing new code in an empty file.
Step 3: Separate a Sourcing Failure From a Screening Failure
Now read the funnel, and read it before anyone proposes a change, because this is where retrospectives most often go wrong in a way that costs a whole quarter. The two failures look identical from the outside. Both end with a role that took too long and a hire you were not certain about.
If the top of the funnel is thin, you never had a decision to make. You interviewed three people, two were unsuitable, and you hired the third because the role had been open for four months. Nothing about your interview process caused that and nothing about your interview process will fix it. The question is why qualified people in this market were not applying, and the answers are usually in the brief, the compensation band or the speed of your first reply.
If the top is healthy and the loss is concentrated at one late stage, you have a calibration problem. Strong candidates are reaching a stage and failing it, which means either the bar at that stage is not the bar the role needs, or two interviewers are applying different bars. That is a fixable problem and a cheap one, and it is the one that training your interviewers properly actually addresses.
Write the diagnosis down in one sentence before moving on. If the room cannot agree on that sentence, the retrospective is not ready to discuss remedies.
Step 4: Ask the 4 Questions a Scorecard Cannot Answer
Scorecards record what you decided to look at. The interesting failures live in what you never looked at, so go hire by hire and answer four questions out loud. Twenty minutes per hire, and the cohort takes an afternoon.
- What did this role actually consist of in month three? Not the job advert. The real work, as the engineer would describe it.
- Which signal in our process predicted performance on that work? If the answer is none, you did not assess the job, and any hire that worked out was luck.
- Which stage should have caught the gap we later found? Name an existing stage or admit there is no stage for it.
- What was this person told the job was? Compare that against question one. The distance between the two is the expectation gap, and it is a brief failure, not a screening failure.
Question two is worth defending against impatience, and it should be run on your successes as well as your misses. A hire who worked out brilliantly, with no signal in your process that predicted it, does not validate the process. It warns you that your results depend on the quality of your applicant pool rather than on anything you do, which works right up until the pool changes.
Question four is where we found our second real finding. Two of the three departures had been told, accurately, that they were joining to build new product features. Both spent most of their first year on migration work that nobody had mentioned, because the migration was urgent and they were the available hands. Neither left over capability. They left because the job was not the job, and no interview loop detects a brief that becomes untrue after the person starts.
The practical defence is to write the brief from the delivery scope rather than from the org chart. If a role exists to ship a defined product, the scope and timeline of that product is a more honest job description than any advert, which is why we tend to start from a build plan such as our outline of what SaaS development in Dubai actually involves and describe the role in those terms.
If the diagnosis is the top of the funnel
A thin pipeline is not an interview problem, and rewriting your loop will not fix it. Tell us what the role genuinely requires in month three and we will shortlist pre-vetted engineers against that scope.
Get startedStep 5: Interview the People Who Said No
This is the step almost everyone skips, and in four rounds of doing this it has produced more usable information per hour than any other. Your own team can only tell you about the candidates who said yes. The candidates who withdrew, or who took another offer, know something about you that nobody inside the building can see.
Go back to every decline and every mid-process withdrawal in the cohort, by phone, and ask one specific question rather than a general one. Not what did you think of our process. Instead: at what point did you decide you were probably not joining us? People answer that question precisely, because the moment is usually a memory rather than an opinion, and it arrives with a reason attached.
Two calls rewrote our conclusions. We had assumed our acceptance rate was a compensation issue, because that is the comfortable assumption in a market where salary expectations move fast. Both candidates named something else: the nineteen day gap between their final interview and our offer. One had already accepted elsewhere by day twelve. Neither mentioned money until I raised it, at which point one said our band had been fine.
We had been preparing a case for a higher salary band. We did not need one. We needed a decision meeting scheduled before the final interviews rather than after them, which cost nothing and which we had not thought to look at because it was invisible from the inside. The same gap is often what sits underneath compressing a hiring round into days instead of weeks.
Expect roughly half of the people you contact to reply, and expect the conversation to be short and unusually frank. You are no longer in a negotiation with them, which is exactly why they will tell you.
Step 6: Convert Each Finding Into 1 Owned Change
A retrospective that ends in a list of observations has not finished. The output is at most three changes, and three is a ceiling rather than a target. We have never successfully landed more than two at once, and a document with nine action points is a document that produces nothing while looking thorough.
Each change needs three things, and a change missing any of them is still a note:
| Element | What it means | Test that it is real |
|---|---|---|
| A named owner | One person, not a team or a function | Could you send them a message about it today |
| A date | Before the next role opens, not this quarter | Is it on a calendar somewhere |
| A visible artefact | A rewritten brief, a replaced stage, a changed scheduling rule | Could a new joiner find it without asking you |
Ours were two. First, replace one greenfield coding exercise with a code reading stage against a real extract from our own repository, owned by the engineering lead, in place before the next role opened. Second, schedule the decision meeting at the point the final interviews are booked rather than after they conclude, owned by whoever was running the loop, effective immediately. The second change took one calendar invitation. It moved our acceptance rate more than anything else we did that year.
Also write down, explicitly, what you decided not to change and why. Six months later somebody will raise the same idea, and the useful record is not the list of things you did but the reasoning you had available at the time. It is the same discipline that makes a written design document process worth the overhead.
Step 7: Carry 1 Number Into the Next Round
Pick the single number each change is supposed to move, and write down its current value before the next cohort opens. One number per change, defined precisely enough that a different person would measure it the same way. Days from final interview to offer issued. Proportion of the cohort still in role at nine months. Candidates reaching the final stage per filled role.
Precision in the definition matters more than the choice of metric, and it is where this step usually fails. Days from final interview to offer sounds unambiguous until you find that one person counts working days, another counts from the interview and a third from the decision. Write the definition next to the number and keep it identical next time: two differently defined numbers compared against each other carry the authority of a measurement and point somewhere arbitrary.
Then put the next retrospective in the calendar now, dated against the cohort rather than the quarter. Retrospectives that are not scheduled in advance happen only after something goes wrong, which means they only ever examine failures, and a process you only inspect when it breaks will teach you nothing about why it works.
The Mistake I Made the First Time
I ran the first one as a blameless postmortem, borrowing the format from how we handle incidents, and I was pleased with myself for it. Nobody was criticised. Everybody spoke. We left with eleven observations and a shared sense that the conversation had been valuable.
Four months later I reread the document before opening a new role. Not one of the eleven observations had turned into anything, and several had been raised again since, as though new, by people who had been in that room. The document was accurate, thoughtful and completely inert: the good feeling had disguised the fact that nothing in it had an owner.
The second run had a hard rule: nothing leaves the meeting without a name and a date against it, and a maximum of three. It felt thinner. Two things actually changed. The number of findings is a vanity metric and the number of owned changes is the real output. If you take one thing from this method, take the ceiling of three, and treat a retrospective that produced eleven findings and no owners as a retrospective that did not happen.
Frequently Asked Questions
When should we run a hiring retrospective, and how often?
Who should be in the room for the retrospective?
How do we review hiring decisions without blaming individual interviewers?
Is early attrition always a hiring mistake?
Run step 5 this week
Call three candidates who turned you down last year and ask when they decided. If the answer points at your pipeline rather than your process, tell us the scope and we will shortlist against it.
Get started