🇦🇪 HireDeveloper.ae

Our 6 Interviewers Scored the Same Candidate 5 Different Ways: the 7-Step Training That Fixed Our Dubai Loop

Two colleagues in a focused one-to-one conversation across a meeting table, representing a trained engineering interviewer running a structured developer interview in Dubai
William

William

Talent Sourcing Expert · November 15, 2026 · 11 min read

TL;DR

  • •The method: 7 steps to turn engineers into interviewers, built around one idea: an interviewer who is asked for one signal can be trained, and an interviewer who is asked for a verdict cannot.
  • •The step that pays immediately: The shadow and reverse-shadow gate in step 3. Nobody scores alone until they have sat in on three loops and had one write-up reviewed.
  • •The honest caveat: Below roughly 25 engineers you want steps 1 to 3 and nothing else. Running the full programme on a team of eight is process theatre.

The candidate was a backend engineer with nine years of experience and we put her through six conversations. Two interviewers wanted to hire immediately. One said no with real conviction. Two wrote three sentences each that could have described anybody. The sixth wrote a page about a design decision he disagreed with and never answered the question he had been given. We spent ninety minutes in a room arguing, and what we were actually arguing about was not the candidate. It was that six engineers had six private definitions of what they were looking for, and nobody had ever told them otherwise. She took another offer while we deliberated. This is the programme I built afterwards.

Step 1: Give Each Interviewer One Signal, Not a Whole Opinion

The default loop asks every interviewer the same implicit question: is this person any good. That question cannot be trained for, cannot be measured, and produces exactly the mess I described, because six people answering the same broad question will each answer a narrower one they have invented privately.

So the first artefact is a coverage map. One row per slot in the loop, and for each slot a single named signal, the evidence that would satisfy it, and an explicit note of what that interviewer is not being asked to judge. A systems design interviewer is not scoring communication. The person running the collaboration conversation is not scoring whether the candidate can write a query. Both of them will have opinions, and those opinions go in a clearly separated section of the write-up that the decision meeting reads last.

This feels bureaucratic for about two weeks and then it becomes the thing nobody will give up, because it converts interviewing from a matter of taste into a job with a definition. If you have not built the instrument that carries these signals, that comes first: our method for the technical interview scorecard covers the document itself, and this article covers the people filling it in.

One signal per slot, and one thing each slot ignoresUntrained loopInterviewer 1: is this person any goodInterviewer 2: is this person any goodInterviewer 3: is this person any goodInterviewer 4: is this person any goodFour answers to four private questions.Nothing to train, nothing to measure.Coverage mapSlot 1: can they debug what they did not writenot scoring: design tasteSlot 2: do they reason about failure and loadnot scoring: syntax or toolingSlot 3: how do they handle being wrongnot scoring: senioritySlot 4: can they scope and cut worknot scoring: depth in our stackFour answers to four different questions,each one trainable and each one auditable.The signals above are an example set. Yours should come from the work the role actually does in your team, not from a generic competency list.Opinions outside an interviewer assigned signal still get written down. They are read last and they do not carry a score.

Step 2: Train on Evidence, Not on Questions

Almost every interviewer onboarding I have inherited was a question bank. It is the wrong artefact. Questions are the easy part, they leak, and an interviewer with a list will read the list rather than listen to the answer.

What a new interviewer actually needs is a working definition of evidence. The version I teach has four parts: what the candidate personally did, in what situation, under which constraint, and with what result they can describe. An answer carrying all four is evidence. An answer carrying none of them is an impression, and impressions are where interviewer bias lives, because an impression is mostly a report on how familiar someone felt.

The exercise that teaches this faster than any explanation takes twenty minutes. Hand a trainee three anonymised answers from past interviews, ask them to mark which parts are evidence, then compare with how an experienced interviewer marked the same three. The disagreement is the lesson. Structured interviewing has a solid public evidence base behind it, and the published guidance Google released through its re:Work programme is a reasonable free starting point to hand someone before their first session.

A concrete contrast, because this is where trainees nod and then write the wrong thing anyway. “Strong communicator, explained things clearly, would work well with the team” contains no evidence at all. It reports the interviewer experience of the hour. Compare: “Asked to explain why their migration took three attempts, she described a foreign key constraint nobody had documented, said she stopped the second attempt herself after noticing row counts drift, and named the check she added afterwards.” Same candidate, same conversation, and only the second version can be argued with by somebody who was not in the room. That is the test I give new interviewers for their own write-ups: could a colleague disagree with this, using only what I wrote down. If not, it is an impression with a score attached.

Step 3: Gate Solo Interviewing Behind Shadow and Reverse Shadow

This is the step that would have prevented my ninety minute argument, and it is the one teams skip because the panel is always short-staffed. The gate has four parts and nobody scores a candidate alone until all four are done:

  1. Three shadows. The trainee observes, says nothing, and writes a full independent debrief afterwards that is compared against the real interviewer debrief. The comparison is the entire point of the exercise.
  2. Two reverse shadows. The trainee runs the interview with the experienced interviewer in the room as a silent observer who takes over only if the conversation breaks down.
  3. One reviewed write-up. A senior interviewer marks up a debrief line by line: this is evidence, this is an impression, this sentence is a decision disguised as an observation.
  4. A named sign-off. Somebody puts their name against the trainee being ready. Shared responsibility produces no responsibility.

Two of these three shadows should be interviews that end in a rejection, which takes deliberate scheduling. Trainees who have only watched strong candidates learn to recognise excellence and nothing else, and most of the work in a loop is the middle of the distribution. If a pairing exercise is part of your loop, the dynamics are different enough to warrant a separate pass, which we cover in running a pair programming interview.

Our expert take

If you only do one thing from this article, do the gate, and do the rejection shadows specifically. In the loops we have reviewed for Dubai employers, the single most common defect is not harshness and it is not leniency. It is interviewers who cannot articulate why they rejected someone, which means the rejection was a feeling, which means it was not reviewable and may not have been fair. A trainee who has watched three experienced interviewers reason their way to a defensible no has learned something that no scorecard template transmits.

Step 4: Calibrate on Your Own Closed Decisions

Calibration sessions built on hypothetical candidates are comfortable and useless. Everyone agrees about the invented person because nothing is at stake and the details are thin.

Use your own archive instead. Take anonymised write-ups from loops closed at least six months ago, strip the decision, and have the panel score them independently before discussing. Then reveal what you did and, where you can, what happened next. The session where two senior engineers discover they would have made opposite calls on somebody you actually hired and who is now doing well is worth ten sessions of abstract agreement.

Run this quarterly, keep it to ninety minutes, and resist turning it into a retrospective on the person. The object under discussion is the bar, not the hire. A related but distinct exercise is the calibration you run on performance ratings rather than on interviews, which has different failure modes and lives in engineering performance review calibration.

Step 5: Anchor the Bar to People, Not to a Numeric Scale

Ask five engineers what a four out of five means and you will get five answers, all sincere. Numeric scales feel objective and are among the least stable instruments in hiring, because the scale has no referent outside each rater head.

Replace the definition with a person. For each level you hire at, name one engineer currently on the team whose work defines that level, and train interviewers to ask a comparison question instead of a rating question: is the evidence I just heard stronger or weaker than what this named colleague would have produced in the same conversation. It is a much easier judgement to make and a much more consistent one, because everybody has seen that colleague work.

Two cautions, since this technique has a sharp edge. Keep the anchors private to the panel, because nobody wants to learn they are the company definition of mid-level. And rotate them as the team changes, otherwise your bar quietly tracks the career of one person who may be growing faster than the role.

StepWhat it fixesEvidence it workedFailure signal
1. Coverage mapSix people answering one vague questionDebriefs stop overlappingEvery write-up still ends in a verdict
2. Evidence trainingImpressions scored as findingsWrite-ups name situations and constraintsAdjectives without examples
3. Shadow gateUntrained interviewers scoring aloneNew interviewers match experienced onesGate waived when the panel is busy
4. Calibration on closed loopsPrivate, divergent barsDisagreement surfaces before a live candidateSessions where everyone agrees
5. Named anchorsA rating scale with no referentScores stop drifting between panelsAnchors never updated
6. Drift measurementSlow decay nobody noticesFour numbers per interviewer, per quarterNumbers collected, never discussed
7. RotationPanels nobody can leaveTerms expire without a conversationOnly removals are disciplinary

Not sure whether your loop or your interviewers are the problem

Send us the last four debriefs you wrote and the decision each one led to. We will tell you whether you have a scorecard problem, a training problem or a panel that needs rotating.

Get started

Step 6: Measure Interviewer Drift Every Quarter

Interviewers drift. It is not a character flaw, it is what happens when someone does a judgement task repeatedly without feedback, and it is completely invisible until you count. Four numbers per interviewer, reviewed once a quarter, catch nearly all of it.

Score distribution tells you who has collapsed into a narrow band. Recommendation rate tells you who has become a gate and who has become a rubber stamp. Time from interview to written debrief is the most underrated of the four, because a debrief written three days later is a memory of a feeling rather than a record of evidence. And agreement with the eventual decision, read carefully, tells you who is calling it differently from the group, which is sometimes the signal that one person is right and everybody else is lazy.

Share these with each interviewer individually and frame them as calibration rather than performance. The moment drift numbers enter a performance review, people optimise them and the instrument stops measuring anything.

The four numbers, and what each one is actually telling youReviewed per interviewer, every quarter, never in a performance reviewScore spreadNarrow band meansthe scale stoppeddiscriminatingFix: re-anchorto named peoplePass rateVery high or very lowagainst the panelmedianFix: calibrationon closed loopsDebrief lagWritten days lateris a memory of afeeling, not evidenceFix: 30 min heldafter every slotAgreementPersistent outlier issometimes drift andsometimes insightFix: read thewrite-ups, not the rateThresholds are deliberately absent. With fewer than roughly ten interviews per person per quarter these numbers are noise, and acting on noisedamages trust in the panel faster than drift does. Collect them anyway, discuss them when the volume supports it.Agreement with the eventual decision measures consensus, not correctness. It is the one number that should never be optimised.

Step 7: Rotate People Out of the Loop by Default

Most panels are staffed by whoever volunteered two years ago and nobody has had the conversation since. That arrangement is bad for the people who want out, bad for candidates meeting someone who resents the hour, and politically expensive to fix one person at a time.

Set a term instead. Four quarters on the panel, then the term expires and renewal is a choice rather than a default. Rotation solves the problem that has no other solution: the senior engineer everyone quietly agrees should not be interviewing. No conversation about their shortcomings, no performance framing, just a term that ended. Keep them in the decision meeting if their technical judgement is good, which it usually is.

The second benefit shows up a year later. A rotating panel spreads interviewing competence across the engineering organisation instead of concentrating it in five people, and the day you need to run twelve loops in a month you have the bench to do it. For the surrounding process questions, there is the interview process structure method, and the internal tooling side sits in our guide to building HR management systems in Dubai.

Our expert take

The Dubai specific wrinkle in all of this is panel turnover. Engineering teams here change shape faster than in most markets, between visa-linked moves, project-funded hiring and people relocating in and out of the region, and a panel built around five long-tenured interviewers is unusually fragile as a result. Rotation reads like a nice-to-have in a stable organisation. In a team where a third of the engineers have been there under eighteen months, it is the only thing that keeps a hiring bar from being reinvented every time somebody leaves.

What the First 60 Days Looks Like in Practice

The order matters more than the speed, because three of these steps produce artefacts the later ones depend on. This is the sequence I would run in a Dubai team of thirty engineers with an existing but inconsistent loop.

  1. Week 1: write the coverage map with the panel in the room. Two hours, one page out the other end. Doing it with the panel rather than for them removes most of the resistance that shows up later, because the people who will be measured against the signals chose the wording.
  2. Week 2: pull six anonymised debriefs from closed loops. Three strong hires, three rejections. These become the training material for step 2 and the calibration set for step 4, so the work is paid for twice.
  3. Weeks 3 to 6: run the gate on two trainees only. Resist the urge to onboard the whole team at once. Two trainees generates enough shadow demand to be absorbed by live interviews without slowing hiring, and you learn where the gate is unrealistic before you have imposed it on ten people.
  4. Week 7: first calibration session, ninety minutes, closed decisions only. Expect to discover one genuine disagreement about the bar. One is a healthy result. Zero means you picked candidates everybody agrees about, which is a wasted session.
  5. Week 8: name the level anchors and start collecting the four drift numbers. Collect them quietly for two quarters before you discuss them with anyone, because the first readings are mostly measuring your new process rather than your interviewers.

Rotation, step 7, is the only piece that cannot start in sixty days, since terms need a starting point and backdating them is unfair to the people already on the panel. Announce it in week one, apply it from the next quarter, and let the first expiries happen a year out.

Where This Method Stops

Three limits worth stating before you copy any of it.

  1. Training interviewers does not fix a loop that tests the wrong things. If your interviews measure algorithm recall for a job that is mostly integration work, a perfectly calibrated panel will reliably select the wrong people. Fix the coverage map before you train anyone against it.
  2. Steps 6 and 7 need volume. Drift measurement below roughly ten interviews per person per quarter produces noise that looks like signal, and a rotation policy needs enough trained people to rotate. Both are reasonable to defer until the panel is in double digits.
  3. This is a consistency programme, not a fairness audit. It will make your decisions more defensible and more repeatable. Whether the bar itself is the right one, and whether your pipeline reaches the candidates you claim to want, are separate questions with separate methods.

Frequently Asked Questions

How much engineering time does an interviewer training programme actually cost?
Budget around eight to ten hours per new interviewer spread over a month, plus one ninety minute calibration session per quarter for everyone on the panel. The shadow and reverse shadow gate is the bulk of it, and it is mostly time that was going to be spent in interviews anyway, just with a second person in the room. The cost people forget is the write-up review, which takes a senior engineer roughly thirty minutes per trainee and is the step that actually changes behaviour. Set against that, the comparison is not with zero. An untrained panel produces loops that end in a two hour debate, candidates who withdraw because the process felt arbitrary, and the occasional mis-hire whose cost is measured in quarters rather than hours.
We are a team of eight. Is this overkill for a company our size?
Steps one, two and three are not overkill at eight people, and they matter most at that size, because a small team has no statistical cushion against a bad hire and usually no recruiter to absorb the process work. Steps six and seven are genuinely premature: you cannot measure drift with four interviews a quarter, and rotating people out of a loop that has three interviewers in it is just a staffing gap. The honest version of this method for a small team is the coverage map, the evidence training and the shadow gate, reviewed when you pass roughly twenty five engineers. Running the full programme at eight people is the kind of process theatre that makes hiring slower without making it better.
Does this work when the panel is spread across time zones and nobody shares an office?
It works better remotely than in person, with one change. Shadowing a remote interview is easier than shadowing an onsite one, because a silent observer on a video call is genuinely unobtrusive in a way a third person in a meeting room never is, and recordings make calibration sessions possible without scheduling everybody at once. The change is that written debriefs stop being optional. In an office the corridor conversation carries the signal and the written record is a formality. Distributed, the written debrief is the only artefact that exists, so the quality bar on it has to be explicit: evidence, situation, constraint, outcome, and a recommendation that can be disagreed with.
What do I do about the senior engineer everybody agrees is a bad interviewer?
Use step seven and let a term expire, because handling it as a performance conversation almost always goes badly and is rarely necessary. Interviewing well is a separate skill from engineering well, and a brilliant engineer who interrogates candidates, or who hires only people who remind them of themselves, is not failing at their job. They are being asked to do a different one without training. Published guidance on structured interviewing is worth handing them, but the practical fix is a rotation that ends without anyone having to be told they were the problem. Keep them in the decision meeting, where their technical judgement is valuable, and take them out of the room where their effect on a nervous candidate is not.

Run the loop with people who have been trained for it

We screen developers for UAE employers and sit inside your loop when you want a calibrated second opinion, including on the debriefs your own panel writes.

Get started