The failure that taught me this was embarrassingly ordinary. A client portal, a marketing push, about 400 people on it at once, and the whole thing went to treacle and then to timeouts. Every dashboard was green, because every dashboard measured averages. The median request was fine. The ninety-fifth percentile had been degrading for three weeks and nobody owned the number. We did not have a tooling problem. We had a missing role, and I had been hiring around it by adding backend engineers and hoping.
This is the method I use now. It is written for an employer in Dubai hiring their first or second performance and load testing engineer, and the order matters more than any individual step, because the brief determines which candidates ever reach you.
Step 1: Write the Brief From a Failure You Can Name
Most requisitions for this role begin with a sentiment: we want better performance, we want confidence before launch. Sentiment attracts generalists. Anchor the brief instead in one incident with a number attached. Ours became: the portal degrades past roughly 400 concurrent sessions, we do not know which dependency saturates first, and we need to be able to state a safe concurrency number before the next campaign.
That sentence does three jobs. It tells a strong candidate exactly what the first month looks like, which is the most persuasive thing you can put in an advert for a scarce skill. It makes clear you want an answer rather than a ritual. And it filters out candidates whose experience is running a vendor tool to produce a report nobody acts on.
If you have no incident yet because you are hiring ahead of a launch, anchor on the decision instead: before March we must be able to state the maximum sustained throughput of the checkout path and the cost of doubling it. A decision is as good an anchor as a failure. Vagueness is the only fatal option.
Step 2: Separate the Three Profiles That Answer the Same Advert
This is the step that saves the most wasted interview time. Three quite different engineers reply to a performance testing advert, and they are not interchangeable.
| Profile | What they are genuinely good at | Hire them when |
|---|---|---|
| Script writer | Building and maintaining test scripts in a given tool, running them on a schedule, reporting results | You already know your load profile and the bottleneck, and need the suite maintained and rerun reliably |
| Performance engineer | Finding why it is slow: profiling, query plans, connection pools, garbage collection, cache behaviour | You have a failure you cannot explain, which is the most common situation |
| Capacity planner | Turning measurements into forecasts and cost: how much headroom, what it costs to double, when to scale | You are making an infrastructure or budget commitment and need a defensible number |
The second profile is the one most Dubai employers actually need and the one most adverts fail to describe, because the advert gets written around tools. An advert listing four load testing tools and no diagnostic skills reliably brings you the first profile. If your Step 1 brief contains the words “we do not know which dependency saturates first”, you need the second, and the advert has to say so in those terms.
Our expert take
If you can only hire one, hire the performance engineer. Scripts can be learned on the job by a competent engineer in weeks, and capacity forecasting is mostly arithmetic once the measurements are trustworthy. Diagnosis is the part that takes years to build and cannot be improvised under launch pressure. I have watched a strong diagnostician produce a usable answer with a crude homemade test harness, and I have watched a tool specialist produce a beautiful report that pointed at the wrong subsystem for a fortnight.
Step 3: Set the Band Against Backend, Not Against QA
This is where most searches in this market quietly die. The role has the word testing in it, so it gets budgeted from the QA line. But the people who can diagnose a saturation point are backend and reliability engineers who specialised, and they are priced accordingly. A QA-priced advert does not attract them at a discount. It simply never reaches them, and you spend six weeks interviewing the first profile from Step 2 and concluding the market is thin.
Set the band from your own data rather than a published survey. Take the last six months of offers you actually made for mid-level and senior backend engineers in Dubai and place this role in the same range, with the senior end reserved for someone who can own a capacity statement and sign off a launch. If you also need infrastructure work, expect the top of your range, because that combination is scarce here.
Publish the band. The strongest candidates for this role are employed and approached regularly, and a range is what gets a reply to a cold message. For the mechanics, our method for creating a developer salary benchmark in Dubai applies without modification.
Step 4: Source Where the Evidence Is Public
Keyword matching on job boards works badly here, because the keyword is a tool name and tool names are the least informative thing on a CV. The useful signals are places where someone showed their reasoning in public.
- Incident write-ups and engineering blogs. Anyone who has published a readable post-mortem that identifies a saturation point and explains the measurement has demonstrated the exact skill in the exact register you need.
- Conference and meetup talks. Regional and online talks on performance, databases and reliability. Speakers are findable, and the talk tells you how they think under questioning.
- Open-source test harnesses and tooling. Contributions to load generation tools, profilers or benchmarking suites are a strong signal, and far rarer than contributions to application frameworks.
- Tool and database communities. People answering hard questions about connection pooling or query planning in public forums are, almost by definition, the second profile.
- Internal referrals. Ask your senior backend engineers who they call when something is slow and they cannot work out why. That produces better names than any search string.
Write to these people about your Step 1 failure, not about your company. A message saying “our portal degrades past about 400 concurrent sessions and we cannot work out which dependency gives first” gets answered by exactly the person who cannot resist a diagnostic puzzle. A message about a market-leading platform does not.
Short on diagnostic candidates in Dubai
We shortlist pre-vetted engineers against a named failure rather than a job title, including performance and reliability specialists.
Get startedStep 5: Screen With 4 Questions That Expose Tool Operators
Thirty minutes and four questions will tell you which profile you are talking to. I ask them in this order and I write down the answers verbatim, because the differences between candidates are in the wording.
- “Tell me about a load test that led you to the wrong conclusion.” The best answer in the whole interview is a specific story, told without defensiveness, about a test whose traffic shape did not match reality. Candidates who have never been misled by their own test have either not run many or never checked against production.
- “Why would I care about the ninety-ninth percentile if the average is fine?” This is deliberately easy, and it is diagnostic anyway. A strong candidate goes immediately to the shape of the distribution and to who experiences the tail, and usually offers an unprompted example about queueing or retries amplifying it. A weak candidate recites a definition.
- “How do you decide what test data to use?” Underrated and highly separating. Realistic data volumes and cardinality are where most load tests silently become meaningless, and people who have been burned by it raise distribution, cache behaviour and privacy constraints unprompted.
- “Tell me about something you refused to sign off.” The role exists to say no on evidence. Someone who has never declined to certify a launch is either lucky or has been a report generator. I am listening for the pressure they were under and what evidence they held on to.
Our expert take
The fourth question is the one I would keep if I had to drop three. A performance engineer who cannot say no is worth very little, because the entire commercial value of the role is the credibility of the statement “this will hold”. If that statement is never withheld, nobody has reason to believe it when it is given. I have hired on the strength of that answer alone and not regretted it.
Step 6: Run One Paid Exercise Against Your Own System
Take-home exercises on toy systems do not work here, because the entire skill is reading a real system with real dependencies. Scope a short paid exercise instead, two to four hours, against your own staging. Give them access, the Step 1 question, and nothing else. Pay the market rate and say so upfront, which is also a competitive advantage because few employers do it.
Grade the written findings, not the script. Five things: did they ask what real traffic looks like before writing anything, did they state assumptions, did they report distributions rather than averages, did they name which dependency saturated first and how they know, and did they write down what the test does not prove. That last one is the strongest single predictor I have found. The best submissions are often shorter than the worst and considerably more hedged.
Do not skip the staging access because it feels like a security hassle. If a candidate cannot get read access to staging for an afternoon, you have learned something about your own process that will also slow this hire down in week one. If you would rather establish a baseline before hiring at all, an external performance testing engagement is a reasonable way to get a harness in place first.
Step 7: Prove the Hire in the First 30 Days
Agree the first month in writing before they sign, and keep it to three deliverables. A baseline: current behaviour of the critical path under a defined load, in percentiles, documented well enough that someone else could rerun it. A reproducible test that runs from one command and produces comparable numbers each time. And one defensible capacity statement, in the form “the checkout path sustains X with latency under Y, and the first thing to give beyond that is Z”.
Three deliverables is deliberately modest and a good hire usually beats it. The point is that each is verifiable by a non-specialist, which protects you against the failure where six months of activity produce no number anyone will stake a launch on. If at day 30 you still cannot answer the Step 1 question, raise it then rather than at the probation review.
After the first month the role compounds in two directions, both easier to staff once a baseline exists: into the automated suite, where our method for building a QA automation team in Dubai takes over, and into production reliability, covered in hiring a site reliability engineer in Dubai. Decide which direction you want before the first hire lands, because it determines which adjacent role you open next.
What This Costs, Honestly
Three line items employers discover late. The paid exercises, across three finalists, a small cost and the highest-return money in the process. Making staging load-bearing enough to be worth testing, which is often a few weeks of the new hire’s first month. And load generation itself, which costs real money at the volumes that matter and appears in nobody’s requisition.
Against that, what this role protects is commercial rather than technical: the ability to open a campaign, sign a client with a volume clause or approve an architecture spend and know what will hold. That is worth considerably more than the band, and it cannot be retrofitted during an incident.
Frequently Asked Questions
What is the difference between a QA engineer and a performance and load testing engineer?
What should I pay a performance and load testing engineer in Dubai in 2026?
Should I hire this role in-house or use an external performance testing engagement?
How do I test this skill in an interview without a week of candidate time?
Run this method with a shortlist already in it
Tell us the failure you want answered and the band you can fund. We come back with pre-vetted engineers who can answer it, and we will tell you plainly if the brief needs rewriting first.
Get started