Why unstructured role-play feedback fails
Unstructured feedback fails for three reasons, and all three are predictable. First, evaluators anchor on whatever happened last — a strong close erases a weak discovery, or one fumbled objection colors the whole call. Second, without named categories, managers grade the traits they personally value; a hunter grades aggression, a closer grades the close, and neither is grading the same call. Third, "good energy" is not coachable. You cannot tell a rep to have more energy and expect a different call next week.
A rubric fixes all three by forcing the evaluator to score defined behaviors before forming an overall impression. It turns "they were great" into "discovery was a 4, objection handling was a 2, here is the clip." That is the difference between a score a rep can argue with and a score a rep can improve.
It also makes hiring defensible. When five candidates run the same scenario against the same rubric, you are comparing performance on identical criteria instead of comparing how much each person reminded you of your best rep. That is both fairer and more predictive.
The rubric: five categories, weighted to 100
| Category (weight) | Score 1-2 (below bar) | Score 3 (meets bar) | Score 4-5 (exceptional) |
|---|---|---|---|
| Discovery & Qualification (25) | Pitches before asking; questions are a disconnected checklist; never quantifies pain or confirms the decision process. | Asks before telling; uncovers a real pain and the basic decision process; questions mostly connect. | Layered, building questions; quantifies the pain (hours, cost, frequency); confirms who decides, the timeline, and the cost of doing nothing. |
| Value Selling (20) | Feature-dumps; value is generic and untethered to anything the buyer said; talks product, not outcome. | Ties value to at least one stated need; frames benefit over feature for the main point. | Every value point maps to a specific thing the buyer said; frames in the buyer's own metrics (cost, risk, time); quantifies impact. |
| Objection Handling (20) | Gets defensive or caves immediately; steamrolls the objection; matches a discount reflexively. | Acknowledges the objection before responding; answers with reasonable substance; does not panic. | Acknowledges, explores the real concern, then responds with value or a concrete plan; trades rather than concedes; stays composed under repeated pushback. |
| Closing & Gaining Agreement (20) | Never asks for a next step; ends on "let me follow up" or lets the call trail off. | Asks for a next step, but it is vague or logistics-level ("I'll send materials"). | Asks for a specific, dated, results-oriented commitment (a meeting, a trial, named next actions) and secures it. |
| Communication (15) | Rambling or robotic; talks over the buyer; fills silence with filler; loses the thread under pressure. | Clear and professional; listens; reasonable pace and structure. | Concise and confident; genuine active listening; comfortable with silence; adapts tone to the buyer and stays steady under heat. |
How to turn 1-5 into a single number
Score each category 1 to 5, multiply by its weight, and sum for a score out of 100. A 3 across the board is your hiring bar — a competent rep who does the fundamentals. A rep scoring 2s in objection handling and closing is a coaching project, not a no-hire, if discovery and communication are strong; those two are teachable faster than instinct for a question.
Resist the urge to average into a mushy middle. The categories exist so a rep can be a 5 on discovery and a 2 on closing and you can see it. A single blended number hides exactly the information a rep needs to improve. Keep the category scores visible on every scorecard, not just the total.
One rule that keeps scores honest: write one specific piece of evidence per category before you assign the number. "Discovery: 2 — asked three questions, none quantified, jumped to demo at 0:90." Evidence-first scoring is the single biggest defense against grading on vibe.
Reweighting by role
SDR / outbound. Shift weight toward the opener and communication. Bump Communication to 20 and fold a "reason for the call / earning the next 30 seconds" emphasis into Discovery. Closing here means booking the meeting, not landing the deal — keep it at 20 but score it as "secured a specific meeting."
AE / full-cycle. The default weights above are tuned for AEs. Discovery and the three selling categories carry the call. If the scenario is a late-stage negotiation or displacement, temporarily lift Objection Handling and Closing to 25 each and trim Discovery, since the deal is past first-call discovery.
Field / clinical / technical sales (device, pharma, research). Add credibility to the mix. Value Selling should reward evidence used honestly and matched to the buyer's specific case or workflow, not volume of claims. For regulated verticals, weight Objection Handling toward composure and accuracy, and make the close a small, low-risk commitment (a trial, a proctored case, a couple of appropriate patients).
Account management / customer success. Rebalance toward Communication and Objection Handling for the service-recovery scenario. Score whether the rep lets the customer vent, owns what is theirs without over-promising, and converts heat into a dated recovery plan. Closing becomes "secured a concrete follow-up commitment."
Score a real call against this rubric
Run a discovery-and-budget scenario against an AI buyer and see the rubric applied automatically. Teams get shareable scorecards for every rep and candidate.
Try a scored mock callHow to run the session: before, during, after
- 1
Before — brief and standardize
Give every candidate the same written scenario: who the buyer is, the situation, and the objective. Decide in advance who plays the buyer and how hard they push, and keep it identical across candidates. Share the rubric categories with candidates beforehand if you are coaching; keep them private if you are hiring and want to see instinct.
- 2
During — play the buyer straight, score after
The person playing the buyer should not also be the primary scorer — running the objections and grading at the same time degrades both. Play the buyer consistently: same objections, same level of resistance, same off-ramps. Take timestamped notes but hold the numeric score until the call ends so recency does not skew it.
- 3
After — score independently, then debrief
Have each evaluator score alone before anyone speaks, to avoid the loudest voice anchoring the room. Then compare category by category and reconcile gaps with evidence. Deliver feedback to the rep in the rubric's language — "discovery was a 3, here is the clip where you skipped quantifying the pain" — so it is specific and coachable.
Consistency across candidates when hiring
The whole value of a rubric in hiring is comparability, and comparability breaks the moment the scenario drifts. If the third candidate got a softer buyer or a different objection than the first, their scores are not comparable no matter how carefully you graded. Lock the scenario, the buyer's behavior, and the objections before candidate one and do not adjust mid-loop.
Rotate which evaluator plays the buyer only between candidates, never within, and keep the same panel scoring the whole slate where you can. If you must swap evaluators, calibrate them first (below) so a handoff does not reset the scale. The candidates cannot see any of this, but it is the difference between a hiring signal and a coin flip.
Calibration: score the same call together first
Before you grade anyone who matters, get your evaluators in a room and score one recorded call together. Everyone scores each category independently, then you reveal the numbers at once and argue out the gaps. You will find that "objection handling" means something different to each person until you force the conversation — one manager scores composure, another scores whether the rep won the point.
Do this until your evaluators land within a point of each other on a call they have not discussed. That is calibration: not that everyone agrees, but that the rubric produces the same number in different hands. Re-run a short calibration whenever you add an evaluator or change the scenario. Twenty minutes of calibration saves you from a hiring slate you cannot trust.
Keep a couple of "anchor" recordings — one clear pass, one clear miss — that new evaluators score as part of onboarding. Anchors turn calibration from a meeting you have to schedule into a standard everyone can check themselves against.
Where AI scoring fits (honestly)
The hardest part of running rubrics at scale is not the rubric — it is applying it the same way on the fortieth call as the first, when the evaluator is tired and the last three candidates have blurred together. That is the specific problem AI scoring solves well: it applies the same criteria to every rep and every candidate without recency, fatigue, or the halo effect, and it produces a written, evidence-backed score you can audit.
What it does not do is replace your judgment on fit, coachability, or the intangibles a manager reads in a debrief. Treat AI scoring as the consistent first pass — the layer that guarantees everyone was graded against the same bar — and keep your evaluators for the calibration conversation and the hiring call. Used that way, it removes the drift, not the manager. That is the honest boundary, and it is where the value actually is.
Give every rep the same scorecard
Run your team and candidates through scored mock calls, get shareable scorecards graded on a consistent rubric, and calibrate hiring on identical criteria.
Set up scored role-plays