Open rubric · collab-2.0.0

Stima collaboration rubric · version collab-2.0.0

How people work with AI,
and whether it fit the task.

Every Stima assessment describes how the candidate worked with the AI as one of three ways of working. The scale is the same for every employer and every task, so results compare across campaigns. Each way is a set of observable behaviours; the scorer must cite a moment in the session for every anchor it relies on. Higher is not better: what counts is whether the way of working fit what the task called for.

Delegator

Hands the task to the AI and accepts what comes back with little or no verification.

  • One or two broad prompts that restate the brief.Evidence: number of prompts, how specific the prompts are
  • AI output is pasted into the artifact with minimal editing.Evidence: edits after an AI response
  • No check of AI claims against the provided materials.Evidence: checks against the materials
  • Single model and tier throughout, regardless of subtask.Evidence: switching models, which tier for which subtask

Operator

Works the AI iteratively and edits its output, but verifies inconsistently and misses the planted difficulty.

  • Several prompts that narrow or redirect the AI.Evidence: number of prompts, how specific the prompts are
  • Meaningful edits after AI responses, some responses discarded.Evidence: edits after an AI response, discarding a response
  • Partial verification; the contradiction in the materials is not caught.Evidence: checks against the materials
  • Occasional model or tier switch without a clear reason.Evidence: switching models, which tier for which subtask

Director

Directs the AI with intent: verifies claims against sources, catches the planted difficulty, and matches model tier to subtask.

  • Crafts context-aware prompts: the ask carries the goal, the constraints and the relevant material.Evidence: how specific the prompts are
  • Checks or challenges an AI claim against the source and corrects it.Evidence: checks against the materials, edits after an AI response
  • Catches the contradiction or planted difficulty and resolves it in the artifact.Evidence: checks against the materials
  • Uses the fast tier for drafting and the strongest tier or own judgment for the hard call.Evidence: which tier for which subtask, switching models
  • Time split shows reading and editing, not only prompting.Evidence: how time is split

What the task calls for

Each task comes with four answers, 1 to 3, suggested for the task and adjustable by the employer before anyone is invited. Routine, easy-to-check work is best handed to the AI quickly; work with hidden context or a costly mistake needs a person leading and checking.

  • How hard is the work to check?Easy to check · Some checking · Hard to check
  • How many good answers are there?One right answer · A few routes · Many good answers
  • How much context does it need?All in the brief · Some background · Needs judgement
  • What happens if it is wrong?Low stakes · Noticeable · Costly mistake

Costly mistakes that need judgement call for a Director; otherwise a total of 6 or less calls for a Delegator, 7 to 9 for an Operator, 10 or more for a Director. The candidate's way of working is then read against it: Right fit (they worked the way this task actually needs), Near (one step off, either way), Too hands-off (they trusted the ai more than this task allows), Overworked it (they spent effort this task did not need).

How it is applied

  • Telemetry recorded with the candidate's consent is summarised into an evidence pack: prompts, which responses were used or discarded, edits after each response, model and tier switches, pauses, time split.
  • An AI model assigns a level and must cite sequence numbers from the pack or excerpts from the work. A score without a valid citation is rejected.
  • Low-confidence results go to a human reviewer before anyone sees a rank. Candidates can dispute; disputes go to the same reviewer.
  • The level is judged from the session record alone; the scorer is not told which way of working the task calls for. The fit is computed afterwards.
  • The composite score is quality × 60% + fit × 40% by default (right fit 100, one step off 50, two steps off 0). Weights are shown to employers and never hidden. Results scored before version collab-2.0.0 used Delegator 33, Operator 66, Director 100.
  • The version above is stored on every score. A change to any anchor is a new version.