A risk-scoring engine surfaces exam sessions for review. A reviewer has to inspect the evidence, interpret what happened, and produce a decision that holds up under audit. I defined the human-AI interaction patterns from the risk-ranked queue through the evidence canvas, structured decision, escalation, and audit trail. The shipped workflow made AI-assisted exam-video review approximately 10× faster while keeping final judgment with people.
The workstation did not sit alone. It was the human-facing surface of a pipeline that started when a client published an exam and ended in case management, result reconciliation, and finance. Before I drew a screen, I mapped the system.
The diagram below shows the six systems that touched a single session, the three concurrent review paths an exam can flow through, and the five outcome states that downstream operations had to be able to act on.
A single exam session is created in publishing, scheduled, delivered, scored by the AI risk engine, and routed to one of three concurrent review paths. Each outcome is logged for audit and reconciled downstream.
The workstation I designed lives at Path C, the post-delivery review queue. To make that surface trustworthy, every outcome it produced had to be clear to three downstream systems: case management for investigation, results / RTEN for score handling, and finance for separate reconciliation between live-proctored and AI-Assisted Review Workstation billing models. The decision log preserved a structured record of every human judgment.
The AI could flag possible exam-integrity events with high recall, but the workflow still required a human to open a case, hunt through dense session video, reconstruct context, and produce a verdict that could survive downstream scrutiny by clients, investigators, and operations.
Raw probability introduced its own risk: without visible supporting evidence, reviewers could over-trust a flag (rubber-stamping), learn to dismiss it (model fatigue), or produce inconsistent decisions (decision drift). The product needed to increase throughput without letting model confidence become the verdict.
The hard tradeoff was security versus operational cost, with each client setting the dial. Clients buying a lower-cost modality still needed real risk coverage. Clients buying live proctoring wanted almost no misconduct slipping through. The same workstation had to support both models through configuration.
What I led: I led the end-to-end design of the dashboard, risk-ranked queue, evidence canvas, event-state model, structured decision paths, audit trail, prototype, validation rounds, and implementation-ready UI. I partnered with Product, Engineering, risk-scoring partners, compliance, and the review operations team that would use the workstation every day.
I worked with reviewers, investigators, and quality partners to understand how they moved from a system signal to a defensible decision: which context changed interpretation, how they compared video moments, when they dismissed an event, what required escalation, and how consistency was evaluated. Prototype walkthroughs made the boundary between system output and human responsibility explicit enough to critique.
Three findings reshaped the interaction model.
"I liked the 'one stop shopping' aspect of having all of the details about the session displayed in one spot. But have the video jump to an event when you click it, and sync the video to the timestamp. And note the infractions on the video event list. Include why the exam was revoked, left screen, third party, etc." Reviewer walkthrough, two participants independently
The reframe was not simply a faster review. It was a defensible faster review, with human judgment preserved at every load-bearing point and a record clear enough to defend.
I designed the workflow as a four-step loop. The model narrowed attention and packaged evidence; the reviewer evaluated context and chose the outcome; the system structured the verdict; and the audit trail carried the decision into downstream operations.
Each step preserves an explicit handoff. The reviewer can override at any point. The AI never decides alone.
The loop makes the workstation accountable. Every human verdict preserves a structured record that downstream teams can interpret. The loop is bounded by four guarantees that keep the human in authority: override, no silent automation, preserved recovery paths, and audit by default.
The dashboard ranked sessions by AI-detected risk score, reviewer-assigned priority, and visible state. Reviewers spent their time where the evidence warranted it. The data-retention deadline was a visible secondary sort, so a soon-to-be-deleted session did not get dropped.
Every event in the session activity log was clickable. Clicking jumped the video to the exact moment, with context explaining why the AI surfaced it. Reviewers stopped starting at minute zero. This was the single highest-impact change in the walkthrough study.
Reasons, screenshots, detection categories, timestamps, and event metadata were grouped on a single canvas. The reviewer did not have to rebuild the case across disconnected views. Cross-referencing between events used persistent anchors.
Controlled outcomes, including COMPLETE, REVOKE, ESCALATE, and DISMISS · RATIONALE, mapped to the downstream adjudication workflow. Rationale was required for revoke and escalate decisions, creating more consistent and defensible records.
Override, escalation, revoke, complete, live-proctor support, and hand-raise paths were first-class actions because real investigations do not follow one linear route. The system tolerated mid-session correction.
Evidence reviewed, actions taken, outcomes selected, and rationale captured all remained visible so another investigator could follow the reasoning without a separate debrief. Downloads containing personal information defaulted to safer handling after privacy review.
For each session stage, the workstation had to coordinate with at least one upstream and one downstream system. The cross-system context map below is the artifact I used in stakeholder review.
| Stage | Upstream signal | Workstation surface | Reviewer action | Downstream consequence |
|---|---|---|---|---|
| Publish | Client sets delivery mode + cost-vs-security threshold | Not applicable | Not applicable | Capacity queue + finance reconciliation routed per modality |
| Schedule | Capacity queue allocates greeter / proctor / reviewer capacity | Not applicable | Not applicable | Live-agent capacity and hand-raise handling aligned |
| Deliver | Browser lock + secure browser config + monitoring stream | Optional: live alert into proctor queue | Revoke or clear-&-dismiss (Path A) | Result + case routing if revoked |
| Score | AI risk engine produces SmartScore + reasons | Session lands in ranked queue with reasons attached | Not applicable | Retention deadline starts ticking |
| Review | Session video + activity log + AI events | Evidence canvas with event-to-timestamp sync | Add events · escalate · revoke · complete | Verdict + rationale persisted to audit log |
| Adjudicate | Verdict from workstation | Read-only history view for downstream investigators | (Investigator continues case) | Case management opens · client notified · result held |
| Reconcile | Verdict + outcome state | Not applicable | Not applicable | Finance reconciliation by modality · decision audit preserved |
The table became the design artifact. Once every row was filled, I knew which screens I had to build and which ones belonged to adjacent teams. The rows with empty workstation surfaces, including Publish, Schedule, and Reconcile, showed where I needed alignment rather than new screens.
Prototype walkthroughs and usability feedback exposed specific issues that became specific design changes before production. Validation focused on whether reviewers could explain why a session appeared, reach the relevant evidence, correct an event decision, and complete or escalate without losing context.
Manager reporting, reviewer, date, and client filters, and broader activity visibility were identified as useful future-state opportunities and are not presented as shipped functionality. Implementation review kept the evidence hierarchy and decision logic legible through delivery.
The shipped workflow made AI-assisted exam-video review approximately 10× faster. The durable contribution was the decision architecture: system evidence remained legible, reviewers could disagree, escalation was a designed path, and final judgment stayed human without hiding uncertainty.
Structured outcomes also created a consistent audit trail. Each reviewer decision preserved evidence, rationale, ownership, and downstream status so later teams did not have to reconstruct the case.
Calibrated transparency builds more trust than performed certainty. In trust-sensitive products, the goal is not only a faster decision. It is a faster decision with evidence, uncertainty, and rationale that remain understandable later to people who were not in the room.
The combination of a risk-ranked queue, evidence canvas, and structured verdict is the pattern I use when a human has to act on AI output under audit.
Show the reason, the alternative, and the override path. Do not hide model confidence behind a verdict.
Override, escalate, revoke, and complete are not edge cases. They are designed alongside the happy path, not added later.
Every human decision preserves its evidence, rationale, owner, and downstream consequence.