AI-Guided Environment Validation explored replacing four static room photos with one guided mobile video, computer-vision issue detection, candidate self-correction, and a human greeter handoff during OnVUE check-in. I led end-to-end product design for the validated proof of concept. A 70% issue-detection rate and a maximum 20% false-positive rate were experiment targets, not production outcomes.
AI-Guided Environment Validation explored replacing a clunky four-photo desktop process with a mobile-captured 360° video. The concept analyzed the scan and routed it to one of three states: green approved, yellow review, or red problem. Each state gave the greeter a specific UI and the candidate a clear path forward.
A single capture flows through AI vision analysis and is classified onto a traffic-light scale. Every state preserves a path forward for both the candidate and the greeter, including when the AI is wrong.
The triage classifier had to be honest about its own uncertainty. GREEN moves the candidate forward without human intervention; YELLOW routes everything the AI could not score to a person; RED does not block the candidate. It gives the greeter the exact problem frames and the specific failure reason. The greeter records whether the signal matched the evidence.
The existing check-in required candidates to capture four desk photos. Many candidates failed that step, which triggered time-consuming 360° scans by human greeters. That increased greeter cost, raised candidate anxiety, and slowed the pipeline. Clients wanted stronger security without spending live-agent minutes on every check-in.
The constraint that mattered most: the system had to remain candidate-tolerant. A test-taker whose camera cannot hit the focus or whose hand is not steady cannot be permanently locked out of a high-stakes credential they paid for. Persistent AI failure had to remain a recoverable state.
The usability study ran with six internal participants using their own phones. Three used Android and three used iPhone, with a deliberate mix of newer and older devices. Sessions were 45 minutes, remote, and recorded. The product was in proof-of-concept state.
The single most consequential finding: zero participants succeeded on their first scan. Every participant restarted between one and four times before completing. The scanner was too sensitive, the "you're doing great" encouragement obscured failure, and several users did not realize they had to restart at all.
"If I was making progress I'd want to try again, but I'd also want the option to indicate that I need to talk to a greeter because I'm having a problem with the scanner. I would expect that after the second time, I'd be redirected to a greeter. If they made me do it a third time, I'd start to get frustrated and think the system's broken." Three participants, independently, on max-retry tolerance
Ratings across the flow were specific. The issue-results page scored 4.1 / 5, overall experience scored 3.8 / 5, requirements scored 3.6 / 5, and the scanner scored 3.4 / 5. The scanner, which was the AI surface, rated lowest. The issue-results page, which was the explanation surface, rated highest. The research showed which part of the system was working and which part needed more work.
The model is not the AI itself. It is the policy around the AI. The state machine below governs what happens when a candidate scans, retries, or asks for help.
The model uses bounded retry, explicit human handoff, and no silent persistence failure.
After two failed attempts, the candidate can choose a greeter handoff. After the maximum number of attempts, the candidate proceeds as flagged, not blocked. This was the research-driven invariant: persistent AI failure does not own a terminal state.
I replaced "you're doing great" with explicit states such as capturing, problem detected, and restart needed. Candidates could see when the scan had failed, and motion guidance was tested through animation rather than text alone.
The single highest-rated screen in the study (4.1 / 5). Each detected problem was a labeled card with a frame of the moment, the AI's reason, and the fix the candidate could try.
The user research aligned: two retries is the tolerance window. The button does not appear on the first failure (where retry is fine) but is visible on the second (where research said frustration spikes).
The candidate's exam is not held hostage by an AI confidence threshold. Red-marked sessions surface to the greeter with the problem frames and the failure reason, ready for human judgment.
The greeter dashboard included a one-tap "AI was right" / "AI was wrong" response. This made disagreement visible in the operating workflow and preserved a structured review record.
Video length range, max retry count, AI opt-out, and the threshold for skip-to-greeter were all configurable. Different exam programs balance security and cost differently. The product had to flex.
| State | Mobile capture surface | AI engine signal | Desktop check-in | Greeter dashboard |
|---|---|---|---|---|
| GREEN | Confirms capture · returns to desktop | All checks passed | Auto-advances candidate | (no surface; routed past) |
| YELLOW | Shows "human review next" | AI could not score · opted out | Pauses for greeter | Front photo + video · greeter views & rates |
| RED · 1st fail | Issue-results page · retry CTA | Specific problem frames + reason | Holds at workspace step | (not yet surfaced) |
| RED · 2nd fail | Retry CTA + "skip to greeter" | Same problem set | Holds at workspace step | (not yet surfaced) |
| RED · max fail | Proceed CTA · "flagged" notice | Persistent fail signal | Advances candidate | Front photo + video + problem frames + failure reason · ready for greeter judgment |
| Skip · self-select | "Help on the way" state | Triage halted | Pauses for greeter | Greeter receives video + the candidate's own note about the problem |
The validated proof of concept showed that candidate-led video capture required strong choreography and that issue detection was only one part of the product. In the referenced study, the issue-results screen rated 4.1 / 5 and the scanner UI rated 3.4 / 5, giving the team a clear target for the next iteration.
The more important design contribution was a bounded recovery model that prevented persistent uncertainty from becoming a permanent candidate block. It treated AI as a signal, preserved candidate agency, and made a human exit part of the primary workflow.
An AI product earns credibility by handling errors clearly. In this kind of workflow, the recovery paths are part of the product.
The three-state classifier with bounded retry can be reused across check-in, identity, and admissions workflows.
The candidate can always reach a person. The AI does not own a terminal fail state.
Disagreement is recorded in the operational UI instead of hidden in a separate process.
The explanation surface can matter more than the capture surface. People are more willing to tolerate friction when they understand why it is happening.