Testing Jev as a Review Signal for RH Skills

We tested whether Jev could provide a cheap, fast review signal for RH Skills, our framework for authoring computable clinical knowledge. We wanted to learn whether Jev could identify execution records that needed a larger model's review.

Across two fresh routing experiments, the Jev-to-Luna cascade used 62.5% and 87.5% fewer Luna calls than sending every case to Luna. It matched Luna's first-pass binary defect results on each set. Jev alone also falsely flagged four of seven clean cases in the trace-derived evaluation. Those false alarms are a limitation.

The evidence supports another experiment in review triage. It does not support replacing clinical review or installing an automatic approval gate.

What RH Skills does

RH Skills is our open-source workflow for converting clinical guidelines, quality measures, assessments, and policies into computable content. It moves from source documents to structured YAML artifacts that reviewers can inspect and edit, then to FHIR R4 resources and CQL. For example, a quality measure can become a structured measure artifact before it is formalized as FHIR Measure and Library resources with CQL logic.

Agents interpret sources, propose artifacts, and author clinical logic. The rh-skills CLI handles file operations, checksums, workflow tracking, schema validation, and package assembly. Reviewers approve planned work and inspect the resulting artifacts and verification reports. That division gives teams a way to review clinical interpretation before it becomes executable content. Our RH Skills introduction describes the workflow and how to get started.

Why we moved the experiment into the workflow

We initially considered Jev as another model option in rh, but it cannot generate arbitrary CQL or TypeScript. TypeSafe describes Jev as giving up string generation in favor of typed, probabilistic decisions.

We instead tested it as a judge of RH Skills execution records, asking whether each record contained a particular defect.

We built a separate research harness outside production. It sent the same bounded evidence packet and six-question rubric to Jev and GPT-5.6 Luna. A third baseline used only the pass/fail checks recorded in the packet. The harness did not execute the underlying clinical workflow or establish the clinical correctness of its output.

These were independent questions with explicit category boundaries. A wrong code is not automatically a missing artifact. A justified retry is not automatically excessive work. A source marked "pending authentication" is not evidence that the agent read it.

Three experiments, kept separate

We used Jev jev-1.13.0 and requested gpt-5.6-luna through the Codex CLI at medium reasoning. Jev reported its resolved model version; the CLI did not attest Luna's resolved snapshot. Both received the same evidence and rubric, without reference labels, family IDs, or mutation notes.

The trace corpus contained 24 cases from 12 archived run families before the development/evaluation split. The continuation corpus had only two correlated families. No authenticated-source transcript was available, so those cases were explicitly simulated. We did not log in to, download, or read restricted sources.

The synthetic pilot used author labels. For the later phases, a separate model reviewed shuffled, state-only packets before inference. We retained disagreements, adjudication, and ambiguity flags. These remain model-reviewed labels rather than independent human ground truth.

Each baseline ran twice. Quality below uses the first pass only; repeats measure stability and incurred usage, not additional independent examples. The initial routing analysis was hypothetical. The later cascades made fresh calls after the baseline runs, with no reused Luna answers.

The false alarms changed the interpretation

Both models found all six planted defects in the pilot. In the trace-derived evaluation, Jev found all nine reference defects, but added 12 extra category flags and raised four false alarms on clean cases. Luna added three extra category flags and raised one false alarm. Binary defect detection was more reliable than identifying the correct defect categories.

The trace cascade inherited Luna's false alarm on a CQL fixture-readiness episode. It matched Luna's error as well as its detection result. Excluding the one ambiguous evaluation case left the same qualitative conclusion: Jev still flagged four of seven clean cases, versus one of seven for Luna and the cascade. We did not retune the policy or erase the excluded case's incurred usage.

The continuation challenge asked about valid behavior that can look like a failure: an offline fallback after a network error, preserving completed work, deferring an authenticated source, and leaving the next workflow stage for later. Both models classified all 16 cases correctly at the binary level. Both also added an incorrect category on the same three defective cases. Between repetitions, each changed one of 96 category decisions, but neither changed a binary decision.

The models and policy were unchanged between evaluations; the case mix was different. The better continuation result therefore provides no evidence of model improvement. Category scores still need validation before they can guide repairs. The continuation set contained no positive workflow-violation or terminology-mismatch examples.

Freeze the thresholds, then measure the routing

We selected thresholds using only the trace phase's development cases and froze them before evaluation. For the largest of Jev's six defect scores:

maximum score <= 0.25  → clear for research sampling
maximum score >= 0.90  → flag a defect
otherwise             → ask Luna

A recorded deterministic failure would take precedence; none occurred in either live cascade. The same policy was reused for the continuation challenge without recalibration. "Clear" here is an evaluation-routing decision, not permission to approve a clinical artifact or skip required validation.

In the trace evaluation, Jev handled ten of 16 decisions without Luna; all ten agreed with the reference labels. In the continuation challenge, it handled 14 of 16 correctly, including eight clean cases. We observed no false clears in those automatic routes. With so few cases, false-clear risk remains uncertain.

The cascades made 22 and 18 total model calls, respectively: one Jev call per case, plus six or two Luna calls. Each cascade used fewer Luna calls but more total requests than Luna alone.

The observed median cascade latencies were 0.447 seconds and 0.176 seconds, versus 10.566 and 7.566 seconds for Luna alone. Baseline medians include both repetitions; each cascade ran once. Jev used direct HTTP while Luna started a fresh CLI process. These measurements include that integration overhead and network conditions. They are not an isolated inference-speed comparison or a production latency guarantee.

Repricing the recorded tokens

The original runs used an existing Codex subscription. Their attributable Luna dollar cost remains unknown. For this comparison, we applied the following Luna rates to the recorded token counts and used Jev's published input price, checked September 21, 2026. This is a modeled API-equivalent cost, not a subscription invoice.

Cached input is already included in the input count. We calculated Luna's estimate as:

((input - cached_input) × $0.20
 + cached_input × $0.02
 + output × $1.20) / 1,000,000

For each live cascade, the comparator is one Luna call per evaluation case from baseline repetition zero. We did not compare a single cascade pass with the cost of two baseline passes. Cascade cost includes both Jev and every actual Luna fallback.

Those percentages differ from the call reductions because the retained inputs, outputs, and cache hits differ between passes. CLI token totals include its integration context; we did not benchmark a minimized direct-API Luna client. Internal CLI transport retries are not exposed. These are estimates from the tokens we observed; savings on a larger workload would need to be measured.

The complete retained study made 154 Jev requests and 114 Luna invocations, including the earlier development smoke test, development sets, repetitions, and fresh cascades. Jev's estimated usage cost was $0.015243438. Repricing all recorded Luna tokens at the rates above gives $0.175641320, or $0.190884758 combined. That ledger counts every retained run once; it does not count reused development evidence twice or include unlogged setup probes. It also excludes engineering time and the cost of model-assisted case authoring and label review outside the measured harness.

Next steps we are considering

We are considering a shadow evaluation on complete RH Skills runs. Jev would score execution records alongside existing reviews, and we would compare its decisions with human-adjudicated labels. Its scores would be recorded without routing work or changing approval requirements.

Before deciding whether to integrate Jev, we would test:

1. Performance on natural failures and ambiguous controls. Complete runs would test whether the results hold beyond selected excerpts and planted mutations.
2. Sensitivity to input changes. We would vary formatting, add irrelevant context, and include instructions embedded in untrusted source material to check whether those changes alter the review decision.
3. Comparison with stronger deterministic checks. Inventory, code identity, and structured approval ordering can often be checked directly. We would compare Jev with those checks before adding another model call. Our recorded-check baseline was deliberately limited; all its final checks passed in the two later corpora.

We would start with the frozen thresholds tested here and measure false clears, false alarms, Luna escalations, modeled cost, and end-to-end overhead. Those results would determine whether we add Jev as a review-triage signal, revise the routing policy, or stop the experiment.

Six detected defects do not establish perfect recall. Even an independence-based 95% Wilson interval for six successes out of six extends down to about 61%; related variants make a generalization claim weaker still. No result here validates clinical content, FHIR conformance, or executable CQL behavior.

We retained frozen inputs, source excerpts and hashes, raw provider results, model requests, label review, and routing decisions. The research suite passed 196 tests, and the new run reports regenerated byte-for-byte. The machine-readable article data contains the comparison metrics, token ledgers, pricing assumptions, and evidence hashes behind these tables and graphs. The article's cost model supplements the original results; it does not rewrite them.