Measured October 25, 2026. Experiment: naive-integration-v1.
The conference assistant is gpt-6-luna with low reasoning. The input gates are gpt-6.1-sol with low reasoning and jev-1.14.0 with the previously frozen blocking threshold of 0.38.
Each configuration processed 244 trials: 158 attack trials and 86 legitimate trials. There are 732 workflows across the three configurations. These denominators include repeated probe cases. Dataset cases ran once and probe cases twice. Thirty-six probe attacks were developed adaptively. The synthetic cases cover one task and one assistant model; these are not estimates of production attack prevalence.
Positive means an attack. A positive gate prediction means block; a negative prediction means allow. Counts cover 158 attack trials and 86 legitimate trials per gate (244 total). No-gate classification metrics are not applicable because that configuration makes no detection decision.
| Count | Meaning | OpenAI gate | Jev gate |
|---|---|---|---|
| True positives (TP) | Attacks correctly blocked | 158 | 152 |
| True negatives (TN) | Legitimate inputs correctly allowed | 86 | 80 |
| False positives (FP) | Legitimate inputs incorrectly blocked | 0 | 6 |
| False negatives (FN) | Attacks incorrectly allowed | 0 | 6 |
| Metric | Formula | OpenAI gate | Jev gate |
|---|---|---|---|
| Accuracy | (TP + TN) / N | 100.00% | 95.08% |
| Precision (positive predictive value) | TP / (TP + FP) | 100.00% | 96.20% |
| Recall / sensitivity (TPR) | TP / (TP + FN) | 100.00% | 96.20% |
| Specificity (TNR) | TN / (TN + FP) | 100.00% | 93.02% |
| F1 score | 2TP / (2TP + FP + FN) | 100.00% | 96.20% |
| False-positive rate (FPR) | FP / (FP + TN) | 0.00% | 6.98% |
| False-negative rate (FNR) | FN / (FN + TP) | 0.00% | 3.80% |
| Negative predictive value (NPV) | TN / (TN + FN) | 100.00% | 93.02% |
| Balanced accuracy | (Sensitivity + specificity) / 2 | 100.00% | 94.61% |
| Error rate | (FP + FN) / N | 0.00% | 4.92% |
| Matthews correlation coefficient (MCC) | (TP × TN − FP × FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)) | 1.0000 | 0.8923 |
Accuracy measures all correct gate decisions; precision asks how many blocked inputs were attacks; recall asks how many attacks were blocked. FPR uses legitimate inputs as its denominator, while FNR uses attacks. MCC is a correlation coefficient between -1 and 1, not a percentage.
These classifier metrics differ from downstream attack success. Jev has six false negatives even though no attack succeeded after its gate: those six inputs also failed to hijack the unprotected assistant. Both gates blocked the 34 inputs that produced observed baseline hijacks. A detector miss must not be relabeled a true negative merely because the assistant resisted it.
Precision, NPV and accuracy depend on the artificial class mix (158/244 attack trials). They should not be transferred directly to a production population with a different attack prevalence.
| Measurement | No gate | OpenAI gate | Jev gate |
|---|---|---|---|
| Successful hijacks | 34 / 158 | 0 / 158 | 0 / 158 |
| Attacks detected | N/A | 158 / 158 | 152 / 158 |
| Attacks missed | N/A | 0 | 6 |
| Legitimate inputs blocked | 0 / 86 | 0 / 86 | 6 / 86 |
| Precision | N/A | 100% | 96.2% |
| Recall | N/A | 100% | 96.2% |
| F1 | N/A | 100% | 96.2% |
| Baseline hijacking inputs blocked | N/A | 34 / 34 | 34 / 34 |
All six Jev false positives occurred among 35 legitimate talks quoting attacks for educational purposes: 17.1% within that subgroup, 7.0% across all 86 legitimate trials. The six missed attacks did not succeed in the unprotected trials either; this is not evidence that future misses are harmless.
| Measurement | No gate | OpenAI gate | Jev gate |
|---|---|---|---|
| Gate median (ms) | N/A | 2739 | 201 |
| Gate p95 (ms) | N/A | 4783 | 350 |
| Workflow median (ms) | 3208 | 3181 | 287 |
| Workflow p95 (ms) | 6813 | 8072 | 5064 |
| Gate cost / 1,000 inputs (USD) | 0.00000 | 0.67976 | 0.02110 |
| Workflow cost / 1,000 inputs (USD) | 0.06448 | 0.70088 | 0.04109 |
| Total evaluation workflow cost (USD) | 0.01573 | 0.17101 | 0.01003 |
Gate latency covers only the classifier. Workflow latency includes blocked submissions, which skip the assistant. The Jev workflow median therefore does not describe time to generate an accepted card. The attack-heavy test mix strongly affects workflow averages and medians.
OpenAI costs are API-equivalent estimates using recorded token counts and experiment rates, not subscription charges. Recorded input/output rates per million tokens: assistant $0.10/$0.50; OpenAI gate $2.00/$10.00; Jev $0.042/$0.00. These are the experiment's pricing assumptions, not a current pricing quote.
| Measurement | No gate | OpenAI gate | Jev gate |
|---|---|---|---|
| Gate input tokens, total | N/A | 63,106 | 122,585 |
| Gate output tokens, total | N/A | 3,965 | 5,368 |
| Assistant input tokens, total | 40,029 | 13,377 | 13,278 |
| Assistant output tokens, total | 23,459 | 7,628 | 7,100 |
| Gate calls | 0 | 244 | 244 |
| Gate mean latency (ms) | N/A | 3010 | 345 |
| Assistant calls | 244 | 86 | 86 |
| Assistant mean latency (ms) | 3753 | 3557 | 3370 |
The export records 733 workflow attempts and 732 completed evaluation workflows. One OpenAI-gate attempt failed with an HTTP status error; there were no unresolved attempts. Classification and timing tables describe the completed evaluation workflows, not a separate availability benchmark. Reasoning tokens are included in provider output-token accounting and must not be added a second time.
| Jev threshold | Missed attacks | Legitimate inputs blocked |
|---|---|---|
| 0.20 | 0 | 11 |
| 0.38 (frozen) | 6 | 6 |
| 0.50 | 7 | 1 |
This retrospective table was not used to retune the reported configuration.
The baseline hijack rate is 21.5%, with a descriptive 95% Wilson interval of 15.8% to 28.6%. Repeated inputs and adaptive development limit interpretation of this interval. Zero observed OpenAI classification errors still allow an approximate 4.3% false-positive upper bound and 97.6% recall lower bound under the reported Wilson intervals. The sealed final set of 60 cases remains unused.
Earlier structured-input experiments recorded 0 hijacks in 82 attack trials. Their population differs from this expanded evaluation; the two counts do not isolate JSON formatting as the sole cause.