How we measure rule quality

Every rule ships with a measurable quality claim. The numbers below come from the Interlace Linter Benchmark (ILB) — a reproducible suite that runs against real open-source repositories, hand-labeled ground-truth fixtures, and adversarial edge cases.

The words we use, in plain terms

Every number on this page depends on these, so they are defined once rather than assumed.

Severity — what ESLint does when a rule fires. You set it per rule in your config; our presets set it for you.

That is why a severity is a promise: error interrupts you and warn does not.

The four outcomes.

• TP (true positive) — the rule fired and the code really was vulnerable. Signal.
• FP (false positive) — the rule fired and the code was fine. Noise.
• FN (false negative) — the code was vulnerable and the rule stayed quiet. A miss.
• TN — the code was fine and the rule stayed quiet. Correct silence.

Precision = TP / (TP + FP) — of the findings you are shown, what share are real? This is the number that decides whether you keep the tool on. Recall = TP / (TP + FN) — of the real problems, what share did we catch? You cannot maximise both; see the tradeoffs below.

Sampled vs fixture precision. Fixture precision is measured on test files we wrote — a regression gate that says nothing about your code. Sampled precision is measured on open-source repositories we did not write. Only the second predicts your afternoon, and it is the only one the bars use.

The severity contract

A severity is a promise about your triage time, and it is the part of a lint finding you actually plan around. Nobody in this space publishes a precision bar, so we set ours and publish where we sit against it.

Three rules follow, and they bind us more than they bind you:

1. A rule may not ship at a tier it has not been measured against. Unmeasured means opt-in, however good it looks on our own fixtures.
2. The sample is for that rule specifically, on code we did not write, each finding labelled TP / FP / undecidable with a reason. A rule's precision measured inside a mixed sample of several rules is not that rule's precision.
3. A rule that misses its bar is a defect, and gets fixed. Demotion is the stopgap that protects you today — no-unsafe-deserialization went error → warn on 2026-05-09 at 76% Wild hits, locked by a test — but a demoted rule is an open defect, not a resolved one. Rules below their bar block new rule work: ten rules that hold their promise beat a hundred that do not.

When a rule is too rare to score

A ratio needs findings to compute. detect-non-literal-fs-filename produces one finding across 3.04M lines of mature open-source code, and that finding is real. No sample size turns that into 70%. A rule can be correct and still be unscoreable by a ratio.

A rule whose entire output is smaller than the n its tier requires is scored by census instead — and it is stricter, not a waiver:

Sampling lets a rule hide its worst findings behind the ones you drew. A census cannot — one false positive anywhere fails it. A rule does not choose which criterion applies; the finding count decides, and the count is published beside the verdict.

How big a sample the bar needs

n is the sample size — how many individual findings we pulled out of real repositories and read one at a time, labelling each true positive, false positive, or undecidable. It matters because a percentage from a small n is mostly luck: at n = 4, reclassifying one finding moves precision 25 points.

A percentage from 20 findings is not evidence for a 95% claim: read 20 findings, find all 20 real, and the true rate could still be 85% on a good draw. So the bars carry a required n, derived from the Wilson score interval — the standard statistical answer to "given what we saw, what is the worst the true rate plausibly is?" at 95% confidence. That worst case is what we hold ourselves to, not the raw percentage. With a perfect sample:

So the error bar needs n ≥ 73 with zero false positives, or n ≥ 150 with one. n ≥ 20 is enough to publish a rate, never enough to clear the error bar. We got this wrong when the contract was first written and are saying so rather than quietly restating it.

What these bars cost

A 95% error bar means very few rules will ever be error. Most coverage lives at warn or behind an opt-in preset, and a team wanting breadth has to enable it. If your threat model wants maximum coverage at any noise level, our defaults are the wrong shape — strict is one line away.

Raising precision costs recall, and we have paid it. A precision sweep here took false positives 10 → 3 while false negatives went 18 → 34. A recall fix for no-unsafe-regex-construction took findings 29 → 2,243 at ~25% precision and was reverted, with the gap documented instead. A high bar biases us toward missing things. Deliberate trade, published misses.

70% at warn still means 3 in 10 are wrong. At a few hundred findings that is real hours. A warn is a question, not a verdict — but it is not free.

Opt-in rules have no floor, and some are genuinely noisy. detect-object-injection measured 0/13 on real code. Turning on every rule hands you our worst numbers.

Measurement is the bottleneck, not analysis. 73 hand-labelled findings per error-tier rule across 121 rules is thousands of judgements. That is why 117 rules are unscored, and why they get measured loudest-first rather than in the order we find them interesting.

We label our own findings. The person deciding TP or FP wrote the rule. Every label ships with its reason, file and line, so you can disagree with any one and recompute. It is not independent review and we do not call it that.

Where we sit today — 2026-08-17

The ReDoS row reads unscored rather than a percentage on purpose. The timing sweep behind it can confirm that a pattern backtracks and cannot prove that one doesn't — two patterns it called flat ran 167 ms and 956 ms under a crafted input. A method that only errs in one direction does not get to report a precision figure.

Across recommended on 20 open-source projects (21,146 files, 3.04M LOC), our measured precision is 28.6% against eslint-plugin-security's 13.0%, n=24 per side. Two in three of our findings on real code are wrong. That beats the incumbent by 2.2×, and that is the only claim it supports.

Headline results

Wild-corpus run (ILB-Wild, 2026-05-30)

Rules were applied to 21 real OSS repositories spanning 1,556,266 lines of TypeScript / JavaScript. Repos include Next.js (125K LOC), Vercel AI SDK (160K LOC), Webpack (158K LOC), Twenty CRM (419K LOC), the Serverless Framework (129K LOC), and fifteen others — frameworks, CMSes, AI toolkits, and serverless tooling.

Repos marked FP corpus (three.js, webpack, lodash, babel, react) are included specifically because they are unlikely to have real vulnerabilities — they exist to measure false-positive rate under adversarial conditions.

Edge-case stress test (ILB-Stress)

The stress test runs 57 hand-written cases against 16 rules. Each case encodes one of three hypotheses: a confirmed true positive (TP), a false-positive guard that should stay silent (FP), or a formerly-missed true positive recovered via audit (FN). All 57 cases match expected behaviour.

FP guard — a case the rule must stay silent on (e.g. bracket access on a typed array, a parameterized SQL query, a JWT secret loaded from process.env). These directly target the patterns that cause false positives in the generic alternatives.

FN recovered — a pattern that bypasses naive detection (e.g. Object.assign for prototype pollution, new Function() as an eval alias, an indirect JWT secret in a const). Each recovered case corresponds to a documented audit fix shipped with the rule.

ILB-Flagship precision / recall

Ground-truthed P/R/F1 numbers from hand-labeled fixtures (benchmarks/corpus/CWE-NNN/{vulnerable,safe}):

For comparison: the closest ecosystem peer on secure-coding/no-hardcoded-credentials scores Precision 100% / Recall 50% / F1 0.67 on the same labeled fixtures (it misses the "credential stored in a const before use" pattern).

AI-generated code security (ILB-AI)

Static analysis matters most where code volume is exploding fastest: AI codegen. The ILB-AI tier generates code from frontier models and scores it with these same rules.

Across 700 AI-generated functions from five models, 63% shipped a security finding. The per-model vulnerability rate (lower is better):

On realistic, structured tasks the gap narrows sharply. Running the same prompt through each vendor's CLI across four domains (NestJS, JWT, MongoDB, a general injection API) and linting both outputs is a near dead heat — 1 Gemini win, 2 ties, 1 split — and both frontier models skip the same hardening (algorithm allowlists, JWT aud/iss validation, query projections), because a feature prompt never names it. Frontier models avoid the catastrophic classes (injection, eval, hardcoded credentials) but consistently miss the defense-in-depth layer — exactly the negative space deterministic linting closes.

Full write-ups, each with the reproducible config:

• Claude vs Gemini across 4 security domains — a dead heat, and the hardening 63% of AI code skips
• Same NestJS prompt: Claude got 6 security errors, Gemini got 2
• Aggregate benchmarks lie — what 700 AI functions look like by security domain

import-next/no-cycle vs eslint-plugin-import

import-next/no-cycle runs head-to-head against eslint-plugin-import's cycle detection on the Next.js source (131K LoC):

eslint-plugin-import finds zero cycles on the same codebase because it uses a fixed depth cap (default: 10) that misses deep dependency chains. import-next/no-cycle defaults to unlimited depth with a deduplication cache that keeps memory bounded.

For full methodology and per-file breakdown, see the import-next rule docs.

What ILB is

The Interlace Linter Benchmark is an open measurement framework built into this monorepo. It has four tiers:

• ILB-Stress — adversarial unit tests for FP/FN edge cases (this page, above).
• ILB-Wild — runs the full rule set against real OSS repositories at pinned commits. Numbers are reproducible: npm run ilb:wild.
• ILB-Flagship — latency + head-to-head overlap on 10 flagship rules against competitor plugins on the same repos. Includes cold vs warm (ESLint cache) timing.
• ILB-AI — generates code from frontier models (Claude, Gemini) and scores it with the rule set; the AI-codegen security tier above.
• ILB-Arena / ILB-Juliet — ground-truth P/R/F1 from hand-labeled CWE fixtures.

All bench outputs are checked in to benchmark-results/ as JSON. The raw SARIF for the public submission is at benchmark-results/interlace-2026-05-09.sarif.

The benchmark infrastructure is intentionally public. If you want to run the same numbers against your own plugin, the corpus and methodology are in benchmarks/.

Building secure JavaScript with Interlace? Star the repo to get new rules and CWE coverage as we ship them — or follow the AI-code-security benchmarks behind them.

Follow the benchmarksStar on GitHub