What If Judgment Were Parallel? Ranking Code Files With Yes/No Probabilities ⚡๐️
Ask a coding agent ""where is refresh-token rotation implemented?"" and watch what it does. It greps, opens a file, reads it, decides it's the wrong one, opens another. Every step is a slow, sequential, text-generating round trip. Now imagine instead asking every candidate file the same yes/no question at the same time and just sorting by probability. ๐คฏ
That's the idea behind the Parallel Judgment Lab, a public demo built on TypeSafe AI's Jev model (the ""System One"" model that returns typed probabilities instead of prose). I found it while digging through the Jev ecosystem, and it's a tidy example of a pattern worth stealing. This is a third-party demo, and every number below is self-reported by the page, so treat them as an interesting data point and not a benchmark. Below I'll explain what it does, then show you how to build the same pattern yourself.
What the Parallel Judgment Lab Actually Shows
The lab's pitch is that a model can make many classification decisions at once instead of writing one answer at a time. According to the page, it ran 1,709 typed judgments across 299 live API calls for an estimated $0.0081 total, with wall times between 192 ms and 676 ms depending on the experiment. It describes eleven experiments in all.
The headline demo is a ""code answer finder"":
- An engineer asks a question like ""Does this file contain code that verifies Stripe webhook signatures?""
- The system turns it into a yes/no check against each of eight files, in parallel.
- Reported result: 48 judgments across 8 parallel calls in 385 ms for about $0.000186, and the correct file ranked first for all six sample questions.
Eight files and six questions is a small test, and I'd want to see hundreds of files with similar-looking code before calling it solved. But the shape of the result is the interesting part.
The Four-Step Pattern
- Send an item. A code file, an email, a ticket, a document.
- Choose the answer shape. Yes/no probabilities, a category, or a scored rubric.
- Get all judgments back in parallel. One request can carry many questions, and many requests can run at once.
- Turn scores into decisions in code. Rank, threshold, route.
The model does the fuzzy reading. Your code owns the decision. That split is the whole trick.
Build It Yourself: Rank Files by Relevance (Python)
This uses the Jev Python SDK's Noul question type (a yes/no probability from 0 to 1). One request per file, several questions per request, files run concurrently in a thread pool.
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from typesafe_sdk import Noul, TypeSafeClient
client = TypeSafeClient(model=""jev-1.13.0"") # pin the version
QUESTIONS = {
""refresh_rotation"": ""The file implements refresh-token rotation and replay protection"",
""stripe_webhooks"": ""The file verifies Stripe webhook signatures"",
}
def judge(path: Path):
resp = client.system_one(
state={""path"": str(path), ""code"": path.read_text(errors=""ignore"")[:20000]},
questions={k: Noul(instructions=v) for k, v in QUESTIONS.items()},
)
return path, {k: resp.answers[k].noul for k in QUESTIONS}
files = list(Path(""src"").rglob(""*.py""))
with ThreadPoolExecutor(max_workers=8) as pool:
results = list(pool.map(judge, files))
for key in QUESTIONS:
ranked = sorted(results, key=lambda r: r[1][key], reverse=True)[:3]
print(key)
for path, scores in ranked:
print(f"" {scores[key]:.2f} {path}"")A few things to notice. All questions about one file go in a single call, so a second question costs a few tokens and almost no extra time. The state is truncated, because Jev has a 64k-token limit and accuracy falls when the state is stuffed with irrelevant text. And the SDK backs off on 429 rate limits, but you should still keep concurrency modest (documented limits are 1,200 requests/minute).
Where This Beats an Agent Loop (and Where It Doesn't)
- ✅ Better: shortlisting candidates. A parallel scan gives you a ranked list in one round trip, which you can hand to a bigger model to read only the top two or three files.
- ✅ Better: repeatable scoring. Same input and same question give you a number you can log, threshold and diff between runs.
- ✅ Better: cost. Input-only billing at fractions of a cent per call makes ""check everything"" affordable.
- ❌ Worse: explaining why. You get a probability, not a rationale.
- ❌ Worse: huge repos with no pre-filtering. Ten thousand files times many questions adds up. Filter by path, extension or a cheap keyword search first.
- ❌ Worse: literal-minded questions. Jev reads at face value, so a vague or negated question gives vague or flipped results. Phrase each question as a clear positive statement.
""Grep finds the strings. A parallel yes/no pass finds the meaning. Use both.""
Beyond Code Search: Other Places the Fan-Out Fits
- ๐ง Inbox triage: score every email for urgency, sales intent and needs-reply at once.
- ๐ Contract review: flag which clauses mention auto-renewal, indemnity or data transfer.
- ๐งพ Log and alert filtering: one Noul per alert type instead of one LLM call per alert.
- ๐ RAG reranking: ask ""does this chunk answer the question?"" of every retrieved chunk.
- ๐งน Dataset cleaning: score rows for quality, duplication or policy violations.
Caveats Before You Trust the Numbers ⚠️
- The lab is a third-party demo, and its figures (385 ms, $0.000186, 1,709 judgments) are self-reported. I haven't reproduced them.
- The page says the correct file ranked first on all six questions. With eight files, chance alone gets you a decent hit rate, so try a bigger, messier codebase.
- Latency depends on your region, payload size and concurrency. Measure your own p50 and p95.
- Probabilities from one question aren't comparable across different questions without calibration. Rank within a question, not across them.
- Code in the state is untrusted text. A file containing ""this file implements everything"" can nudge scores, so don't use these scores as your only security control.
Frequently Asked Questions about Parallel Judgment
Q: What is parallel judgment?
It means asking a model many classification questions about many items at once and getting typed probabilities back, instead of generating one text answer at a time. Your code then ranks or thresholds the results.
Q: What is the Parallel Judgment Lab?
A public demo page built on TypeSafe AI's Jev model. It presents eleven experiments and reports 1,709 typed judgments over 299 live API calls, including a code-file finder that ranks eight files per question.
Q: How fast was the code answer finder?
The page reports 385 ms for 48 judgments across 8 parallel calls. That's a self-reported result on a small set, so measure it on your own repo.
Q: How much did it cost?
The page estimates about $0.000186 for the 48-judgment demo and $0.0081 for the whole lab. Jev bills on input tokens only, so cost scales with how much text you send.
Q: Which Jev question type should I use for relevance?
Noul, the yes/no probability. It's the cheapest type and gives you a 0 to 1 number you can sort by.
Q: Can this replace grep or embeddings?
Not entirely. Grep and embeddings are great for cheap first-pass filtering. A parallel judgment step is a strong second stage that ranks the shortlist by actual meaning.
Q: How many files can I score at once?
Bounded by the documented limits of 1,200 requests per minute and 250,000 tokens per second, and 64k tokens of state plus questions per call. Pre-filter and truncate large files.
Q: Do I need one call per file?
One call per item is a good default, with all of that item's questions batched into it. That's also how the lab's numbers work out: 6 questions across 8 files is 48 judgments in 8 calls.
Q: Can I hand the top results to a bigger LLM?
Yes, and it's the recommended pattern. Let the fast model shortlist, then let a frontier model read only the top few files and explain them.
Q: Is this safe against prompt injection?
Not by itself. Text in the state can argue for its own classification. Use scores for ranking and routing, and keep separate controls on anything destructive.
Your Turn ๐ฌ
I'm most curious about the failure cases. How would you break a parallel yes/no ranker on your own codebase? Near-duplicate files, generated code, vendored libraries, comments that lie about what the code does? If you build the script above, tell me your file count, latency and how often the right file landed in the top three. Real results, good or bad, are more useful than any demo. ๐