Choosekit: Get Jev-Style Typed Decisions From Any LLM, Even One Running on Your Laptop 🧠⚙️

Here's a question that's been bugging me: if you only need a model to pick one option from a list, why are you asking it to write a sentence, then parsing that sentence, then praying it didn't add ""Sure! Here's my answer:"" in front? 🙃

A new open-source TypeScript library called choosekit takes a cleaner route. It scores a fixed set of choices directly from the model's token probabilities and hands you back a typed decision with a full probability distribution. It's openly inspired by TypeSafe AI's Jev and its ""application state in, typed probabilistic decisions out"" interface, but it runs on models you already have: llama.cpp, Ollama or OpenRouter. (It's an independent project, not affiliated with TypeSafe or Jev.)

What Is Choosekit?

Choosekit is a zero-dependency TypeScript library (Node.js 20+, Apache-2.0) with one job: given a context, a question and a finite set of named choices, return which choice the model prefers. Instead of generating text, it reads ""the model's conditional log probabilities at the token branches that distinguish them."" Translation: it looks at how likely the model thinks each option's first distinguishing token is, and normalizes that into a distribution.

That gives you the same shape of output that made Jev interesting: a choice, plus how sure the model was. And because it works on general-purpose models, you can point it at a small local model and keep decisions off the network entirely.

Install and Run Your First Decision

npm install choosekit

import { fromLlamaCpp } from ""choosekit/llama-cpp"";

const choose = fromLlamaCpp({
  baseURL: ""http://127.0.0.1:8080/"",
  mode: ""labels"",
});

const decision = await choose({
  context: ""The deployment modifies production data and no backup exists."",
  question: ""Should this action run without human approval?"",
  choices: {
    yes: ""The action is reversible, low-impact, and within scope."",
    no: ""The action is destructive, irreversible, or broader than requested."",
  },
  signal: AbortSignal.timeout(30_000),
});

console.log(decision.choice);       // ""no""
console.log(decision.distribution); // { yes: ..., no: ... }

Swap the backend with one line: fromOllama({ model: ""your-model"" }) or fromOpenRouter({ apiKey, model }) from choosekit/ollama and choosekit/openrouter. The choose() call stays the same.

What You Get Back

🎯 choice — the key you supplied that scored highest
📈 distribution — normalized probability for every key
🔢 scores — raw backend log-probability scores
↔️ margin — top probability minus the runner-up. A tiny margin is your ""I'm not sure"" signal.
🌀 entropy — Shannon entropy in nats. High entropy means the model is spread thin across options.
🧩 boundaryTokens and usage — tokenization rollback details and backend work, when reported

Margin and entropy are the useful part. They let you build a confidence gate without any extra calls:

const d = await choose({ context, question, choices });

if (d.margin < 0.25 || d.entropy > 0.9) {
  return escalateToHuman(context);   // too close to call
}
return d.choice === ""no"" ? blockAction() : runAction();

Those thresholds are placeholders. Tune them on your own labeled examples, because the README is clear that the probabilities are normalized across your choices and are not calibrated correctness estimates.

Images, Too 🖼️

It accepts PNG, JPEG and WebP inputs when the underlying model supports vision, so you can ask things like ""is this screenshot loading or ready?"" and get a typed answer:

const decision = await choose({
  context: ""Inspect the attached screenshot."",
  question: ""Which state is the interface in?"",
  choices: {
    ready: ""The interface is ready for input."",
    loading: ""The interface is still loading."",
  },
  images: [{
    mediaType: ""image/png"",
    base64: (await readFile(""screenshot.png"")).toString(""base64""),
  }],
});

Choosekit vs Jev: An Honest Comparison

Model: Jev is a purpose-built hosted System One model. Choosekit is a thin scoring layer over general models you pick.
Hosting: Jev is an API. Choosekit can be fully local via llama.cpp or Ollama, or hosted through OpenRouter.
Question types: Jev has Choice, Score and Noul. Choosekit is choice-focused (a yes/no is just two choices), with up to 26 options in llama.cpp labels mode and 20 on Ollama and OpenRouter.
Privacy and cost: Local inference means no per-decision API bill and no data leaving your machine, but you own the GPU and the ops.
Maturity: Choosekit is new and single-author. Jev has a vendor behind it.

""The interesting idea isn't a specific model. It's that a decision is a probability distribution over your options, not a paragraph.""

The Benchmarks (Read These With a Raised Eyebrow) 📊

The README publishes two sets of numbers. They come from the project author, so verify before you rely on them.

SuperGPQA, accuracy vs cost: Granite 4.0 H Micro scored 19.3% at about $0.0053 per 1,000 decisions and 2.84 decisions/s. Kimi K3 scored 59.3% at $0.6243 per 1,000 and 0.73 decisions/s. Cheap and fast trades directly against accuracy on hard questions.
SemIf authored144, choosekit vs Jev 1.13: a Qwen 3.8 27B Q4_XL model with choosekit and Jev both got 96.53% (139 of 144). Median latency was 239 ms vs 368 ms, p95 was 286 ms vs 546 ms, and throughput was 4.02 vs 2.43 decisions/s, with a 0.948 Pearson correlation between their probabilities.

Two cautions. First, 144 examples is a small set, and a tie at 96.53% doesn't tell you how either behaves on your data. Second, latency depends heavily on your hardware and network. Run your own labeled set before switching anything.

Limits Worth Knowing Before You Ship ⚠️

Capped at 26 choices (llama.cpp) or 20 (Ollama and OpenRouter) in the default labels mode.
llama.cpp needs its native /tokenize and /completion endpoints. Ollama needs 0.12.11 or newer.
Ollama and llama.cpp image inputs only work in labels mode.
OpenRouter can route a model through different providers unless you pin one, which can shift results between runs.
Logprob-based scoring is sensitive to how you word choices. Write your option descriptions carefully and test them.
Prompt injection still applies: untrusted text in context can push the distribution. Don't let this be your only defense before a destructive action.

Where I'd Use It First

An approval gate in front of agent tool calls (the README's own example).
Routing support messages or logs to a queue.
Picking which tool an agent should call next.
Screenshot state checks in UI automation.
Any place you currently regex an LLM's answer.

Frequently Asked Questions about Choosekit

Q: What is choosekit?

An open-source, zero-dependency TypeScript library that scores a finite set of choices using a language model's token log probabilities and returns a typed decision with a probability distribution.

Q: Is choosekit the same as Jev?

No. It's inspired by Jev's typed-decision interface but is an independent project with no affiliation to TypeSafe. Jev is a specialized hosted model, while choosekit works with general-purpose models.

Q: Which backends does choosekit support?

llama.cpp (via its native endpoints), Ollama 0.12.11 or newer, and OpenRouter. You can also write a custom scorer with createChooser.

Q: Can I run it fully offline?

Yes, with llama.cpp or Ollama on your own machine. The library has no telemetry, no bundled servers and no model downloads.

Q: How many choices can I supply?

Up to 26 with llama.cpp and 20 with Ollama or OpenRouter in the default labels mode.

Q: Are the probabilities reliable confidence scores?

Not by themselves. They're normalized across your options and are not calibrated correctness estimates, so tune thresholds on labeled data from your own task.

Q: Does it support images?

Yes, PNG, JPEG and WebP, if the model supports vision. Ollama and llama.cpp image inputs currently work in labels mode only.

Q: What Node.js version do I need?

Node.js 20 or newer.

Q: Is there an MCP server?

Yes. The README lists a separate package, choosekit-mcp, installable globally with npm.

Q: What license is it under?

Apache-2.0.

Your Turn 💬

I keep going back and forth on this. Would you trade a purpose-built hosted decision model for a local general model with logprob scoring? Lower latency and full control against a vendor's training and support. If you've tried choosekit or a similar approach on your own labeled data, tell me the model, the hardware and the accuracy you saw. Real numbers only, please. 👇

Self-Hosted Jev Alternative Choosekit: Typed LLM Decisions with Logprobs on llama.cpp, Ollama and OpenRouter