Decision profiles

How it works

You describe the decision and upload 100–2,000 past cases with the answer that was right. Hesperan answers each case, then fits a handful of numbers to your data — a temperature and one bias per answer — so that its probabilities mean what they say on your cases. No model is trained; the model is the same for everyone.

Every case is scored by a calibration that did not see it (five-fold cross-fitting). On those honest numbers Hesperan finds the lowest confidence threshold at which the one-sided 95 % lower confidence bound (Wilson) of the automated decisions' precision still reaches your target. Below the threshold, it abstains.

Create profiles in the console. The texts are used for the calibration only and are not stored — the profile keeps the fitted numbers and the scores.

What a calibration costs

Each example is billed exactly like a /v1/decide request with that text: from a plan, one decision for every started 4,000 tokens of the example together with the question and its answers (about 16,000 characters) — a short example is one decision, a long one two or more; from the balance, its input tokens.

A calibration is all or nothing. Before the first example is scored, the price of all of them is reserved from your plan (or, if the plan cannot cover all of it and pay-as-you-go applies, from your balance); if neither can, the calibration is refused at once and nothing runs. If the calibration fails — the model is unavailable, more than 10 % of the examples cannot be scored, or it is interrupted — the whole reservation is given back. If it succeeds, you pay for the scored examples; the few that could not be scored (at most 10 %) are given back and left out of the calibration.

Your examples

A CSV with a header row. The text column may be named text, state, input, ticket or message; the answer column label, answer, team, class or category.

text,label
"I was charged twice for order 4812, please refund",billing
"Mein Paket ist seit einer Woche unterwegs",shipping
"The app crashes at login",technical

Limits: 100–2,000 examples, each up to 20,000 characters, together up to 4,000,000 characters. 300 or more examples give tighter numbers. A high target needs enough evidence: proving 99 % precision takes a few hundred correct automated cases. The console tells you what your data supports.

Decide

curl https://api.hesperan.com/v1/decide/ticket-routing \
  -H "Authorization: Bearer $HESPERAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"state": "I was charged twice for order 4812."}'

{
  "decision_id": "0f6c…",
  "profile": "ticket-routing",
  "decision": "billing",
  "confidence": 0.9931,
  "action": "auto",
  "threshold": 0.962,
  "target_precision": 0.99,
  "calibration_version": 1,
  "probabilities": { "billing": 0.9931, "shipping": 0.0041, "technical": 0.0017, "account": 0.0011 },
  "raw_probabilities": { "billing": 0.9412, "shipping": 0.0301, "technical": 0.0176, "account": 0.0111 }
}

Billing and rate limits are the same as for /v1/systemone: the input tokens of the state plus the profile's question and options. Nothing is charged for an error: 404 unknown profile, 409 no calibration yet (the first one is still running or failed), 400 invalid request, 502 the model was unavailable or answered invalidly, 503 no model connected yet or a serverless model is starting (retry after Retry-After).

Each decision is logged with the threshold, target and calibration version in force, the model and a keyed hash of the input (not the input itself), so it can be audited later. Logged decisions and their outcomes are kept for 400 days.

Retries without double charges

Send an Idempotency-Key header (up to 255 visible characters, for example your ticket id). If the same key comes again within 24 hours with the same state, you get the first decision back — same decision_id, header idempotent-replayed: true — and are not charged again. The same key with a different state is refused with 409.

curl https://api.hesperan.com/v1/decide/ticket-routing \
  -H "Authorization: Bearer $HESPERAN_API_KEY" \
  -H "Idempotency-Key: ticket-4812" \
  -H "Content-Type: application/json" \
  -d '{"state": "I was charged twice for order 4812."}'

Report outcomes

When you learn the right answer — the team that finally handled the ticket — report it. The console then shows the live precision of automated decisions, counted under the current target, threshold and calibration only. It warns when the outcomes rule the target out (even the upper 95 % bound is below it) or when the share of automated decisions over the last 7 days differs from what calibration predicts by more than chance explains (binomial test, 1 %). Only reported cases count: report a random sample or all outcomes, not mostly the mistakes. This is coverage and precision monitoring, not a general drift detector.

curl https://api.hesperan.com/v1/outcomes \
  -H "Authorization: Bearer $HESPERAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"decision_id": "0f6c…", "actual": "billing"}'

Report outcomes for auto and review decisions alike; actual must be one of the profile's answers. Each decision takes one outcome: the same answer again is accepted (200 with "duplicate": true), a different answer is refused with 409, so reported precision cannot be rewritten afterwards. Outcomes are free.

Changing the target

Move the target in the console and the new threshold applies to the API at once — no recalibration. The precision is measured on your past cases; if your traffic changes, recalibrate with recent examples. It is a measurement, not a contractual guarantee.

Recalibrating

Calibrate an existing profile again with a new set of examples: same profile, same URL. While the new calibration runs — and if it fails — /v1/decide keeps answering with the current one. When it succeeds, the new numbers replace the old ones in one step; your target precision stays and its threshold is found anew on the new examples. One calibration per profile runs at a time.