PostHog details production use cases for decision models like Jev, which return structured, probabilistic answers (e.g., Yes/No, Choice, Score) with explicit uncertainty calibration, contrasting them with verbose, overconfident LLMs. Decision models are up to 444.6x cheaper and 193.6x faster than frontier models, enabling three primary patterns. First, they gate expensive LLM calls: in Replay Vision, two narrow yes/no questions filter PII with a 0.4 threshold, letting 98.7% of summaries skip Gemini rewrites; thresholds are tuned per error cost (low for false negatives, high at 0.85 for Slack bot routing to avoid annoyance). Second, they route requests: a choice question picks the cheapest adequate model (e.g., GPT Luna for simple tasks) using descriptive option notes, though thresholds require per-task, per-model calibration via an internal AI gateway. Third, they handle face-value checks on pre-processed data: embedding-derived candidate sessions are reranked by a single-pass match question, improving search ranking ~25% on 20 queries. Multi-step reasoning (e.g., chained "worth watching?" then "why?") flooded servers with 12-38k token requests and is operationally unsound. Decision models also replace overqualified LLMs for structured evaluation (e.g., chart picking from SQL and result rows), but fail on open-ended reasoning (AUC 0.52 for PR review ranking) due to context limits (64k tokens default, 262k opt-in, slower/less accurate). Hybrid models like Jeeves, which reason before deciding, now serve as PostHog's default for unspecified decision calls.
Read original
hackernews