Judgment Calibration Dataset v1

Download

See the README for structure, statistics, the adopted labeling rule, desensitization policy, and citation.

Coverage: Feishu bot 2026-08-15 → 2026-09-08 (24 days); WeChat bot dialogue 2026-08-09 → 2026-09-08, calibration log 2026-08-26 → 2026-09-08
Two production bots: Feishu (DeepSeek-primary, 4 models by call) + WeChat Work (Claude)
Distinct people who talked to the bots: 244 (Feishu) + 110 sessions (WeChat)
Operator messages: 152 (Feishu, 41 explicit 🔧 tuning) + 84 directives (WeChat) = 236
User messages: 1,062 (Feishu) + 533 questions (WeChat)
API calls with tokens and latency: 2,162 (Feishu; 103.3M tokens in / 1.9M out; median latency 8.3 s)
adopted label: 72 of 152 operator messages labeled by a documented lexicon rule (13 positive / 59 negative); 80 left null rather than guessed

Longitudinal, not cross-sectional — one operator, one system, first boot to autonomous operation
Natural pain signal — “你TMD” is the negative label, not a 1–5 Likert scale
Cross-model transfer — corrections found on DeepSeek applied to Claude through policy files under the iLang protocol, recorded step by step
Production environment — real paying customers, real stakes, real profanity
Raw and unfiltered — grey-area requests, bot fabrications caught live, operator frustration intact
Honest labels — heuristic labels are marked as heuristic; unlabeled rows stay unlabeled

arXiv:2510.27328, Lu, Song & Wang (2025): prior, independent interpretability work finding, inside eight LLMs, a Valence-Assent Axis that subordinates reasoning to judgment. Their evidence is internal to the model. This dataset is behavioral: a 24-day record of an operator correcting two production agents’ judgment through the iLang protocol layer, with every correction and its uptake logged. It contains no activations and no judgment vectors; it records how judgment was calibrated from outside.