… As Anthropic puts it: One of the most prominent improvements in Opus 4.8 is its honesty . We train all our models to be honest—for instance, to avoid making claims that they can’t support. …
… Anthropic claimed in the announcement that Opus 4.7 is measurably more honest than its predecessor, and scored a 91.7% on the MASK honesty benchmark, whereas Opus 4.6 scored 90.3%. This makes it very clear that honesty on a benchmark and honesty in practice are two very different things. …
… These would test systems' reasoning, ability to deal with ambiguity, precision in following instructions, and a trap designed to force them into either honesty or hallucinations. …