LLM Thematic Generalization Benchmark
Thematic Generalization Benchmark: measures how effectively various LLMs can infer a narrow or specific "theme" (category/rule) from a small set of examples and anti-examples, then detect which item truly fits that theme among a collection of misleading candidates.
Open source
Reactions (1)
Similar products
by description similarity
Benchmark evaluating LLMs on their ability to create and resist disinformation. Includes comprehensive testing across major models (Claude, GPT-4, Gemini, Llama, etc.) with standardized evaluation metrics.
New LLM outperforming GPT-3.5
news.ycombinator.comRefuel LLM (84.2%) outperforms trained human annotators (80.4%), GPT-3-5-turbo (81.3%), PaLM-2 (82.3%) and Claude (79.3%) across a benchmark of 15 text labeling datasets.
An LLM purpose built for data annotation
news.ycombinator.comoutperforms GPT-3.5
A multi-source LLM benchmark across text, image, and audio that measures JSON value accuracy per field, not just schema compliance. 7 metrics, full leaderboard.
LLM Creative Story‑Writing Benchmark V3
github.comThis benchmark tests how well LLMs incorporate a set of 10 mandatory story elements (characters, objects, core concepts, attributes, motivations, etc.) in a short creative story