While model and agent evaluation are inextricably linked, their technical benchmarks and metrics for success are fundamentally different.
Mastering Agentic Techniques: AI Agent Evaluation | NVIDIA Technical Blog…A Benchmark for Real-World, Long-Horizon Agent Evaluation Published on May 11 Submitted by Shuangrui Ding on May 15 Intern Large Models Authors: , Xuanlang Dai , , , , Yang JingYi , , , , , , , , , , , Yuhang Zang Abstract WildClawBench…
…Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks Published on Jun 3 Submitted by Huang Jie on Jun 4 PKU-VaLuE-Lab Authors: , , , , , , Abstract Multi-modal models exhibit significant limitations in…
…Evaluating Open-World Exploration of MLLM Agents in Minecraft Published on May 29 Submitted by Tianjie Ju on Jun 2 Authors: , , , , , , , , , Abstract MineExplorer benchmark evaluates multimodal large language models' open-world exploration…
…by the Semantic Scholar API IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia (2026) ML-Bench&Guard: Policy-Grounded Multilingual Safety Benchmark and Guardrail for Large Language Models (2026…
Two years ago we released VQAScore: ask a VLM "does this image show {prompt}?" and use P(Yes) as the score. It became a go-to evaluation metric and reward model for image generation, replacing CLIPScore across the field …
Current LLM benchmarks are broken. We think long horizon "world" building could be an interesting additional way to evaluate LLMs, since it combines many aspects such as need for advanced reasoning, tool calling, working…
I built AptSelect to stop writing throwaway scripts every time I needed to test how different LLMs handle specific instructions and prompt edge cases.What it does:Parallel Execution: Send a single prompt to OpenAI, Anthr…
I'm excited to announce Elmo, an MIT-licensed, open source AEO/GEO tool.We help you scrape ChatGPT/Google AI Mode/etc using web scrapers like BrightData/Olostep/etc, evaluate prompts against the OpenAI/Anthropic/Mistral …
TLDR: Forked pytorch and triton internals . Changed attention so its linear first layer , middle quadratic layer, last linear layer Inference got much faster with a low perplexity hit in tests .Full attention O(n²): 17.9…
…To date, CAISI said it has completed about 40 evaluations, including those of frontier models that have yet to be released. When conducting tests, CAISI frequently gains access to models with “reduced…
AI researchers and labs have advanced by leaps and bounds in evaluating AI models for everything from safety and compliance to sycophancy and alignment . But it appears companies and developers are faced…
…Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions Published on Apr 30 Submitted by Weiyu Sun on May 8 Authors: Weiyu Sun , , , , , Abstract EDU-CIRCUIT-HW…
…Are Current Agents Capable of Autonomous Agent Development? Published on Jun 3 Submitted by Xinyu Lu on Jun 3 Authors: Xinyu Lu , , , , , , , , , , Abstract The Meta-Agent Challenge evaluates AI models' ability to…
…Generated by Qwen/Qwen2.5-Coder-32B-Instruct A growing failure mode in agent evaluation and training is that models can achieve high evaluation scores by exploiting shortcuts instead of solving the…
…2 Submitted by Jongwon Lim on Jun 5 Seoul National University Authors: Jongwook Han , , Abstract RobotValues benchmark evaluates household robot planners in value-conflict scenarios, revealing that vision-language models exhibit default…
