← Latest products

Benchmark

AI doesn't find bugs unless you tell it what's wrong. Given a real repository, an agent must discover & repair as many bugs as they can. Agents are not given any hint about the type of bug or its location. New benchmark from researchers at Meta, Stanford, Harvard, UW, including the researchers who've worked on SWE-bench, ProgramBench etc.

Visit swesweep.comexternal site

Reactions (1)

Similar products

by description similarity

  1. A registry of agent benchmarks (including many OSS agent trajectories)

    explorer.invariantlabs.ai

    A better way of testing, inspecting, and analyzing AI Agent traces. - invariantlabs-ai/explorer

    Hacker News6 pts

  2. CyberGym/BountyBench-AI agents find zero-days and solve bug bounties

    twitter.com

    A turning point on AI agents in Cybersecurity, shown in two recent research papers from UC Berkeley and Stanford: CyberGym: AI agents discovered 15 zero-days in major open-source software BountyBench: AI agents solved real-world bug bounty tasks worth tens of thousands of dollars This represents a p

    Hacker News5 pts

  3. Cheddar-bench

    github.com

    unsupervised benchmark for coding agents

    Hacker News9 pts

  4. ML-Dev-Bench

    github.com

    Benchmarking AI Agents on Real-World AI Workflows

    Hacker News6 pts

  5. A high-level search agent

    gensee.ai

    After speaking with many AI developers and experiencing it ourselves, we found that building agents requiring web content is often bottlenecked on the “search” part, as it involves iterations of search, crawl, extract, re-query, and error handling.

    Hacker News7 pts