Mosaic

DETERMINISTIC EVALUATION

Mosabench
Agentic SWE Benchmark

Evaluate any local or cloud LLM across 70+ deterministic multi-language coding tasks executed inside the real Mosaic autonomous loop.

Install CLI Suite
$curl -fsSL https://mosaic.methil.group/api/install-bench | sh

Run mosabench --list to inspect all 70+ test suites and task definitions.

TASK SUITES

Comprehensive Multi-Language Coverage

Every task is graded by automated test suites in an isolated temporary workspace.

16 tasks

TypeScript & Node.js

Dependency refactors, async race fixes, typing migrations

14 tasks

Python & Pytest

Data pipelines, algorithmic repairs, mock fixtures

12 tasks

Rust & Cargo

Borrow checker resolution, lifetime annotations, unsafe fixes

10 tasks

Bash & Linux Shell

Sed/Awk pipelines, log parsing, POSIX script fixes

8 tasks

Go & Concurrency

Goroutine channels, deadlock elimination, interface fixes

10 tasks

Java & Kotlin

Gradle build fixes, Spring controllers, null safety

EVALUATION SCALE

Strict Ranking Matrix

S++
90-100%

Autonomous perfection, 0 human intervention, 100% test pass on first attempt

A+
75-89%

High autonomy, self-heals after 1 retry, passes complex multi-file diffs

B-
60-74%

Capable on single-file tasks, occasionally gets stuck in tool loop

C+
45-59%

Syntax valid but fails edge cases or introduces regression bugs

Quick Commands

mosabench --provider local --model qwen3.5-4bmosabench --suite curated --parallel 4mosabench --provider lmstudiomosabench --provider openai --model gpt-5-mini