Local LLM Coding Bakeoff

All those benchmark graphs on huggingface of how good new LLMs are for local coding confuses me. Somehow they all seem to be the best pick according to their own creators... 🤔

To gain some actual real-world knowledge, I staged a local coding bakeoff for models I can run on my M5 Max MacBook Pro. Eight local models, twelve tasks each, no cloud involved, with Claude Code (Fable 5) acting as test designer, judge, and lab assistant.

I learned a lot through this test about which of these I can actually hand a coding task and expect something useful back from. I hope you will too.

The setup

I picked four languages that are relevant in my everyday coding life: Bash, Python, Ruby/Ruby on Rails, and Swift/SwiftUI. Three task types per language:

• Create a new project from a written spec
• Find and fix bugs in an existing codebase
• Implement a feature in an existing codebase

That's 12 tasks per model, 96 runs total. The testing grounds were real repos, not toy exercises: my own bash scripts, a Rails 8 ad-server app with its 655-test suite running against MySQL in Docker, a SwiftUI-for-terminals Swift package with XCTest, and pallets/click for Python (1,968 tests, runs in under two seconds, chef's kiss).

For each "fix" task, three realistic regressions were hand-injected into a frozen branch: a fencepost error in a rect implementation, a plain assignment quietly turned into ||= in a Rails callback, a sort key flipped from descending to ascending in a jq pipeline. Each bug was verified to fail specific tests before any model saw it. The models got the failing test output and the relevant source, same as you'd get paged with.

For the "new project" and "feature" tasks, the models were graded by hidden acceptance test suites, and every single verifier was validated against a reference implementation written before any model ran. If the reference couldn't pass the verifier, the verifier got fixed, not the reference. No task was un-passable.

The protocol was identical for everyone: same prompts, temperature 0.2, output as complete files, one repair round with the verifier's failure output fed back, and a hard 25-minute cap per generation. Every attempt ran isolated on its own git branch or worktree. Afterwards, every diff got a code review pass scoring idiom, latent bugs, and diff hygiene, because a passing test suite and good code are not the same thing.

The contenders

Everything local in my Ollama, plus two pulls for the occasion: qwen3:8b as a small baseline, gpt-oss:20b and gpt-oss:120b, qwen3.8:27b-mlx, muse-glimmer:30b-mlx, qwen3-coder:30b, qwen3-coder-next (52 GB), and laguna-s-2.1 (117.6B, nvfp4, and yes, it's an MLX model even though the tag doesn't say so). As a reference point, glm-5.2:cloud ran the identical gauntlet afterwards, out of competition.

Results

Final score is 50% automated correctness (full marks for a first-try pass, partial for passing on the repair round) and 50% reviewed code quality.

For context: glm-5.2:cloud also passed 11 of 12. The best model on my laptop matched the cloud reference's pass count. That sentence would have been science fiction two years ago.

What I learned

The great local-model tax is emission, not reasoning. The harness required complete files as output, and faithfully reproducing a 1,400-line file around a one-line fix turned out to be the single biggest killer. The gpt-oss models correctly diagnosed bugs, then destroyed their own fixes by emitting patch fragments or truncated files. The 8B baseline corrupted nearly every large file it touched. Greenfield tasks, where there's nothing to reproduce, were the field's best category. If you run local models for coding, this is an argument for diff-based or agentic tool-use harnesses over "print the whole file."

One Python task went 0 for 8. The click bug hunt stumped every local model. Two of the three bugs got found repeatedly. The third was a deleted x.reverse() line in the argument parser, and a deletion is invisible: there's no wrong line to stare at, you have to know what should be there. Only the cloud reference recovered it, and even then only on its repair round. muse-glimmer came closest locally with two perfect one-line reverts and no idea about the third.

A single idiom can decide a whole task column. Five of eight Swift feature attempts died on the identical compile error: comparing the codebase's custom Extended type to Int without calling .intValue. Four of seven Rails greenfield failures traced to Rails 7+ open-redirect protection, a missing allow_other_host: true that post-dates whatever Rails these models internalized. And both gpt-oss models sank the Bash project on the classic set -e plus ((var++)) trap, a one-character bug neither found in two rounds. Local models don't fail on architecture, they fail on the sharp edges of framework versioning.

Models' own tests are a liability. Three attempts shipped working code that passed my hidden acceptance suite and failed verification purely on wrong expectations in their self-authored tests. They wrote the bug into the test instead of the code.

Size buys nothing if you can't finish a thought. laguna-s-2.1 is the biggest model in the field and produced byte-exact, reference-grade fixes on the two tasks it completed. It timed out on the other ten, thinking at 26 tokens per second until the 25-minute cap killed it. On this hardware it's a very smart colleague who never returns from lunch.

Who earns their keep

qwen3.8:27b-mlx is the new daily driver. Eleven of twelve across all four stacks at 45 to 55 tok/s, best-in-field solutions on most of the greenfield tasks. One genuine vice: it re-emits files with gratuitous whitespace and indentation churn, which would poison a real code review. It needs a stern system prompt about not reformatting untouched lines.

muse-glimmer:30b-mlx wrote the best code, full stop. Highest quality score, surgical minimal diffs, the most craftsmanlike bash in the field. It's about twice as slow and lost two repair rounds to overthinking timeouts, but if a human reviews the diffs afterwards, this is the one I'd want producing them.

The qwen3-coder pair are speed demons with a narrow lane: superb surgical fixers (byte-exact reverts in seconds), hopeless at building from spec. qwen3-coder-next fixed my bash script's three bugs in 59 seconds and wrote a working dupe-finder in 19, then face-planted on every feature task and never once recovered from feedback.

gpt-oss:120b is the pragmatist: 8 of 12 in the least wall-clock of any large model, effective use of the repair round, and code the reviewer summarized as "correct-but-messy." It refactors things nobody asked it to touch.

The 8B baseline went 0 for 12, which is exactly why you run a baseline.

Caveats

Single run per cell, so no variance estimates. The whole-file output contract is honest for comparison but punishes models that would shine in an agentic harness with proper edit tools. And the 25-minute cap is a real-world constraint, not a capability measure; laguna would likely score differently with an hour per task, but I don't have that kind of patience and neither does my battery.

The rematch, when it happens, will be agentic: same tasks, but each model driving real tools in a loop instead of emitting files blind. My money's on the ranking shuffling considerably.