Agentic Coding Lab

A lightweight research framework for systematically comparing agentic coding setups — models, agent architectures, and prompt strategies — using reproducible benchmarks.

Motivation

When working with agentic coding, many practical questions arise: Does a setup with dedicated sub-agents for testing produce cleaner code than a single agent doing everything? Does Sonnet deliver better code quality than Opus on a refactoring task? How do different prompt styles affect the outcome? This framework was built to answer such questions with data instead of gut feeling.

What This Framework Does

The framework runs controlled experiments across four dimensions:

• Workflow variants — from simple one-shot generation ("vibe coding") to structured TDD with specialized agents per phase
• Model configurations — Opus 4.7, Sonnet 4.6, Haiku 4.5, with and without extended thinking
• Coding katas — standardized tasks, known (game-of-life, mars-rover) and novel (claim-office, claim-office-lite); see Katas
• Prompt styles — prose, example-mapping, user-story

Each experiment produces measurable metrics: token usage, code complexity, code smells, test coverage, TDD cycle discipline, and more.

Aggregation is research-question-driven: A research question (RQ) defines a selector query (factors × controls × outcomes) over experiments/runs/, and tooling generates runs.csv + summary.md on demand. Batch plans are pure data-collection helpers that fill missing replicates.

Core Concept: Research-Question-Driven Aggregation

Earlier iterations of this lab were batch-driven: each batch produced its own results folder, and a finding was tied to "the batch it came from". That coupling made it hard to evolve hypotheses (every new analysis required a new batch) and hard to combine evidence across batches.

The current model decouples three concerns:

1. A research question (RQ) declaratively describes what is being studied — its factors, controls, outcomes, and required replicates.
2. Runs are produced once and live in a single flat pool (experiments/runs/), independent of any RQ or batch.
3. Aggregations are derived on demand by selecting all runs matching an RQ's frontmatter — across all batches that ever produced matching cells.