AI Evaluation Engineer for Code Agents

What I do

Agentic-benchmark task authoring

Agentic-benchmark task authoring is the process of converting a real GitHub issue or pull request from an open-source project into a self-contained evaluation task for grading AI coding agents. Each task ships as a pinned Docker environment with structured problem and prompt statements, an interface contract, a requirements file, a golden patch, a test patch, and an implementation-agnostic rubric covering functional, robustness, and style criteria. The rubric never references a specific implementation, so any correct solution path scores fairly. Before a task ships, it is validated end-to-end through an automated grader: the golden patch must pass the test patch, and the unmodified repository must fail it. Through Mercor's expert program I have authored 130+ SWE-bench-Extended tasks this way across Go, Rust, Java, Kotlin, C++, JavaScript, TypeScript, and Python, each graded at a sustained ≥0.95 QC threshold.

Rubric design & QC

Rubric design is the discipline of writing grading criteria that both human reviewers and LLM judges can apply consistently to an AI agent's work. The alignment rules that matter: every functional criterion must be verifiable by an offline test, behaviors stay atomic (no AND-stacked criteria), descriptions are behavioral only with no implementation details, every criterion carries source attribution to the requirements, prompt, or interface contract, and each rationale ties back to user or codebase impact. Criteria that break these rules produce noisy scores and unfair grades, so quality control re-triggers until both the reviewer and super-reviewer scores clear the program threshold. In practice this means several QC passes per task and a standing habit of splitting compound behaviors into separate, independently testable criteria linked by explicit dependencies. The result is a rubric an LLM judge can grade reproducibly, run after run.

LLM-as-judge pipeline operation

An LLM-as-judge pipeline uses a language model to grade another model's output against a structured rubric, replacing slow manual review with reproducible automated scoring. I operate the full orchestration toolchain that surrounds these judges: context generators, problem-statement generators, golden-plan generators, planning and execution graders, and the QC checks that test for alignment, prompt clarity, fairness, and rubric quality. Each stage produces artifacts the next stage consumes, so a weak problem statement or a leaky test surfaces before it can corrupt downstream grades. When the stock pipeline is not enough, I extend it with custom Python tooling — new graders, dataset filters, or report generators fitted to a program's specific quality bars. This is the same tooling discipline behind the 130+ tasks I have shipped through Mercor's expert program in Q1 2026 at a sustained ≥0.95 QC threshold.

Eval-harness consulting

Eval-harness consulting is for companies building internal AI coding agents or assistants that need an evaluation suite to grade them honestly. An engagement typically covers four deliverables: a rubric taxonomy that defines what good agent behavior means for your product, a harness architecture that runs tasks reproducibly (usually Docker-pinned), a dataset-sourcing strategy that draws realistic test cases from your own repositories, and an LLM-judge pipeline that quality-controls each evaluation cycle. The goal is an eval suite your team can extend without me: I design the taxonomy, prototype the judge, pilot it on a small dataset, and hand off the harness with documentation. This mirrors the methodology used in frontier-lab benchmark programs such as SWE-bench-Extended, adapted to the scale and budget of a single engineering team. Engagements are scoped hourly or per-project, sized to your roadmap and budget.

AI integration audits

An AI integration audit is an independent, fixed-bid review of the GPT or Claude integrations already running in your production code. I examine prompt quality and injection risk, fallback paths when the API times out or refuses, per-request cost ceilings and caching opportunities, failure modes, and whether errors surface in your observability tooling or fail silently to users. The deliverable is a written report with severity-ranked findings and concrete recommendations — prompt caching, streaming, fine-tuning versus RAG, or model selection — that your team can act on within a sprint.

Q1 2026 output

Languages I author tasks in

Across 130+ tasks shipped Q1 2026, with Go and Rust dominant:

Authoring a task in a language requires reading a real OSS PR end-to-end, building a reproducible Docker repro, and designing a working test rubric. So beyond what my shipped product code shows, I can credibly read and reason about production code in all of the above.

Methodology (NDA-safe summary)

Two alignment rules I never break

• Tests → rubric. Every functional criterion is verifiable by an offline assertion intest.patch. Two unverified criteria caps the score at ~0.80 regardless of other quality.
• Requirements → rubric. Behaviors specified in requirements.json and problem_statement.md have rubric coverage. Coverage gaps are penalized but less severely than alignment gaps.

Stacking rule

One observable behavior per criterion. If a criterion contains an AND joining two independent behaviors, split it into two criteria with adependent_on link.

Description quality

• Behavioral outputs only, no "correctly parses", no "appropriately handles", no "either A or B" disjunctions.
• Self-contained, no "before the change" back-references.
• Rationale is the user / codebase impact, not a description restatement.

Online-test handling

Criteria only verifiable by@pytest.mark.online tests are marked minor with the verification limitation called out in the rationale.

How to engage

Task authoring contracts

Volume work, typically per-task fixed pricing or hourly. I'm currently active in the Mercor expert program; I have capacity for 1-2 additional eval programs (no overlap with Mercor's customer set).

Rubric & harness consulting

Hourly or per-engagement. Common scope: design the eval taxonomy, prototype the LLM judge, pilot on a small dataset, hand off the harness for your team to extend.

AI integration audits

Fixed-bid 1-week engagement: review the integration end-to-end, deliver a written report with severity-ranked findings and concrete recommendations.

Polyglot code review

Hourly. Useful when you need an outside reviewer for Go / Rust / Java / Kotlin / C++ code where you don't have in-team depth.