Engineering in practice

Case Studies

Production engineering stories that follow a system question through evidence, implementation decisions, and tradeoffs.

Self-hosted AILocal LLMs · Agent workflows

Evaluating a Self-Hosted LLM for Real Development Work

Leadership wanted to know whether a model in our own data center could do coding work we'd otherwise pay a cloud AI for. It only finished once the task was cut down to a single unit test, and even then took 13 minutes to the cloud model's one.

Inside the case: each scope cut and why, a same-prompt comparison on time, quality, and cost, and the midnight-UTC flake a blind review caught

Selected investigations from my work on production software. System details are generalized where appropriate.