← Case Studies

Self-hosted AI / Local LLMs · Agent workflows

Evaluating a Self-Hosted LLM for Real Development Work

A one-day proof of concept on whether a model on our own GPU server could handle real coding work. It took three scope cuts to get one result I could compare against a cloud model.

Situation

Leadership wanted to know whether developers could use a self-hosted model, running on a GPU server in our own data center, for coding work we’d otherwise send to a paid cloud AI.

I owned the developer-workflow demo for that PoC. We presented our demo and recommendations to the leadership team.

The plan was to have the local model implement a new endpoint in one of our existing APIs. It ended up as a feasibility experiment instead where I found a task the local model could finish in a reasonable amount of time and compared it against a cloud model on quality, time, and cost.

The problem

The local model took a very long time. On the endpoint implementation task, it spent over 30 minutes reading the repository and never wrote any code. So I had to cancel the run before the agent was able to complete it.

I then narrowed the task to have the agent write unit tests for a single backend service, which took more than an hour. Again, I had to cancel it halfway.

Meanwhile, I was running out of time to try things. We had allocated only a day for this PoC. The hardest part was that I didn’t know why it was slow. It could have been the model, the server it ran on, how much context it was carrying, or the agent tooling around it, and the right fix depended on which one it was.

What I considered

The first choice was what to demo. The AI assistant I used for planning suggested a self-contained exercise: have the model write tests, catch a defect seeded into the code, and verify the fix. The alternative was implementing a new endpoint in an existing API plus its unit tests - a typical backend API update.

Once I realized that the local model was struggling, I considered some alternatives for the PoC.

  • A cloud model could gather the repository context up front so the local model didn’t have to explore.
  • Taking it one step further, the cloud model could do all the planning and hand the local model small, pre-scoped implementation steps.
  • A different option was to stop and diagnose the slowness before changing anything else.

At the end there was a decision about what to show leadership: just the final comparison, or the whole path to it, including the parts that didn’t work.

What I chose and why

I initially went with the endpoint implementation because it covers both writing code and writing tests, which is what a typical backend API story involves.

When the local model appeared to be struggling as it was spending a lot of time just exploring the code repository, I decided to have a cloud model gather the repository context and hand it to the local model to save time. However, that didn’t help much.

I then tried a hybrid workflow where I use a cloud model to generate an implementation plan and have the local model write the code per the provided plan. However, it was still struggling.

The deadline was close so I dropped the hybrid approach and switched to having the local model write unit tests for an existing service file and compared the implementations between the local and the cloud model.

That run took more than an hour and I had to cancel it halfway. So I decided to narrow the scope again. I cut the task to writing one specific happy-path unit test, which the local model handled in a much more reasonable time. Granted it was still much slower than the cloud model, but at least it wasn’t more than half an hour wait time.

Last, I decided to present the whole journey to leadership instead of just the final comparison. The stakeholders were trying to judge whether local models are feasible for development work, and where the model struggled tells them as much as where it succeeded.

What happened

Scoping down the task is what made the local model usable. On the endpoint, it spent more than 30 minutes reading the repository and didn’t even finish it let alone writing any code. Unit tests for a single backend service took more than an hour and still didn’t finish it. One specific unit test, with the reference test file and the exact cases named in the prompt, took about 13 minutes.

For the last task I gave both models the same prompt and the same code baseline. The cloud model finished in about a minute; the local model took 13. Both tests passed, and a blind review by a cloud model in a separate Copilot session accepted both. The review also found a latent problem in the test written by the local model. It computes its expected dates from the clock after the service call returns, so a run that crosses midnight UTC would fail by one day even though the service is correct. The cloud test avoids this by checking that the result matches either day. It’s a real flake waiting to happen.

On cost, the local run had no model API charge. The cloud run used 17.6 AI credits, about 18 cents of usage value, which may well be covered by included credits. The local run also processed about 3.7 times as many input tokens (1.2 million against roughly 331,000), most likely because it ran longer and re-set its context on every turn, so that number reflects the agent session more than the model itself. Waiting 12 extra minutes to avoid 18 cents isn’t worth a cost saving in my opinion, and I hadn’t counted hardware, power, or administration cost for the server running the local model. Therefore, I decided to conclude that the saving is unproven. What the demo did show is that a self-hosted model can produce an acceptable unit test when the task is small and spelled out.

What I’d do differently

The biggest gap is that I cut scope without knowing why the model was slow. After the hour-long unit-test run, we could have diagnosed the issue by handing both models everything they need in the prompt, turn off their access to the repository, and time how long each takes to write one test. If the local model was still slow, the bottleneck was the model or the server. If it was fast, the time had gone into exploring the code. But I didn’t have enough time to dive into that.

So the next time I’d run that diagnostic before shrinking the task, because it tells us whether the fix is a smaller task, a different model, or better hardware. I’d also run each model more than once. One run each shows the task is feasible, not how reliably. And I’d actually try the hybrid workflow I had planned, with the cloud model planning and the local model implementing in small pieces, since it’s the closer test of whether local compute can take on real story work rather than single tests.