Running a coding agent on a CPU

I wanted a coding agent that never calls the cloud. No API key, no per-token cost, no code leaving the machine. Just Ollama, a laptop CPU, and whatever a 3B model can do.

The first version failed constantly, and it failed in the same place every time: the tool call.

JSON is a bad interface for a small model

Every tool-calling API assumes the model emits well-formed JSON:

{ "name": "read_file", "arguments": { "path": "src/index.ts" } }

A frontier model does this essentially always. A 3B model quantised to 4 bits does it most of the time, which is the worst possible reliability for something in a loop.

The failures were rarely about intent. The model knew it wanted to read src/index.ts. It produced:

• a trailing comma before the closing brace
• single quotes instead of double
• the JSON wrapped in a ```json fence, sometimes with commentary before it
• a correct object followed by a second, half-finished one
• "arguments" as a JSON string containing JSON

Every one of those is a parse failure, and a parse failure in an agent loop is not a small thing — the model gets an error, tries again, drifts further, and the loop burns its context on recovering from its own syntax.

You can push at this. Constrained decoding via a grammar helps a lot. Few-shot examples help. But you are still asking a small model to spend capacity on punctuation.

So I stopped asking.

A syntax with one delimiter

<<<TOOL
read_file
path: src/index.ts
>>>

Tool name on its own line, arguments as key: value lines, terminated by a sentinel. That is the whole grammar.

The parser is forgiving in the ways the model is unreliable:

const TOOL_BLOCK = /<<<TOOL\s*\n([\s\S]*?)(?:>>>|$)/;
 
export const parseToolCall = (output: string): IToolCall | null => {
    const block = TOOL_BLOCK.exec(output);
 
    if (!block?.[1]) {
        return null;
    }
 
    const [nameLine, ...argumentLines] = block[1].trim().split('\n');
    const name = nameLine?.trim();
 
    if (!name) {
        return null;
    }
 
    const args: Record<string, string> = {};
 
    for (const line of argumentLines) {
        const separator = line.indexOf(':');
        if (separator === -1) {
            continue;
        }
        args[line.slice(0, separator).trim()] = line.slice(separator + 1).trim();
    }
 
    return { name, args };
};

Note (?:>>>|$) — an unterminated block still parses. The model runs out of tokens mid-call more often than you would like, and the closing sentinel is the least informative part of the message. A line that is not key: value is skipped rather than fatal, which absorbs the running commentary a small model likes to add.

The failure modes did not disappear. They stopped being fatal. Same model, same quantisation, same prompt content — the task completion rate roughly doubled, entirely from removing a syntax the model was bad at.

The lesson generalises past this project: the output format is part of the prompt, and a format designed for a 200B model is a tax on a 3B one.

Multi-line values

The obvious hole is writing a file, where the value contains newlines. key: value cannot express that. Rather than escaping — more punctuation, exactly what I was removing — the last argument can run to the sentinel:

<<<TOOL
write_file
path: src/add.ts
content:
export const add = (a: number, b: number): number => a + b;
>>>

A key: with nothing after it means "everything until the sentinel is the value". One special case, and it is the case that comes up constantly.

The loop that never calls out

const run = async (task: string): Promise<IResult> => {
    const history = [systemPrompt(tools), userPrompt(task)];
 
    for (let step = 0; step < MAX_STEPS; step += 1) {
        const output = await ollama.generate(history, { stop: ['>>>'] });
        const call = parseToolCall(output);
 
        if (!call) {
            return { status: 'answered', text: output };
        }
 
        const tool = tools.get(call.name);
 
        if (!tool) {
            history.push(assistant(output), user(`unknown tool "${call.name}". available: ${[...tools.keys()].join(', ')}`));
            continue;
        }
 
        const result = await tool.execute(call.args);
        history.push(assistant(output), user(formatResult(result)));
    }
 
    return { status: 'exhausted', text: 'step budget spent' };
};

Three things that mattered more than the model choice:

stop: ['>>>'] ends generation at the sentinel. On a CPU every token is wall-clock time you feel. Stopping the moment the call is complete cut latency noticeably, because the model's instinct after a tool call is to narrate what it expects the result to be — tokens that are always discarded.

Errors go back as plain text the model can act on. unknown tool "read_files". available: read_file, write_file, list_dir gets corrected on the next step. A stack trace does not.

A step budget, not a completion check. Small models loop. They will read the same file six times looking for something that is not there. A hard budget turns an infinite loop into a bounded failure that reports what it did.

What a 3B model can and cannot do

After a few weeks of real use, the boundary is sharp.

It does well: mechanical, well-specified, local edits. Add a field to an interface and update the places that construct it. Write tests for a pure function. Rename a concept across a handful of files. Read an error and fix the obvious cause.

It does badly: anything needing more than a few files in mind at once. Design decisions. Anything where the right answer is "the requirement is wrong". It will confidently produce something plausible and structurally wrong, and because it is fluent it looks reviewed when it is not.

That boundary is fine, because the first category is most of what I actually wanted automated, and it is the category where sending code to a third party is the least justifiable.

The interesting result is not that a small model can do this. It is how much of the failure was interface rather than intelligence. I spent the first week trying to make the model better at JSON. The win came from deciding it did not have to be.