Kilo Code’s 40 Trillion‑Token Study Shows Trust Drives AI Adoption

At QCon AI, Brendan O'Leary of Kilo Code presented findings from 40 trillion tokens of real‑world usage. He argued that developers adopt AI tools not by chasing benchmark scores but by building trust. The talk framed the problem as a ladder, with each rung representing a more complex use of AI in coding.

Building Trust on the Lower Rungs

The data came from Kilo Code’s open‑source CLI, VS Code, and JetBrains extensions and a cloud gateway that lets teams choose from more than 500 models. Users are largely early‑adopters who experiment with the tool, and the dataset excludes automated traffic and anonymous users to focus on human choices.

The trust ladder has four rungs. The first rung is simple autocomplete; the second adds chat‑style coding assistance; the third introduces multi‑agent orchestration; the fourth involves end‑to‑end task automation. O'Leary stressed that if any lower rung is shaky—slow suggestions or wrong edits—the higher rungs fail because the developer loses confidence.

Three technical failure points emerged in the data. Context construction errors mislead the model; model‑routing decisions sometimes pick the wrong architecture; and feedback loops that capture user corrections are sparse at higher levels. The presentation outlined concrete fixes, such as improving context extraction, refining routing logic, and building measurable trust signals.

These findings suggest that adoption will grow as teams build solid foundations. For engineering leaders, the message is clear: prioritize trust, measure it, and iterate on the lower rungs before scaling more ambitious AI workflows.