Harness design dictates prompt‑cache efficiency

Neuralwatt’s new harness controls how much of a prompt is sent to the AI model. When most of the prompt is already stored in the model’s key‑value cache, the server can serve the cached tokens instead of recomputing them. Serving a cached token uses far less energy than computing it again. A cache miss means the entire changed portion of the prompt must be recomputed, which can double the energy use of a request.

Cache hit rates differ by harness

Over a month of traffic, the share of input that could not be served from cache ranged from 1.6 % to 11.1 %. A harness that cached 97 % of a conversation let a customer send only three uncached tokens per hundred, while a harness that cached 92 % required eight. That difference means about 2.7 times more pre‑fill work for the same conversation. On the GLM‑5.3 model a miss can cost even 2.5 times more energy. The extra energy also slows other requests on the same server because pre‑fill competes with generation.

Most cache breaks in lab tests were caused by the harness editing text it had already sent. For example, OpenCode’s plan‑mode reminder added a phrase to each user message and removed it on the next turn, forcing a re‑read of the whole previous prompt. File‑reading pruning by omp, tool‑list changes, and daily‑date updates also triggered re‑reads.