Ancaster

System Overview

When an LLM stream fails mid-generation (e.g. at token 800 out of 1000), standard implementations lose the entire response. Ancaster sits at the edge and buffers tokens as they stream. If the provider connection drops, Ancaster maintains the client connection, instantly spins up a new request to the provider with the prompt appended by the successfully buffered tokens, and seamlessly continues the stream to the client. The end user never knows a failure occurred.