When people talk about speeding up the responses of large language models today, the discussion usually focuses on new hardware. Cerebras, with its chips designed specifically for this task, filed for an IPO in May, and the markets have responded positively. French startup Kog is taking a different approach. It claims that the graphics cards data centers have owned for years still contain an enormous amount of untapped performance that can be unlocked purely through software. The company promises up to thirty times faster response generation. Its demo, which immediately caught the community’s attention, generated 3,000 tokens per second for a single request. By comparison, ChatGPT manages roughly one hundred.
Kog attracted attention in May with a technical demonstration. The goal was to prove that lightning-fast response generation for a single user could be achieved even on standard server cards commonly used by companies. Specifically, the startup worked with AMD MI300X and Nvidia H200 chips.
While part of the community was disappointed that the results did not apply to laptop graphics cards, others quickly recognized the potential. The speed and cost of running models currently limit virtually every company seeking to deploy AI in production. The possibility of unlocking additional performance on already-purchased hardware therefore attracted enormous attention. “We received two hundred business inquiries,” company CEO Gaël Delalleau told TechCrunch.
A new perspective
The foundation is an approach Kog calls hardware-software co-design. The goal is to keep the card’s compute units running continuously and eliminate idle periods when the chip is merely waiting for data to be transferred between individual components.
Startup Kog therefore replaced the standard RCCL communication layer with its own library, the Kog Communication Library (KCCL). This ensures that when a model is distributed across multiple high-end cards, performance scales linearly rather than being lost to communication overhead. The second component is the LaneFormer architecture. It delays communication between cards by one layer, allowing computation to continue uninterrupted and text generation to proceed without a single synchronization pause.
Kog also optimizes memory operations based on the chip’s physical architecture, ensuring that every access is directed exactly where it needs to go. While other tools merely estimate the memory layout, Kog knows it precisely. The practical detail engineers will appreciate most is the ease of integration: the engine can be deployed as a direct replacement for commonly used LLMs, with absolutely no need to rewrite any code.
The company backs its data with direct comparisons against LLM and TensorRT-LLM on the same hardware and using the same model. On AMD Instinct cards, its engine achieved speeds up to 3.5 times higher. For the Llama 3 (8B) model, the company reports performance of 1,368 tokens per second.
Every second costs money
Delalleau expects software development to be the first major market. Anyone who works with autonomous coding agents knows the experience: the user assigns a task and then waits, sometimes for hours. Anthropic also recognizes that people are willing to pay for speed, charging several times the standard price for Fast Mode on Claude models.
Kog is targeting precisely those customers for whom this waiting is critical because their daily work depends on smooth AI processes. The startup is already working with partners developing tools that generate games and applications from a single prompt, as higher speeds in this field immediately translate into direct revenue. Nevertheless, the company admits that the market is not yet fully ready. Business discussions have shown that customers lack both the capacity and the desire to fine-tune small, specialized models. Since its launch, Kog has therefore focused all its efforts on accelerating large, complex models.
And that is precisely where its greatest challenge lies. The impressive demo generating 3,000 tokens per second ran on the small LaneFormer 2B model with two billion parameters, which the startup developed itself and recently released as open source. However, there is a long road from this success to the promised thirtyfold acceleration of full-scale large models. Parameter count is the single greatest obstacle to fast response generation.
Nevertheless, Delalleau maintains that the same approach will also work for large models. According to him, graphics cards have enormous potential, and the claim that they are unsuitable for text generation is mistaken. Newer chips offer ever-increasing memory bandwidth that current software cannot fully utilize.
Months of work for each type of chip
The disadvantage of such deep optimization is that it cannot be rushed. The team must devote several weeks or even months of intensive research to each new graphics card. For a startup with only eleven employees, this greatly limits the number of different chips Kog can support.
In the future, the company wants to transfer its methodology to processes managed by AI agents, which could integrate additional chips and models much more quickly. For Europe, which is striving to build its own independence in both hardware and models, the startup’s success is a key element. Moreover, Kog is not alone. The French ZML project has also released software that bypasses the Nvidia CUDA ecosystem and enables fast generation across competing chips.
