Olivia Koshy, who works in product management at Hex, described her experiences over the past few years. During a conversation with a friend who also works on AI products, they realized that many projects that had seemed like failures suddenly became successful thanks to rapid advances in technology. Features they previously worked on for an entire quarter can now be completed within weeks. This experience led them to the so-called "bitter lesson" in the context of building AI products.
This lesson comes from long-term research in artificial intelligence (AI), which has shown that methods leveraging computational power ultimately prove the most effective. In practice, this means not trying to adapt AI to existing plans using complex engineering solutions. Many such solutions become obsolete with the arrival of a newer model trained on more data and compute. Instead, it is better to focus on understanding the capabilities of models and adjust plans accordingly.
Olivia Koshy now asks questions such as: What unique value do we provide that others cannot easily replicate? How can we leverage the emerging capabilities of models? Are we building something that will improve with better models, or are we merely working around their current weaknesses? After more than two years of building AI features, the team at Hex has also changed how they work: they have moved away from demos, track changes in model capabilities, and shut down projects that are not working more quickly. These changes were not obvious and came after many mistakes.
The First Failed Attempt at Notebook Agent
Notebook Agent became a major hit with customers, but its core idea dates back to early 2023. The team at Hex was convinced that it was the future, and they were right—they were just too early. For an entire year, they tried to create Notebook Agent with a simple goal: users would ask a question about data, and AI would generate SQL, charts, and Python code to answer it. At the time, models were not yet ready for complex multi-step reasoning, so the team devised a number of clever solutions to work around their shortcomings.
For example, the model often got stuck in an infinite loop, creating cells endlessly if it chose which cell to create next on its own. So they had the model first select a cell template—such as SQL, SQL plus a chart, Python, or Python plus a chart. Then, based on the selected template, they generated code for each cell in a single attempt. The feature looked amazing in demos but failed at anything more complex. The "agent," now better known as a workflow, had trouble selecting the right data, and when it managed to do so, it encountered too many unexpected problems and could not recover from them.
The team continued working on it longer than they should have because of the time they had invested and their deep belief in their notebook expertise. It was difficult to admit that it was not working. Later, when the Sonnet 3.5 model was released and brought significant improvements, they were not as enthusiastic about reviving the project because the original version had dragged on for too long.
Blinded by a "Good Enough" Result
Creating beautiful interactive visualizations with AI had been a priority for years. In October 2024, Hex released the Explore feature, which makes it possible to visualize data with new capabilities such as pivots, totals, charts, and spreadsheet-like functions. When releasing it, they decided it should also work with their AI. They underestimated the complexity—it looked like a simple cell, but it was actually a complex JSON structure similar to the Vega specification, with different rules depending on the type of visualization.
They devised a two-stage generation process that used reasoning models such as o3 and a small fine-tuned model for the visualization itself. It worked fairly well, and that was perhaps the biggest problem. Because it worked fairly well, they focused on optimizing the existing solution and overlooked new model capabilities. They missed the shift toward tool calling for complex tasks, as opposed to their two-stage generation of a JSON specification.
When they began building a framework for agentic tools, it became clear that a simple tool-calling approach was ten times better. Since then, they have released Threads, which enables anyone in an organization to independently answer data questions with trusted context. This feature is far more capable than anything that came before it and surpassed within weeks what had previously taken months. Threads can create multiple native Explore visualizations and provide useful outputs such as summaries and visualizations. Once again, the bitter lesson was confirmed—their clever engineering solutions were surpassed simply by leveraging improved models.
What They Do Differently Now
Based on their experiences over the past few years, Hex has changed its approach to building AI features. The fundamental principle is to get a feature into users' hands before it is finished. When they show it to beta customers, it is not a demo—they simply let them use it directly. If the meeting feels more like a bug hunt, they know they do not yet have something that works. Real validation takes place in customers' messy environments. Do they want to use it despite its imperfections? From personal experience, Olivia Koshy knows that AI demos often do not work in the real world and that it is easy to fool yourself into believing you have built something useful.
Another change is tracking shifts in model capabilities. Models such as Sonnet 3.5 and 3.7 brought major advances that the team did not fully take into account. They used them internally but did not fully leverage their potential, instead sticking to the old way of thinking involving RAG (retrieval-augmented generation) and multiple attempts. The release of Agent Mode by Cursor and Claude Code in February 2025 showed that agents really work. The main task for those managing roadmaps is not only to verify whether an idea is good and customers want it, but also whether today's models can handle it. If not, no amount of complex engineering will save it.
Finally, they shut down projects more quickly. As soon as they realize that the team has to devise solutions to compensate for insufficient intelligence, they stop the project. It is easy to let a team continue because of the time invested or a promised release date—but forget the date. When that happens, celebrate the failure if the team was able to change direction quickly. The ability to say "no, this isn't working" is often the hardest skill, and you need leaders who can do it even under pressure.
In addition, they retry failed ideas every three months or whenever a new model is released. Sometimes you are simply a little too early.
The reason for sharing these failures is to celebrate them. It is difficult to set your ego aside when you realize that several things did not work out. But Olivia Koshy advises giving yourself some grace; everyone is building in a new field where the foundations change every day. It is challenging, but it is also the most exciting time to build.
Source: hex.tech
