SERIES · 17 PARTS
From a teenage dream to a conversation partner of my own: my personal experience of building a local AI assistant, its first words, long pauses and the next steps.
Ever since I was a teenager, I had wanted to talk to my computer. Many years later, I was sitting in front of an application I had built myself, listening to the first words coming from the speakers. I could not understand a single one. It was still a wonderful moment.
Hearing those first words from my speakers was a wonderful moment. Afterwards, I noticed how long I had waited for them. My assistant could speak. It was still far too slow for a conversation, though.
My assistant could now listen and speak. Next, I wanted my voice to trigger actions. Those first simple commands grew into an intent pipeline that expanded with every new idea. Along the way, I kept having to decide how much freedom I actually wanted to give my assistant.
The skill system gave my assistant new capabilities. Controlling music, searching the web, checking the weather: much of this was exactly what I had wanted. It also took me further from my original idea of a completely local application. I needed to decide where my assistant required my permission. And I learned that a convincing answer was no proof that an action had actually taken place.
My assistant sometimes claimed to have completed things that had never happened. I wanted to be able to rely on its answers. So I began explaining more clearly what it was, what purpose it should serve, and which boundaries it needed to respect. Along the way, it also gained something I called its “personality.”
After the break, I made my decision: my assistant would get a web application. The question was how much of my existing work would survive that change. Did I have to start again? Or could I build on what already worked?
The web application gave my assistant a face. It also let me present information differently. Weather, the time, and system states no longer had to pass through the console as text or appear briefly in a spoken answer. They could have a place of their own.
Widgets let information remain visible. But that didn't settle what my assistant should retain. Continuing a conversation, reminding me of something later, and giving me an overview of the day were different tasks. Each needed a place of its own.
The first widgets had shown how useful visible information could be. News, stocks, and crypto brought more sources into the application. Individual displays developed into the Signal Desk. Along the way, I needed to keep track of what the data actually said—and what the language model was allowed to make of it.
My assistant now had a web interface, widgets, and a Signal Desk. The next addition didn't need to show more on screen. With Telegram, a message from my phone could reach the same assistant. That also raised a new question: who would be allowed to send it requests through this channel?
In part 2, I reduced my assistant's long pauses with streaming and a prepared Whisper server. The web interface introduced more steps. A fast model call alone now told me even less about when I would actually hear the answer.
My assistant kept gaining capabilities. But demonstrating a feature once was different from using it regularly. After a cancellation, restart, or lost connection, the application needed to return to a usable state. The work between the visible features became increasingly important.
