What became of my childhood dream

I had not understood the first word that came out of the speakers. Even so, that moment was incredible. My program had answered. With a voice, beyond the line of text in the terminal.

Looking back over the previous chapters, that is still where it all begins. As a teenager, I wanted to be able to talk to my computer. Seeing Jarvis in a film later gave that idea a face. My own project started on a much smaller scale: a CLI application, a local model and the question of whether I would get any answer at all.

That first answer became a conversation. Gradually, the conversation opened up other possibilities.

The dream became a set of concrete tasks

Eventually, a voice alone was no longer enough. I wanted my assistant to carry out tasks, keep information visible and remember things. Each addition brought another question.

Each row involved more than adding a feature. An answer could arrive too late. An action might need confirmation. An interface had to show what had actually happened. And a model could sound convincing while being wrong.

That turned a rather ambitious idea into a series of problems I could work through individually.

The architecture gave me room to change things

The early separation of responsibilities in C# proved particularly useful when moving to the web interface. I did not have to throw away the core. Existing capabilities could stay while I added another way to interact with them.

The shared core still connects the console and the web host. That separation makes a real difference to me: I can work on the interface without rewriting every capability at the same time. And I can replace or test individual components without rebuilding the whole project from scratch.

That did not make every addition easy. Browser audio, state and shared resources needed work of their own. But the question was more often: Where does this new responsibility belong? The existing structure gave me somewhere to start.

I wanted to keep the freedom to change my mind without having to throw everything away.

That freedom had already mattered when I was trying different models, voices and approaches to speech recognition. It remained valuable as the project grew.

Developing with AI and learning along the way

Over these weeks, I learned a lot about developing software together with AI agents. I could try ideas quickly and see what came of them. At the same time, I had to learn to describe tasks clearly, review changes critically and keep an overview of the architecture. That combination was a lot of fun and helped me grow as a developer.

Integrating a small language model locally had become an ordinary part of development for me. Even on my own hardware, things were possible that I had once only dreamed of. With more graphics memory, I could later use a more capable model and improve the quality of the answers further.

The limits didn't disappear. But I got to know them: what would fit in memory? Which tasks could the model handle reliably? Where did I need a clearly defined skill? And how far could I optimize the processes before increasing effort brought little improvement?

Faster did not automatically mean better

At first, waiting was the most obvious problem. Later, it became just as important for a process to remain understandable and repeatable. A quick initial response did little good if a state then got stuck or I could not tell whether a task had actually been carried out.

The logs helped me look more closely. I could trace the path through the application and measure parts of it. A feeling such as “This is taking too long” became a more specific question: Where is the time going?

• An extra processing step has to justify its value.
• A clear result needs a visible state.
• An executed action needs more than a claim from the model.
• A new capability has to work alongside the existing processes.

These questions appeared in different forms across speech, widgets and vision. Getting something to work once was only part of the task. It should behave properly the next time, too.

My rules remained part of the project

The idea of a local assistant became more open as weather data, news and additional access paths arrived. The core could run locally; some capabilities still needed other services or devices. Those differences had to remain visible.

Instructions and approvals also had different responsibilities. Instructions could influence the model's answers. The application had to decide whether a particular action was allowed to run.

The moments when my assistant claimed to have done something it had never done made that separation particularly important. I wanted a way to check beyond its answer.

Meanwhile, development continued elsewhere

While I was working on Joris, the major AI systems gained new features too. Some capabilities I had built for my own assistant were becoming part of finished products. After several weeks of work, I did wonder how much I would still need to develop myself in the future. The value of what I had learned along the way, however, remained unchanged.

What building it myself has given me

Taking this route means work. Dependencies, bugs, tests and maintenance do not disappear because the result runs on my computer. When I want to change something, the responsibility for it comes with the freedom.

But I can change it. I can try an idea, add a skill, rebuild an interface or reconsider an earlier decision. The project does not have to stop at what I originally expected it to become.

For me, that is a large part of its value. Building the assistant also meant learning about speech, models, interfaces and devices, and understanding more of how those pieces work together.

The first voice from the speakers is no smaller an achievement just because so much more is possible now. It was the beginning. Without that moment, I might never have reached all the questions that followed.

A voice in new places

Telephony most recently added another way in. I could call Joris from the Gira G1 and talk to it. Once again, the familiar conversation core remained while another device became a way to reach it.

For me, that fits the whole journey: an idea became an application, and each new experience raised the next question. Not everything needs to be built immediately. The planned display of device states and possible external phone calls can remain ideas for now.

Joris can rest for a while

I have other projects and ideas I want to give time to. So I am setting Joris aside for now. After these intense weeks, that feels right. Not every new possibility needs to become the next task immediately.

My idea is to integrate an autonomous Hermes agent. Together with a suitable local model, it could handle tool selection and invocation more directly. I would like to try native tool calling as a new approach to parts of my existing intent processing.

Whether that works better would need to be established. Enough RAM would give me more room when choosing a model. Which model performs the tasks reliably, and how the agent connects to my existing capabilities, would be the next interesting question.

Anyone who has worked on an agent harness or agent loops will know the additional challenges. In my existing deterministic intent pipeline, the paths for known commands were clearly defined. An agent, by contrast, decides step by step which tool to call and how to proceed with its result. It can get stuck in loops, take unnecessary actions or let errors influence later decisions. I would need to carefully design when it should continue, ask for clarification or stop.

These new questions are exactly what interest me. I know this approach could bring different problems, and I look forward to trying it out.

The device integrations, interface and all that experience with speech would provide a foundation. I would not be starting from scratch. Much of what I have built so far could feed into that next step.

More pictures for this story

When I get the chance, I also want to add screenshots and videos to this series. I want to show what Joris looked like on screen, how the widgets worked, and what a conversation with it felt like. Some of these moments are simply easier to convey with an image or a recording than with text alone.

This review is therefore no finishing line. Being able to talk to my assistant, show it things, and even call it would already have meant a great deal to me at the beginning. Between then and now lie many decisions, detours, and small successes. That experience stays with me.

I hope you enjoyed reading this series and found something useful in my experiments. I have plenty of other ideas and look forward to showing you what comes of them.