Like many others, I've grown to adopt AI chatbots more and more for various things. Often reaching for ChatGPT/Claude/Codex/Gemini/whatever when I need something quick done. The interesting thing, is that for many workflows, we don't actually need the cloud, or the most advanced models.
It's become a learned habit to reach for the most powerful model for everything, but another way is emerging. I'm a big fan of local AI models, and of making the most of the compute we already own. Our phones and computers are already quite powerful, and we can do a lot with them as-is.
Why Local?
The first question a savvy reader (you, probably) would ask: why do we need local models if the cloud-deployed ones work just fine?
Great question, with a few interesting answers.
Privacy
We typically don't want to share sensitive information with other parties. For example, medical records, identification documents, sensitive financial details, and so on. Local intelligence means you don't have to trade convenience for privacy.
This got more concrete with the release of Fable. For security reasons, prompts and outputs from Mythos-class models are retained for 30 days. Local sidesteps the question of data-rentention neatly. There's nothing to retain because nothing leaves the device doing the work.
Cost
Finding the right tool for the right job matters. As the saying goes, if all you have is a hammer, everything looks like a nail.
With Claude, I had a hammer, and everything looked like a nail. I deployed it for everything, big and small. That changed a few months ago, when I got more interested in running some models locally, and realised local models could handle a chunk of my workflows. The models themselves have no running costs. They are typically open-weights, and only need hardware good enough to run them. The hardware isn't free, of course, but they are typically already paid for. On that basis, I've cut my AI spending considerably.
Access
You don't need always-on internet to use local models. I've been in situations with no connection where being able to keep using the tools I'd gotten used to was needed. And there are communities and countries where internet isn't cheap enough to keep "always-on". Local models are useful there too, for individuals and companies alike.
Are the models good enough?
The second and arguably most important question: are local models good enough to actually use?
From my experience, the answer is a very satisfying "it depends." It depends on the task, where you're working, how much creativity you need, both from yourself and the model, and a lot of other things. Basically, it depends.
What makes local viable is that we don't need the most advanced models (or SOTA) for many of the automations that give us real productivity gains. In a 2025 study, small open models like Llama 3.2 3B and Phi-3 Mini matched 70B models on news summarisation, and wrote tighter summaries doing it. They shine on narrow, well-defined jobs. For open-ended work the big models still win, but that's not what we're asking of them most of the time.
The tasks that make day-to-day life more productive: classifying notes, summarising articles, tagging bookmarks, making sense of transactions, answering simple questions, pulling structure out of messy data. The models already on our phones do many of these well today.
This is the boring stuff we no longer want to do, or spend time figuring out, not because they aren't important, but more because many people don't find them interesting. Every one of these small, boring tasks reveals a lot about you, and none of it is fun to maintain. Narrow enough for a small model, private enough that you'd want it on your device. Perfect candidates for local.
Use cases and experiments
Some places I've tested local models, and where I'm looking to invest more soon. These all run locally, on my phone and MacBook.
Vessel: AI bookmarker
I built an iOS app called Vessel that runs entirely on my iPhone using foundation models. It organises, classifies, and resurfaces everything I save across the web and socials, and makes it all semantically searchable. It also surfaces interesting connections between saves I'd otherwise forget I made.
If you're interested, I'm opening access to a few people. Add your email here to show interest!
Ollama: Chatbot
Using the Ollama desktop app and Qwen 3.5, I can chat with a local model and get quick questions answered easily. I typically use this when I want to brainstorm on simple topics, don't have internet access, or want to chat about something private.
Scribe: Meeting notetaker
I also built a Mac toolbar app that takes personal notes during meetings, similar to Granola (but much much less feature-rich). Transcription and summarisation both happen on-device. I get a transcription, a summary synced across my devices so I can access on the go, and any actions to take from the meeting. Simple, but very effective.
OpenCode and Ollama: Coding agent / engineering harness
Coding with open models through OpenCode. Not as good as the frontier stuff yet, but I'm keeping an eye on it. It can already handle small, targeted changes reliably.
I've found a lot of joy using the SOTA models to use/control the local models to get engineering work done. For example, classifying a massive amount of data, or doing data-cleaning before analysis. This type of work would typically consume a lot of tokens, and use up usage allowances quickly, but with this delegation workflow, I'm able to do this type of work multiple times a day without significant costs or exhausting my SOTA usage allowances.
Conclusion
Most of what we hand to AI isn't the exciting work. It's the mundane-but-necessary stuff we'd rather not do ourselves. And it turns out a lot of that doesn't need the most powerful models we have access to. We've spent years buying genuinely powerful machines, and open-weight local models give them the ability to help us do so much more.
Let me know if you have any usecases you already have, or questions about mine you'd like to discuss. I'm happy to chat about this anytime!