I run a headless Hermes Agent on an M4 Pro Mac mini with 64GB of unified memory. It handles email, web search, scraping, and scheduled jobs. I chat with it from Telegram anywhere in the world, and when it needs actual maintenance, I SSH in over Tailscale.
Getting there took weeks of trial and error, and almost every instinct I started with turned out to be wrong. Here is the honest account: what I tested, what actually moved the needle, and what cost me weeks so it doesn’t cost you any.
If you’re just getting started, these two guides will get your headless Hermes Agent up and running before you dig into tuning:
- How to Set Up a Headless Hermes Agent on a Mac Mini
- How to Give Your Headless Hermes Agent Real Tools to Work With
Variable 1: Model size (the wrong axis)
My first assumption was the obvious one. Slow agent, so pick a smaller model.
I started on GLM-4.7-flash, 19GB on disk and about 26GB resident with a 128K KV cache. It loaded fine on 64GB. It also took 296 seconds to answer a one-line question at the 64K context Hermes requires as a floor.
So I tested a much smaller model, qwen3.5:9b, same machine, same 64K context. 6.4 seconds. A 46x difference.
That looked like proof that smaller wins. It was not. It was proof that something about the big model’s memory profile was wrong, and I nearly drew exactly the wrong conclusion from good data.
Variable 2: Architecture (the axis that actually mattered)
The real number governing speed on Apple Silicon is memory bandwidth. My M4 Pro has roughly 273 GB/s. A dense model reads every parameter for every token it generates, so a dense 27B at Q4 (about 17GB) caps out near 16 tok/s in theory and lands lower in practice. That ceiling is arithmetic, not tuning.
Mixture of Experts changes the math. An MoE model activates only a small slice of parameters per token. A 35B MoE with 3B active reads roughly 2GB per token instead of 17GB, so it generates at close to a 3B model’s pace while carrying a 35B model’s knowledge.
I moved to qwen3.6:35b-a3b, 24GB, MoE with 3B active. At the same 64K context, on the same machine: 10.2 seconds cold, including loading 23GB from disk. I ran it again immediately to get the number that actually matters:
curl ... -o /dev/null 0.01s user 0.01s system 0% cpu 5.843 total
5.8 seconds warm at full 64K context. A 50x improvement over the same machine’s 296 seconds, and the winning model is four times larger than the one it beat.
| Model | Type | Context | Time |
|---|---|---|---|
| glm-4.7-flash (26GB resident) | MoE | 131K | 296s |
| glm-4.7-flash | MoE | 8K | 7.8s |
| qwen3.5:9b (6.6GB) | dense | 64K | 6.4s |
| qwen3.6:35b-a3b (24GB) | MoE | 64K | 5.8s |
The larger model won. Size was never the variable. Published Apple Silicon benchmarks make the same point more starkly: an M4 Pro can outrun an M3 Ultra on MoE workloads despite the Ultra having roughly triple the bandwidth, because once you stop being bandwidth-bound, extra bandwidth stops buying you anything.
RAM tells you whether a model loads. Bandwidth tells you how fast it runs. MoE breaks the link between the two, because it only reads a fraction of the model for each token.
Variable 3: Context and KV cache
The GLM disaster was never about weights. It was the KV cache. At 128K context with an f16 cache, the cache alone can cost more memory than the model. Quantizing it to q8_0 roughly halves that, and the Hermes docs show q4_0 cutting KV memory by about 75% versus f16 on a 9B model at 128K.
Holding everything else constant, same machine and same warm model and same trivial question, context size alone produced this:
- 131,072 context: 296 seconds
- 32,768 context: 11.5 seconds
- 8,192 context: 7.8 seconds
The timing breakdown made it undeniable. Ollama reported 4.8 seconds of actual generation and 0.1 seconds of prompt evaluation. The other 291 seconds were memory thrashing while allocating an enormous KV cache. The inference was never slow. The allocation was.
Note that this is a cliff, not a slope. Below a threshold, everything is fast. Above it, you fall off to minutes. Find your cliff by testing and sit comfortably below it.
There is a floor to respect too. Hermes requires 64K minimum context, so “just shrink the context” is not an unlimited lever. I settled at 65536 with OLLAMA_KV_CACHE_TYPE=q8_0 and OLLAMA_FLASH_ATTENTION=1, and the thrashing disappeared.
Variable 4: Keeping the model always warm
That 5.8-second number only holds if the model never leaves memory. Ollama unloads after five minutes of idle by default, so a Telegram message sent an hour later pays a 23GB reload before it does anything. My agent sits idle most of the day, which means without keep-warm I would have hit the cold path almost every time.
Getting this working took three separate discoveries, and each one looked solved before it was.
First, the restart command was lying. The Ollama Mac app runs two processes: a GUI wrapper called Ollama and the actual server, ollama serve, as a child.
/Applications/Ollama.app/Contents/MacOS/Ollama
/Applications/Ollama.app/Contents/Resources/ollama serve
pkill -x Ollama matches only the first by exact name. The server never restarted, so it never picked up any environment variable I set. I “restarted” it a dozen times before noticing the PIDs never changed. Use pkill -i ollama instead.
Second, launchctl setenv does not reliably reach the server. The app also hardcodes OLLAMA_CONTEXT_LENGTH=131072, which is exactly the setting that caused the 296 second disaster. You can set variables all day and watch the server ignore them.
Third, and this is the one that actually mattered: Hermes talks to Ollama over the OpenAI-compatible endpoint at /v1. keep_alive is not part of the OpenAI spec, so Hermes cannot pass it in the request body even if it wanted to. My early curl tests showed UNTIL Forever only because I was passing keep_alive: -1 explicitly, which the framework cannot do. Hermes is entirely dependent on the server default. That is why it kept going cold at exactly five minutes while my manual tests looked perfect.
Variable 5: The clever architecture that failed
I built the setup everyone designs on a whiteboard: a warm pool with a small fast model for routing and tool calls, a bigger one for reasoning, with delegation between them.
On 64GB it caused more problems than it solved. Models evicted each other, KV cache contention at 64K was constant, and every bug came with the extra question of which model had handled the request.
What worked was the opposite. One well-chosen MoE model, pinned warm, with the context sized deliberately. Simpler, faster, and far easier to debug.
If you are starting today
- Pick MoE over dense, and let architecture drive the choice before size.
- Quantize the KV cache before you blame your RAM.
- Set the context deliberately, and check your agent framework’s minimum first.
- Run the model server as a real service, not a menu bar app, with keep-alive in the plist.
- Verify warmth against the live process, not the setting.
ollama psshould readUNTIL Foreverand100% GPU. - Trim your toolset ruthlessly, then measure again.
- Test the boring failure modes: reboot after power loss, restart after update, cold start after a week idle.
The wins were unglamorous. Right model type for the hardware, correctly sized context, a service that actually receives its environment variables, and a session that is not bloated. Chase those first, and skip the clever part entirely.
