January 7, 2026 · Daniel Kleissl

Local AI Hosting on the Mac: A Practical Guide for Code

Qwen3 Coder and Devstral with llama.cpp on Apple Silicon — the parameters that really matter

AI-generated illustration: a MacBook with a glowing AI chip and data streams emerging from the screen — AI-generated image

After sharing my experiences with the cloud-based AI assistant Codeium in my last article, here is the next step: the switch to a fully local setup. The reasons are obvious: maximum control over sensitive data, no latency from internet connections, the ability to work offline and the freedom to experiment with all kinds of open-source models. For anyone using a Mac with Apple Silicon, the good news is: thanks to the Metal architecture and clever software such as llama.cpp, hosting powerful models is not only possible but astonishingly performant.

In this article I will share our findings on hosting two coding models: Qwen3 Coder 32b and Devstral. I will break down the recommended command-line parameters, explain the most important sampling parameters in a practical way and show how to configure each model optimally for its specific strengths.

The Foundation: llama.cpp and the Magic of Quantisation

For local hosting on a Mac, llama.cpp is by far the most important tool. It is an inference engine written in C/C++, optimised for Apple Silicon, that uses the GPU (via Metal) for the compute-intensive matrix multiplications.

The key to running large models on consumer hardware lies in quantisation. A model such as Qwen3 Coder 32b would originally have a size of over 64 GB with 16-bit floating-point numbers (FP16). Quantisation reduces the precision of the weights in the neural network, e.g. to 4-bit integers. Models for llama.cpp come in the GGUF format. When choosing a file, you come across labels such as Q4_K_M or Q5_K_S. For getting started, Q4_K_M has proven to be an excellent compromise between performance, small file size and high quality.

If the hardware allows it, though, the rule here is: bigger (Q8 instead of Q4) is clearly better when it comes to the accuracy of the answer. If a fast answer is the priority, then it is advisable to choose the lowest quantisation that still delivers good answers. Normally this means everything up to and including Q4. In any case, a little willingness to experiment is required to find the optimal model and the best parameters for your own use case.

If you would like to learn more about quantisation and its technical background, this link provides a starting point for your own research: https://huggingface.co/docs/optimum/en/

The examples in the latter part of the article are intended as a starting point. If you want to make larger adjustments, you should as a first step inform yourself more precisely about the background and effects of the individual parameters. The level of detail and complexity can quickly grow immeasurable here. And even once you have decided on new parameters, they should ideally also be tested. Since changes in the parameters sometimes cause subtle or hard-to-quantify changes in the model’s behaviour, caution is advised.

Normally you will find enough suggestions for every model on the web, and we oriented ourselves on these too in order to keep the testing effort small. Most of the time, the effort of comprehensively testing these parameters bears no relation to the demonstrable improvements achieved with them.

The Art of Sampling: The Most Important Parameters at a Glance

Before we dive into the model-specific configurations, here is a brief overview of the most important sampling parameters that control the behaviour of the AI, and what they mean:

Parameter	Purpose	Effect
`--temp <value>`	Creativity: controls the randomness of the word choice.	Low (~0.2): deterministic, precise. High (~0.9): creative, but more error-prone.
`--top-k <value>`	Focus: restricts the choice to the K most probable words.	Prevents nonsensical words, but can lead to repetitive answers.
`--top-p <value>`	Dynamic focus: selects words whose summed probability reaches p.	More flexible than Top-K, often a better compromise between creativity and coherence.
`--mirostat <mode>`	Adaptive control: tries to keep the “surprise” (perplexity) constant.	An alternative to temp/top-p, good for long, coherent texts. 2 is a common mode. It rarely appears in recommended parameters and should therefore be considered experimental.
`--repeat-penalty <value>`	Anti-repetition: penalises the model for repeating words.	A value of 1.1 is almost always a good idea to avoid endless loops.

Examples from Practice

Model 1: Qwen3 Coder 32b – The Logical Problem Solver

Qwen3 Coder is an all-rounder with a particular strength in logical reasoning and the generation of high-quality, correct code. With its 32k-token context window it can easily process larger files. It is excellently suited to tasks where algorithmic correctness and adherence to complex instructions matter.

Use Case A: Precise Code Generation and Unit Tests

Here we want the model to follow our instructions exactly and generate predictable code. Creativity is undesirable.

./llama-server -m Qwen3-32B-Coder-Q4_K_M.gguf
-ngl 99
-c 32768 #shorthand for --ctx-size
-t -1 #-1 sets the number of allowed threads to unlimited
--temp 0.2
--top-k 10
--repeat-penalty 1.1

Why these parameters?

-ngl 99 or --n-gpu-layers 99: Ensures that all layers of the model are loaded into GPU memory.
--temp 0.2: A very low temperature forces the model to almost always choose the statistically most probable next token. The result is more deterministic and less error-prone — perfect for formulating a known logic.
--top-k 10: We restrict the choice to the 10 most probable tokens. This eliminates the risk that the model comes up with an odd but “creative” idea that would render the code unusable.
Example prompt: Write a Python function that implements a bubble-sort algorithm. Add type hints and a docstring that explains the time complexity.

Use Case B: Algorithmic Brainstorming and Refactoring Ideas

Sometimes we do not know exactly what the solution should look like and want the model to suggest different approaches.

./llama-server -m Qwen3-32B-Coder-Q4_K_M.gguf
-ngl 99
-c 32768
-t -1
--temp 0.7
--top-p 0.95
--repeat-penalty 1.1

As an experimental alternative, Mirostat lends itself:

./llama-server -m Qwen3-32B-Coder-Q4_K_M.gguf
-ngl 99
-c 32768
-t 1
--mirostat 2
--repeat-penalty 1.1

Why these parameters?

--temp 0.7 & --top-p 0.95: This combination allows the model to also consider less obvious tokens, as long as they belong to the group of the 95% most probable candidates. This encourages the generation of alternative solution paths without becoming completely nonsensical.
--mirostat 2: Mirostat is an excellent alternative, as it tries to keep the output interesting over longer passages, which is ideal for elaborating different concepts.
Example prompt: I have a slow data-processing function in Pandas. Show me three different ways I could optimise it, e.g. through vectorisation or the use of Dask.

Model 2: Devstral – The Architect with the Super Memory

Devstral’s outstanding feature is its gigantic 128k-token context window. That corresponds to about 100 pages of text. With it, the model can keep not just a single file but an entire codebase “in mind”. Its strength lies in large-scale refactorings, the analysis of project architectures and in tasks that require consistency across many files.

Note: Using the full context window requires a considerable amount of RAM (ideally 64 GB or more).

Devstral truly plays to its strength in combination with the agent framework “OpenHands”. Going into more detail on that, however, would go beyond the scope of this article. If you want to learn more about it yourself or try out the framework, you will find all the information here: https://docs.openhands.dev/

Use Case A: Large-Scale and Consistent Refactoring

Imagine you have to replace an outdated library across an entire project or migrate from JavaScript to TypeScript. Thanks to its large context, Devstral can track relationships between different files well and thereby provide better support for such “highly interconnected” tasks.

Important: The larger the context, the longer the answer also takes. Depending on the hardware, that can sometimes take several minutes. So for such complex tasks it is best to keep an eye on the logs to make sure the model is still working and not “stuck”.

For our tests we used the quantised model available on Hugging Face from the group “Unsloth”.

Important note: At the time of publishing this article, the latest “2507” version from Unsloth had a bug related to --jinja. We therefore use the “2505” model, which does not have this bug.

./llama-server -m devstral-Q4_K_M.gguf
--threads -1
--ctx-size 131072
--cache-type-k q8_0
--n-gpu-layers 99
--seed 3407
--prio 2
--temp 0.15
--repeat-penalty 1.15 #higher than with Unsloth
--min-p 0.01
--top-k 64
--top-p 0.95
--jinja

Why these parameters?

--ctx-size 131072: The most important parameter here. We give the model the maximum context so that it understands dependencies between files.
--temp 0.15: In such a large refactoring, consistency is decisive. An extremely low temperature ensures that the model retains the style and naming conventions once chosen.
--repeat-penalty 1.15: Slightly raised in order to avoid, during very long generations across multiple files, subtle repetitions in style or in comments.
--jinja: Enables the use of the chat template in Jinja syntax that is automatically shipped with the model file.
Example prompt: Here are five interconnected classes. Refactor them to replace the singleton pattern with dependency injection. Make sure to update all instantiations across the entire code consistently.

Use Case B: Codebase Analysis and Onboarding for New Developers

A new team member has to familiarise themselves with a complex project. Devstral can act as a personal project tutor.

./llama-server -m devstral-Q4_K_M.gguf
--threads -1
--ctx-size 131072
--cache-type-k q8_0
--n-gpu-layers 99
--seed 3407
--prio 2
--temp 0.5
--repeat-penalty 1.15 #higher than with Unsloth
--min-p 0.01
--top-k 64
--top-p 0.9
--jinja

Why these parameters?

--temp 0.5: Here we need an understandable, human-like explanation, not exact code generation. A medium temperature produces a smoother, more natural style of language.
--top-p 0.9: Gives the model enough flexibility to formulate complex relationships in different ways.
Example prompt: Based on the entire codebase I give you: explain the main responsibilities of the ‘AuthenticationService’ and how it interacts with the ‘User’ and ‘Session’ models. Create a Mermaid.js diagram that visualises these relationships.

Conclusion

The step towards locally hosting AI models on my Mac was an incredibly instructive experience. Precise control over the sampling parameters, tuned to the strengths of specific models such as Qwen3 and Devstral, was the key to being able to make these models — hosted on “consumer hardware” — available to our developers.

This way, all information shared with the AI stays under our control, and it allowed us to use AI-assisted coding even in highly sensitive projects. And we make all the learnings gathered on our path to self-hosted AI available to everyone via rzfz.ai.

Sources and Further Information

This article was written with the support of AI.

Originally published at SEQIS Blog