Site icon Lachie's Lifestyle

How to Run Qwen3.8-27B Locally on Windows with llama.cpp

Lachie
Qwen3.8-27B local llama.cpp setup diagram for Windows

Qwen3.8-27B is now released, and Windows users can run a quantised version locally with llama.cpp. The simplest documented route is to install llama.cpp with WinGet and let llama-server download Unsloth’s 17.9 GB UD-Q4_K_XL GGUF directly from Hugging Face.

The model itself is a dense 27-billion-parameter vision-language model. Qwen lists a native context length of 262,144 tokens, switchable thinking, three reasoning-effort levels and image/video understanding. Do not start by allocating the full context window, though. Model weights, the KV cache and runtime overhead all compete for memory.

Qwen3.8-27B release status

Item Verified status
Official weights Public in Qwen’s Hugging Face organisation, including an FP8 repository.
Licence Apache 2.0 on the official FP8 model card and the Unsloth GGUF repository.
Model type 27B dense causal language model with a vision encoder.
Context 262,144 tokens natively; Qwen documents a 1,000,000-token YaRN extension for supported engines such as vLLM, SGLang and TokenSpeed. This guide does not configure that extension in llama.cpp.
Thinking controls Thinking is on by default. Qwen documents xhigh, medium and low reasoning effort.
Local GGUF Available from Unsloth in 2-bit through 8-bit sizes. This is a third-party quantisation.
Hands-on result Not tested by me yet.

Qwen’s benchmark table reports large gains over Qwen3.6-27B, but those numbers were produced by Qwen. Treat them as vendor results until independent testing uses comparable prompts, runtimes and quantisations.

What you need

A 17.9 GB model file does not mean that 18 GB of VRAM is sufficient for every configuration. A larger context window needs a larger KV cache. llama.cpp can split work between CPU and GPU, but spilling more of the model into normal RAM generally reduces speed.

Choose a Qwen3.8-27B quant

These are published file sizes from the Unsloth model card. They describe the model file, not total runtime memory.

Quant Published size Practical use
UD-IQ3_XXS 11.9 GB Lower-memory experiment; expect a larger quality compromise.
Q3_K_M 13.8 GB A middle option when the recommended Q4 file will not fit.
UD-Q4_K_XL 17.9 GB The starting point used in Unsloth’s llama.cpp examples.
Q5_K_M 19.8 GB More weight precision with a higher memory requirement.
Q6_K 22.9 GB For systems with more headroom.
Q8_0 29 GB Large local file; leave substantial extra memory for context.

I would begin with UD-Q4_K_XL if it fits comfortably. Dropping straight to a 2-bit file saves space, but that can make it harder to tell whether an odd answer comes from the model, the prompt or aggressive quantisation.

How to run Qwen3.8-27B locally on Windows

1. Install llama.cpp

Open PowerShell or Windows Terminal and use the package ID from the official llama.cpp installation documentation:

winget install --id ggml.llamacpp --exact

If llama.cpp is already installed, update it before testing this release-day model:

winget upgrade --id ggml.llamacpp --exact

Close and reopen the terminal after installation. This refreshes the command path.

2. Check that the command is available

llama-server --version

If Windows says that llama-server is not recognised, see the troubleshooting section before downloading the model.

3. Start the local server and download the GGUF

llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL --ctx-size 32768

The first run downloads the GGUF from Hugging Face. The terminal should show download and model-loading progress. Keep that terminal open while you use the model.

The -hf loading option comes from llama.cpp and points it at Unsloth’s GGUF repository. It uses a community conversion rather than Qwen’s FP8 weights because the GGUF is much more practical for a typical llama.cpp workstation.

For text-only use, add --no-mmproj to avoid downloading and loading the vision projector. Keep the projector enabled if you intend to test image input.

4. Open the local web interface

Once the server is ready, open:

http://127.0.0.1:8080

Start with a short text prompt such as:

Write a PowerShell command that lists the ten largest files in a folder. Explain each part of the command.

Text is the right first test. Qwen3.8-27B is multimodal, but image and video support also depends on the runtime build, interface and model packaging. Do not assume every GGUF front end exposes every capability from Qwen’s full model.

5. Reduce the context before troubleshooting memory

Qwen supports a very long native context, but allocating the maximum on the first run is unnecessary. If your llama.cpp build exposes context controls, begin around 8K to 32K and raise the value only when a real task needs it.

For a fixed 32K context, current llama.cpp builds commonly accept:

llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL --ctx-size 32768

6. Adjust reasoning effort if the model thinks for too long

Qwen3.8 uses xhigh reasoning by default. That can be wasteful for simple prompts. Qwen documents xhigh, medium and low; support for passing the setting depends on the local runtime and chat template.

With a recent llama.cpp build, try:

llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL --ctx-size 32768 --chat-template-kwargs '{"reasoning_effort":"medium"}'

If that flag is rejected, update llama.cpp and check the current help output. Do not hide long thinking by simply cutting the final output limit: that can stop the model before it provides an answer.

Optional: run the same GGUF with Ollama

Hugging Face documents this repository-and-quant syntax for loading GGUF files with Ollama:

ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

This does not make the GGUF an official Ollama build. It tells Ollama to load a Hugging Face model from Unsloth’s repository. If the command fails, update Ollama and verify that your installed version supports Hugging Face GGUF references.

Recommended sampling settings

When your interface exposes these controls, Qwen recommends the following values:

Mode Temperature Top-p Top-k Presence penalty
Thinking 1.0 0.95 20 0.0
Non-thinking 0.7 0.80 20 1.5

Those are Qwen’s suggested starting values, not universal best settings. Change one setting at a time and keep the prompt fixed if you want a meaningful comparison.

Troubleshooting

llama-server is not recognised

Close every open PowerShell window, start a new one and run llama-server --version again. If it still fails, confirm WinGet completed successfully with winget list --id ggml.llamacpp --exact. Reinstall only if the package is missing.

The model closes while loading or reports an allocation error

Reduce the context size first. If it still will not load, select a smaller quant such as Q3_K_M or UD-IQ3_XXS. Close GPU-heavy programs before retrying. A working image-generation application or game can occupy enough VRAM to force part of the model onto the CPU.

The first launch appears stuck

The first multimodal load can download the 17.9 GB Q4 model plus an approximately 0.93 GB vision projector. Check the terminal for transfer progress, available disk space and Hugging Face errors. Later launches should reuse the cached files rather than download them again. Add --no-mmproj for a text-only first run.

Responses are extremely slow

Check whether the model is partly running on the CPU, then reduce context and close other GPU workloads. Update llama.cpp before changing several flags at once. Qwen also enables high reasoning effort by default, so a long wait can be generation rather than loading.

The model keeps thinking but never answers

Switch reasoning effort from xhigh to medium or low where your front end supports it. Also confirm you are using the repository’s current chat template. Early community reports are useful warning signs, but the runtime’s documented controls are the safer fix.

Image input does not work

Verify that your current llama.cpp build and interface support Qwen3.8 vision input with this GGUF package. The official model is multimodal, but a text-generation command proving that chat works does not by itself prove that the local UI has loaded the vision path.

Frequently asked questions

Is Qwen3.8-27B officially released?

Yes. Qwen’s model repositories and model card are now public. The status information in the original version of this tracker is no longer current.

Is the Unsloth GGUF official?

No. Qwen publishes the base and FP8 repositories. Unsloth publishes a third-party GGUF conversion with practical llama.cpp and Ollama commands.

How much VRAM does Qwen3.8-27B need?

There is no single correct number. The chosen quant, context size, KV-cache precision, runtime and GPU offload all matter. The recommended Q4 model file alone is 17.9 GB, so leave memory headroom rather than treating that as the complete requirement.

Can I run Qwen3.8-27B with 16 GB of memory?

A smaller 3-bit model file can fit inside 16 GB, but runtime overhead and context still need space. CPU/GPU splitting may let the model start, although that does not guarantee useful speed. I would not promise a good 16 GB experience without testing the exact hardware.

Does Qwen3.8-27B support images and video?

The official model does. Local support depends on the inference framework and model package, so verify multimodal input separately after text chat works.

Should I use Q4, Q5 or Q8?

Start with UD-Q4_K_XL because it is the quant used in Unsloth’s setup examples. Move upward only if you have enough headroom and a reason to trade more memory for weight precision. Move downward if the Q4 model cannot load after reducing context.

What to do next

Use the Q4 GGUF and a modest context size for the first clean test. Once it answers reliably, change one variable at a time: context, reasoning effort, then quantisation. That sequence makes failures much easier to diagnose than starting with a 256K context and several experimental flags.

If you want to connect the local server to tools, read my llama.cpp MCP setup guide. For a broader OpenAI-compatible local stack, see how to install LocalAI on Windows with Docker.

Update log

Primary sources

Exit mobile version