How to Install LocalAI on Windows with Docker (4.8.1+)

If you want to install LocalAI on Windows and keep routine inference on your own hardware, Docker is the most approachable starting point. LocalAI has a compelling pitch: an OpenAI-shaped API that can run a model on your own computer. In the local-model setup below, inference prompts stay on that machine. Model downloads, optional cloud backends, agents and MCP tools can still use the network, so “local” is a configuration choice rather than a magic privacy sticker.

Version 4.8 was a large release, but it was followed a day later by 4.8.1. That small-looking patch matters: it repairs the new experimental vllm-cpp backend after an ABI change, fixes parts of its engine configuration, and corrects several web-interface and 3D Studio problems. If you are installing now, do not deliberately stop at 4.8.0.

Verified scope: I checked the commands and behaviour in this guide against LocalAI’s official 4.8.0 and 4.8.1 release notes and current documentation on 7 August 2026. I have not run this exact installation on a live Windows computer. Docker image tags and the model gallery can change, so confirm the image version in your own logs before relying on it.

What you will build

• A LocalAI container running through Docker Desktop on Windows.
• Four persistent volumes so models, backends, configuration and application data survive a container replacement.
• An API that listens only on your computer and requires a key.
• A small Qwen3 4B model installed from LocalAI’s gallery.
• An optional NVIDIA image and VRAM ceiling for supported graphics hardware.

The conservative path in this guide uses LocalAI’s established llama-cpp route. Version 4.8 introduced vllm-cpp, which can run GGUF or Safetensors without Python or PyTorch at inference time, but LocalAI labels it alpha. Interesting? Absolutely. The thing I would choose for my first dependable install? Not yet.

Choose LocalAI when…

• Choose LocalAI when you want a reusable OpenAI-compatible API, authentication, a management interface and access to several backend families.
• Choose Ollama or LM Studio when your main goal is the shortest path to downloading one model and chatting with it.
• Choose llama.cpp directly when you want a lean GGUF runtime and are happy to manage its server and configuration yourself.

For this guide, LocalAI earns the extra setup because the endpoint is intended to be reused by other applications. If all you want is a chat box, the simpler tool is often the better tool.

Prerequisites

• A Docker-supported Windows 10 or Windows 11 build with hardware virtualisation enabled.
• Docker Desktop running Linux containers through WSL 2, with WSL 2.1.5 or later.
• At least 8GB of system RAM, plus enough memory for the particular model and context you choose.
• Enough free disk space for the Docker image, backend files and model. Downloads can be several gigabytes.
• An optional supported NVIDIA GPU. Acceleration also needs a current WSL kernel and an NVIDIA driver with WSL 2 GPU-PV support. LocalAI does not require a GPU.

Open PowerShell and make sure Docker is actually responding:

docker version
docker info

If those commands cannot reach the Docker engine, start Docker Desktop and wait for it to report that the engine is running. Fix that layer before troubleshooting LocalAI; otherwise every later error is just Docker wearing a LocalAI hat.

Step 1: Choose the correct LocalAI image

The official quickstart uses moving latest tags. Pulling one today should fetch the current stable build, not create a permanent 4.8.1 pin. That is convenient for a tutorial but worth understanding. For a production deployment, select and test a versioned image from the official Docker Hub tags rather than silently accepting every future update.

On Docker Desktop for Windows, use CPU mode or an NVIDIA CUDA image with the WSL 2 backend. LocalAI also publishes AMD ROCm, Intel and Vulkan containers, but their device paths are primarily Linux or Podman routes. I have not turned their existence into pretend Docker Desktop instructions here.

This guide starts with CPU mode because it removes a whole category of driver problems:

docker pull localai/localai:latest

Step 2: Create a reusable Compose file

Create an empty folder for this setup, open it in a text editor and save the following as compose.yaml:

services:
  localai:
    image: localai/localai:latest
    container_name: local-ai
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"
    environment:
      LOCALAI_API_KEY: ${LOCALAI_API_KEY}
    volumes:
      - localai-models:/models
      - localai-backends:/backends
      - localai-configuration:/configuration
      - localai-data:/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      start_period: 60m
      interval: 1m
      timeout: 10s
      retries: 3

volumes:
  localai-models:
  localai-backends:
  localai-configuration:
  localai-data:

LocalAI exposes four persistent paths: models, downloaded backends, dynamic configuration and application data. Compose creates a named volume for each one. You can replace the application container during an upgrade without automatically throwing away model downloads, runtime settings or agent data. A volume is not a backup, though. If the installation becomes important, include these volumes in your normal backup plan.

Step 3: Generate an API key

Do not put an unprotected AI management API on your network. This command generates a random key for the current PowerShell session:

$localAiKeyBytes = New-Object byte[] 32
[Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($localAiKeyBytes)
$env:LOCALAI_API_KEY = [Convert]::ToBase64String($localAiKeyBytes)

Run $env:LOCALAI_API_KEY once if you need to copy the generated value, then store it in a password manager before closing PowerShell. Be aware that displaying it leaves the value in the terminal scrollback. The simple legacy key is fine for a private, single-user test, but LocalAI’s documentation says it has full administrator access. For multiple people, use LOCALAI_AUTH=true and the role-based user system instead of sharing one master key.

Step 4: Run LocalAI

if ([string]::IsNullOrWhiteSpace($env:LOCALAI_API_KEY)) {
  throw "LOCALAI_API_KEY is not set. Restore the saved key before starting LocalAI."
}

docker compose up -d

Run that command from the folder containing compose.yaml. Binding to 127.0.0.1 is deliberate: it makes the service reachable from this computer without advertising it to every device on the local network. The API key is still useful protection against other local applications and accidental configuration changes.

Once startup settles, confirm the Compose health check has moved to healthy:

docker inspect --format '{{json .State.Health}}' local-ai

Watch the startup log:

docker logs --tail 100 -f local-ai

Press Ctrl+C to leave the log view; that does not stop the container. Look for the listening address and any backend or storage errors. Then ask the protected API for its version:

$headers = @{ Authorization = "Bearer $env:LOCALAI_API_KEY" }
Invoke-RestMethod -Uri "http://localhost:8080/version" -Headers $headers

Confirm that the response reports 4.8.1 or newer before experimenting with the repaired vllm-cpp backend. If the moving latest tag returned an older cached image, run docker pull localai/localai:latest and deliberately recreate the container with the same four volumes.

Step 5: Open the web interface and install a model

Open http://localhost:8080. Enter the API key if prompted. In the current LocalAI interface:

1. Open the Models or Discover page.
2. Search for qwen3-4b.
3. Choose the gallery entry named qwen3-4b and select Install.
4. Wait for both the model and its matching backend to finish downloading.
5. Open Chat, select qwen3-4b and send a short test message.

LocalAI’s current quickstart recommends Qwen3 4B as a relatively small model that also supports tool calling. It is a sensible smoke test, not a promise of instant speed on every CPU. First inference is often slower because files and model state still need to load.

Step 6: Verify the OpenAI-compatible API

List the installed models from PowerShell:

$headers = @{ Authorization = "Bearer $env:LOCALAI_API_KEY" }
Invoke-RestMethod -Uri "http://localhost:8080/v1/models" -Headers $headers

Then send a chat request:

$body = @{
  model = "qwen3-4b"
  messages = @(
    @{ role = "user"; content = "Reply with one sentence confirming the local API works." }
  )
} | ConvertTo-Json -Depth 5

Invoke-RestMethod `
  -Uri "http://localhost:8080/v1/chat/completions" `
  -Method Post `
  -Headers $headers `
  -ContentType "application/json" `
  -Body $body

For an OpenAI-compatible application, use http://localhost:8080/v1 as the base URL and the LocalAI key as the API key. Some clients append /v1 themselves, so check that application’s setting before blaming the server for a double /v1/v1.

Optional: switch to an NVIDIA GPU image

First confirm Docker can see the NVIDIA GPU. Docker Desktop needs working Windows drivers and GPU support:

docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi

If that works, edit the localai service in compose.yaml. Change its image and add the two highlighted settings at the same indentation level as ports and environment:

services:
  localai:
    image: localai/localai:latest-gpu-nvidia-cuda-12
    gpus: all
    environment:
      LOCALAI_API_KEY: ${LOCALAI_API_KEY}
      LOCALAI_VRAM_BUDGET: 80%

Keep the remaining ports, volumes and health-check lines from the original file, then apply the change:

if ([string]::IsNullOrWhiteSpace($env:LOCALAI_API_KEY)) {
  throw "LOCALAI_API_KEY is not set. Restore the saved key before starting LocalAI."
}

docker compose pull
docker compose up -d

Compose recreates the container but retains the named volumes. Do not add -v to docker compose down during a routine update; that option removes declared volumes and can destroy persistent LocalAI data.

Use the new VRAM budget

LocalAI 4.8 can treat only part of the detected graphics memory as available. That is useful on a desktop where Windows, a browser and perhaps a game also want some of the GPU. You can express the ceiling as a percentage or an amount:

-e LOCALAI_VRAM_BUDGET=80%
-e LOCALAI_VRAM_BUDGET=12GB

The budget is a ceiling, not extra memory. Setting 12GB on an 8GB card does not create another 4GB, and a percentage above 100 is invalid. The setting helps LocalAI make allocation decisions, but estimates still vary by backend, quantisation and context size. Keep an eye on actual usage with nvidia-smi.

What changed in LocalAI 4.8.1?

• The new vllm-cpp backend was updated for ABI v10 so it can load again.
• More of the vllm-cpp engine configuration is now passed through correctly.
• The React traces interface no longer hits the reported rendering crash.
• 3D Studio results and history were restored.
• CLI socket activation now ignores a half-populated activation environment.
• VRAM processing now contains malformed GGUF metadata instead of letting it break the workflow.

The practical conclusion is simple: install 4.8.1 or newer, use llama-cpp first, and treat vllm-cpp as an advanced experiment. LocalAI’s own 4.8.0 notes say not to depend on that alpha backend yet.

Troubleshooting

Docker says the container name is already in use

Check the existing container before deleting anything:

docker ps -a --filter "name=local-ai"
docker start local-ai

If it is an obsolete container, inspect its mounts and configuration, then deliberately stop and replace it. Do not remove a container merely because a tutorial reused its name.

Port 8080 is already allocated

Find the program already using the port or map LocalAI to another local port, such as -p 127.0.0.1:8081:8080. The browser URL and API base then become http://localhost:8081.

The API returns 401 Unauthorized

Make sure the request sends the same key that was present when the container started. A new PowerShell window will not automatically inherit the temporary environment variable. Retrieve the key from your password manager; do not disable authentication to make the error disappear.

The model download stalls or the disk fills up

Check Docker Desktop’s disk allocation, host free space and docker logs local-ai. Start with one small model. Repeatedly clicking Install can create several large jobs without making the first one faster.

The model runs out of memory

• Choose a smaller or more heavily quantised model.
• Reduce context length and parallel requests.
• Add LOCALAI_MAX_ACTIVE_BACKENDS: "1" under the Compose environment section, then run docker compose up -d to recreate the container with one active backend.
• Use the VRAM budget to leave operating-system headroom.
• Confirm that the selected backend is really using the intended GPU.

vllm-cpp fails to load

Confirm the log shows LocalAI 4.8.1 or newer; 4.8.1 specifically repaired the ABI break affecting this backend. If it still fails, return to a gallery model using llama-cpp. Alpha software is allowed to be fascinating and inconvenient at the same time.

The web interface opens but a page crashes

Check the LocalAI version and browser console, then refresh after upgrading past 4.8.0. Version 4.8.1 includes fixes for a traces rendering crash and missing 3D Studio history. Preserve logs before replacing the container if the problem continues.

Frequently asked questions

Does LocalAI need Python?

Not for the Docker setup in this guide. Some LocalAI backends use Python internally, but the container handles their environment. The new alpha vllm-cpp route is specifically designed to serve supported models without Python or PyTorch at inference time.

Can LocalAI run without a GPU?

Yes. Use localai/localai:latest and start with a small quantised model. Expect CPU speed to depend heavily on your processor, model size and context.

Can I expose LocalAI to my LAN or the internet?

Technically yes, but the localhost binding in this guide intentionally does not. Remote access needs authentication, TLS, network restrictions and careful handling of management endpoints. For multiple users, use LocalAI’s role-based authentication rather than a shared legacy key. A raw port-forward from your router is not a deployment plan.

Is LocalAI a drop-in OpenAI replacement?

It implements familiar OpenAI-compatible endpoints, which makes many integrations straightforward. Compatibility is not a promise that every hosted model feature, parameter or response will behave identically. Test the particular application and model you care about.

Should I use LocalAI 4.8.0 or 4.8.1?

Use 4.8.1 or a newer stable release. The patch fixes several regressions introduced alongside the large 4.8.0 feature set.

Related Lachie’s Lifestyle guides

• How to connect llama.cpp to MCP for a lighter, more direct local-model route.
• How to install Code Llama locally on Windows for a model-specific GGUF workflow.
• How to install Llama 2 locally on Windows for historical context on the install format that first drew readers to this site.

Official sources

• LocalAI 4.8.1 release notes
• LocalAI 4.8.0 release notes
• LocalAI installation guide
• LocalAI container and persistent-storage guide
• LocalAI quickstart
• LocalAI VRAM and memory management
• LocalAI authentication and authorisation
• Docker Desktop GPU support for Windows
• LocalAI source repository

Source check completed 7 August 2026. Re-check the official image tags and release notes before installing because the latest tag moves.