Local AI is impressive right up until it needs to do something outside the chat box. Ask a model to organise a folder and it will cheerfully explain how you can organise it yourself. Useful, but not exactly magic.
This llama.cpp MCP setup changes that. A local GGUF model can connect to compatible tools and use them as part of a real task, while the model itself keeps running on your own machine.
That is the exciting part. It is also where I stop being casual about permissions. Giving a model access to a tool is closer to handing someone a key than enabling another chat feature, so this guide starts with one empty allowlisted test folder—not your Documents folder, work repository or entire drive.
If you only want private local chat, you can safely skip MCP. If you want a model to inspect a test folder or call a narrowly defined tool, this is where it becomes interesting.
Lachie’s take: What I like is that this adds useful tool access without requiring a large agent framework. What deserves caution is the permission boundary: the setup is only as safe as the folders and commands you expose.
A quick honesty note: I checked the commands and behaviour against the merged llama.cpp change, current llama-server documentation and the official MCP filesystem server instructions. I have not run every operating-system path on fresh hardware, so I will keep documented behaviour separate from anything that still needs real-world testing.
What changed in llama.cpp?
The merged implementation adds stdio-based MCP support to llama-server. It reads a Cursor-compatible JSON configuration, starts the configured MCP processes, discovers their tools and makes those tools available through llama-server. MCP tools are exposed as <server>_<tool>, for example filesystem_read_text_file.
Think of llama.cpp as the brain and MCP as a labelled tool drawer. The model can be shown the tools it needs without being handed the keys to the entire workshop. I like this direction because it adds a practical bridge between “the model knows what to do” and “the model can use a tool to do it.”
One important distinction: this is an MCP client inside llama-server. It does not magically turn every OpenAI-compatible application into an MCP client. The simplest place to try it is llama.cpp’s own Web UI at http://127.0.0.1:8080.
This backend MCP support is still new and upstream describes it as experimental. I would not use it with untrusted servers or untrusted users; this is a controlled local-workflow feature, not something to casually expose on the internet.
Before you start
The setup has a few moving parts, but none of them is especially mysterious. Think of llama.cpp as the engine, the model as the driver and the MCP server as the carefully limited set of tools in the boot. Check each part before connecting them.
- This guide uses llama.cpp
b10142or newer. MCP first appeared inb10133, but b10142 includes the immediate follow-up subprocess changes and is a cleaner baseline for this walkthrough. - A GGUF instruct model that can use tools reliably.
- Node.js and
npxfor the filesystem example. - A terminal or PowerShell window.
- One non-sensitive test directory that the MCP server is allowed to access.
If you are new to local models, first read our local Llama installation guide. Its model choices are older, but it explains the basic local-model workflow. You can also browse the newer AI guides hub.
Step 1: Update llama.cpp
Download a current binary from the official llama.cpp releases page, or rebuild the latest source. Confirm the server binary is present:
llama-server --version
llama-server --helpOn Windows, the executable is normally llama-server.exe. If --help does not show --mcp-servers-config, the build predates the MCP merge and needs to be updated.
Step 2: Create an allowlisted MCP test folder
Now comes the useful—and potentially dangerous—part. Start with an empty test folder. If the model misunderstands an instruction, the most interesting thing it should be able to damage is a disposable text file. Do not point your first test at Documents, a cloud-drive root, a work repository or anything containing credentials.
Windows PowerShell:
New-Item -ItemType Directory -Path "$env:USERPROFILE\mcp-test"macOS or Linux:
mkdir -p "$HOME/mcp-test"Step 3: Create the MCP configuration
Create a file named mcp.json. This small JSON file is where you decide which MCP process may start and which folder it can see. Replace the example path with the exact allowlisted test-folder path on your computer; this is one place where copying a path almost correctly is not good enough.
Windows configuration:
{
"mcpServers": {
"filesystem": {
"command": "cmd",
"args": [
"/c",
"npx",
"-y",
"@modelcontextprotocol/server-filesystem",
"C:/Users/YOUR_NAME/mcp-test"
]
}
}
}The official server instructions use cmd /c on Windows so that npx.cmd launches correctly. JSON also requires each backslash in a Windows path to be doubled.
macOS or Linux configuration:
{
"mcpServers": {
"filesystem": {
"command": "npx",
"args": [
"-y",
"@modelcontextprotocol/server-filesystem",
"/absolute/path/to/mcp-test"
]
}
}
}The folder allowlist limits what the filesystem MCP tools expose to the model, but it is not operating-system isolation. The spawned Node process still runs with your normal user-account permissions.
Use a disposable folder containing no credentials or irreplaceable files. The server’s write_file, edit_file and move_file tools can make destructive changes. If you need stronger isolation, run the experiment inside a container or virtual machine.
Step 4: Start llama-server with MCP
This is the point where the pieces meet. Use your own GGUF model and configuration paths. I keep --jinja explicit here so the template behaviour is visible, although it is already enabled by default in the release used for this guide. It is not sufficient on its own: the model still needs reliable tool calling and a compatible chat template. The --host 127.0.0.1 setting keeps the first experiment on your own machine, which is where I would leave it until the setup is working and properly secured.
Windows:
llama-server.exe -m "C:/Models/your-model.gguf" --jinja --mcp-servers-config "C:/path/to/mcp.json" --host 127.0.0.1macOS or Linux:
./llama-server -m "/path/to/your-model.gguf" --jinja --mcp-servers-config "/path/to/mcp.json" --host 127.0.0.1By default, llama-server listens on port 8080. Wait until the model and MCP server finish loading, then open http://127.0.0.1:8080 in your browser.
Step 5: Test the MCP tool safely
Do not jump straight to a complicated automation. Boring tests are excellent here because they make permission problems obvious and leave very little room for the model to bluff.
- Put one harmless text file in
mcp-test. - Open llama.cpp’s Web UI.
- Start with a read-only prompt: “List the files in the allowed directory and summarise the text file. Do not create, change, move, or delete anything.”
- Watch which tool is called and inspect the result. Do not assume the UI will show an approval prompt before a write operation.
- Only test file creation after read access works and only inside the allowlisted test folder.
A small model may describe a tool call without producing the structured call llama-server expects. That is a model or chat-template limitation, not necessarily an MCP connection failure. Try a tool-capable instruct model and keep --jinja enabled.
Security rules worth keeping
Local does not automatically mean safe. The model may be running on your computer, but every MCP server is still a new permission boundary. I would treat the folder list and command list with the same care as a service account: give it the smallest job that works.
- Use trusted MCP servers only. Each server is a child process running with the permissions of your user account.
- Allow the smallest possible directory. Never begin with an entire drive, home folder or user profile.
- Keep llama-server on localhost. CORS settings are not authentication. Do not change the host to
0.0.0.0unless you have deliberately secured the network boundary. - Prefer a config file. Inline JSON containing API keys may appear in the operating system’s process list.
- Protect and exclude the config. Restrict its file permissions and never commit a secret-bearing MCP config; add it to
.gitignore. - Do not rely on an approval prompt. A filesystem tool may be able to write, edit or move files without the Web UI asking first. Use disposable data, backups and least privilege.
- Pin what you trust.
npx -ycan fetch the newest package. For a repeatable setup, review and pin a specific package version.
Troubleshooting llama.cpp MCP setup
If the first attempt fails, resist the urge to grant broader permissions or reinstall everything. An MCP problem is usually a path, process, configuration or tool-template issue. Work through those separately and keep the allowlist narrow while you diagnose it.
Unknown option: –mcp-servers-config
Your llama.cpp build is too old. This guide uses b10142 or newer. Confirm the installed build with llama-server --version, then check llama-server --help again.
MCP config reports no servers
Validate the JSON and make sure the top-level property is exactly mcpServers. Curly quotes copied from a document editor will also break JSON.
npx does not start on Windows
Run node --version and npx --version first. If either command fails, install Node.js and open a fresh terminal so the updated PATH is available. On Windows, keep the cmd plus /c configuration shown above.
The filesystem server exits immediately
Check that every allowed directory exists and that the account running llama-server can access it. The official filesystem server requires at least one allowed directory.
Tools appear, but the model never uses them
First check the startup log for MCP warmup, then open http://127.0.0.1:8080/tools as a diagnostic. That endpoint is internal and may change, but it should reveal whether tools were registered; the Web UI may group MCP tools with its built-in tools. If the tools exist but are never called, the likely problem is the model or chat template. Some models need an explicit --chat-template or --chat-template-file. Test with a direct request such as “Use the filesystem tool to list the allowed directory.”
Frequently asked questions
Does llama.cpp now support every MCP server?
The new integration targets MCP servers that communicate over stdio and can be launched from the JSON configuration. It does not mean every transport, server, or downstream client combination has been verified.
Can I use several MCP servers?
Yes. Add more named entries under mcpServers. Start with one, verify it, and add others individually so failures and permissions remain easy to diagnose.
Is an internet connection required?
Local inference and an already-installed local MCP server can run without a cloud model. This example uses npx -y, which needs Node/npm and normally downloads the latest package on its first run. For reproducibility, review and pin a specific version. Individual MCP servers may also call online services, so check each server’s documentation and code.
Is MCP access automatically safe because the model is local?
No. Local processing can improve privacy, but a tool can still read or change local data within the permissions you grant. Treat MCP configuration as privileged access.
Sources and next steps
- Merged llama.cpp stdio MCP implementation
- Official llama-server documentation
- Official filesystem MCP server instructions
- llama.cpp release b10142
- llama.cpp security guidance
My take: native MCP support makes llama.cpp considerably more useful than a local chat box. It creates a path from “the model knows what to do” to “the model can use a tool to do it,” without automatically moving the whole workflow into a cloud service.
That does not make local tool use harmless. Treat every MCP server as a new permission boundary: expose the smallest possible working directory, test with disposable data, read the server’s documentation and expand access only when there is a clear reason.
For a first experiment, deliberately boring is good. Let the model list files in the test folder, read one harmless text file and stop. Once that works, add permissions one folder and one tool at a time.