Test Local LLMs with Promptfoo: Catch Regressions Before You Switch

Before you replace a local model, save the questions it needs to answer and the rules its answers must follow. A repeatable test can catch a missing field, a changed label or an invented device number that a casual chat would miss.

This guide uses Promptfoo with Ollama to check a small, fictional IT-ticket classifier. You will create a baseline, change one part of the setup and inspect the failures. The checks run without a paid API or a second model grading the answers.

What you need

• Windows PowerShell, macOS Terminal or a Linux terminal, plus a text editor.
• Node.js and npm. Promptfoo 0.122.2 requires Node 22.22.0 or newer; the installation guide recommends Node 24 LTS.
• Ollama installed and running, with enough free disk space and memory for your chosen model.
• A new folder for the configuration and result files. Use fictional inputs while learning.

The commands pin Promptfoo 0.122.2, released 28 August 2026. Local-model evaluation already existed before that release. This is a repeatable configuration, not a claim that the release introduced the workflow.

1. Check the local model

node --version
ollama --version
ollama list

The example uses llama3.2:3b. Its Ollama download is about 2GB; running it also needs working memory. That download size is not a RAM requirement. You can use another installed chat model by changing the provider ID below. Check its licence before using it for work.

If you want the example model and have the resources for it, download it with:

ollama pull llama3.2:3b

For help narrowing down models that fit your machine, start with our llmfit guide. Hardware fit and suitability for your task are separate questions.

2. Keep the evaluation local

Create a file named .env in your working folder, without a .txt suffix:

OLLAMA_BASE_URL=http://127.0.0.1:11434
PROMPTFOO_DISABLE_TELEMETRY=1
PROMPTFOO_DISABLE_UPDATE=1
PROMPTFOO_DISABLE_REMOTE_GENERATION=true
PROMPTFOO_DISABLE_SHARING=1
PROMPTFOO_SELF_HOSTED=1

These settings reduce optional Promptfoo network features. They are not a firewall: package and model downloads use the internet, and the privacy FAQ describes a possible one-time telemetry opt-out acknowledgement. Results are stored locally by default. Keep passwords, customer tickets and private documents out of this starter exercise.

3. Save an original test fixture

Create promptfooconfig.yaml beside .env. This deliberately small task checks three fields. Its urgency rule is an artificial teaching example, not an incident-management policy.

description: Local ticket regression checks
sharing: false

prompts:
  - |
    Classify this fictional IT ticket.
    Return only a JSON object with category, device and urgency.
    category: access for sign-in or password problems;
    hardware for physical equipment faults; otherwise other.
    device: copy the device ID exactly, or use null if absent.
    urgency: urgent only if the ticket contains "work stopped";
    otherwise routine. Do not add extra fields or Markdown.
    Ticket: {{ticket}}

providers:
  - id: ollama:chat:llama3.2:3b
    config:
      temperature: 0
      num_predict: 256

defaultTest:
  assert:
    - type: is-json
      value:
        type: object
        required: [category, device, urgency]
        additionalProperties: false
        properties:
          category:
            type: string
            enum: [access, hardware, other]
          device:
            type: [string, "null"]
          urgency:
            type: string
            enum: [routine, urgent]
    - type: javascript
      value: |
        try {
          const answer = JSON.parse(output);
          return answer.category === context.vars.expected_category
            && answer.device === context.vars.expected_device
            && answer.urgency === context.vars.expected_urgency;
        } catch {
          return false;
        }

tests:
  - description: Sign-in problem with a device ID
    vars:
      ticket: "I cannot sign in on LT-204."
      expected_category: access
      expected_device: LT-204
      expected_urgency: routine
  - description: Equipment fault with explicit urgency
    vars:
      ticket: "The screen on MON-017 is cracked; work stopped."
      expected_category: hardware
      expected_device: MON-017
      expected_urgency: urgent
  - description: General question without a device ID
    vars:
      ticket: "How do I change the editor colour theme?"
      expected_category: other
      expected_device: null
      expected_urgency: routine

The JSON-schema check validates the structure; the JavaScript check compares the fields with your answer key. Only ticket appears in the prompt template, so the expected answers are not sent as hints. The Ollama chat provider sends that prompt to the local server.

4. Run and inspect a baseline

Open a terminal in that folder and run:

npx --yes promptfoo@0.122.2 eval -c promptfooconfig.yaml --env-file .env -j 1 --no-cache --no-share -o baseline.json

This downloads the pinned CLI package if needed. The -j 1 option processes requests serially. --no-cache requests fresh answers; it does not disable normal result storage. --no-share prevents result sharing for the run. Read the terminal results and open baseline.json in your text editor to inspect the answers. The options are documented in the CLI reference.

For the first ticket, the answer key expects this object; it is an example, not a recorded model response:

{"category":"access","device":"LT-204","urgency":"routine"}

Changing the device to LT-240 still produces valid JSON, but fails the field comparison. Adding a confidence field fails the schema. Surrounding the answer with Markdown fences fails the strict JavaScript parser. Read the actual output and assertion failure before changing the prompt.

5. Compare one change at a time

Keep the baseline configuration and record the model tag and ID from ollama list, Ollama version, Promptfoo version and generation settings. Save these alongside baseline.json. Then change either the prompt, model or runtime version, keeping the test inputs and answer key fixed.

npx --yes promptfoo@0.122.2 eval -c promptfooconfig.yaml --env-file .env -j 1 --no-cache --no-share --repeat 3 -o candidate.json

With one prompt, one model and three tickets, this requests nine responses. Repeat the baseline too if you want comparable repeated runs. Temperature zero does not guarantee identical answers across machines or software versions.

Inspect the individual failures alongside the overall pass rate. Keep separate cases that you do not use while tuning the prompt. When a real, non-sensitive failure appears, add a representative case with a checked answer. For runtime changes, the backup-and-rollback approach in our llama.cpp update guide is useful background; this configuration itself uses Ollama.

Troubleshooting

• Connection refused: start Ollama and confirm .env uses 127.0.0.1:11434. There is no need to expose Ollama to your network.
• Model not found: compare the full provider tag with ollama list, or finish the optional model download.
• Node engine error: check node --version in the same terminal and update Node before retrying.
• YAML error or missing file: check indentation, use spaces rather than tabs, and confirm the terminal is in the folder containing both files.
• Answers stop midway: inspect whether the output limit was reached. If you raise num_predict, rerun the baseline with that same setting.

What does a passing result prove?

It proves that those returned answers met those checks. It does not establish general accuracy, good reasoning, safe tool use or production readiness. These are deterministic assertions, not a complete quality assessment. Three fictional tickets are a starting fixture, not a benchmark for ranking models.

For summaries or open-ended advice, review representative answers against written criteria. If you later add a model-based grader, choose its provider deliberately: that changes the resource, privacy and possibly cost assumptions. The supplied configuration has no such grader.

Source check: 6 September 2026. Examples target Promptfoo 0.122.2 and the official Ollama-provider configuration. The fictional tickets and answer key were written for this guide. See the linked installation, provider, JSON-validation, command-line and privacy documentation for the supported options.