Soup can fine-tune Llama 3.1 8B on some 4GB NVIDIA GPUs by streaming one frozen transformer layer through VRAM at a time. The LoRA adapters remain on the GPU while the base model stays in system memory or on NVMe storage. That reduces peak VRAM, but it also makes training slower and does not guarantee that every 4GB card will work.
The project’s measured example used a laptop RTX 3050 with 4GB VRAM, Windows 11 and about 17GB of system RAM. Soup reported a 3.32GB peak while training a 4-bit Llama 3.1 8B model. That is a developer result on one machine, not a result I reproduced.
What Soup layer streaming changes
Normal QLoRA keeps the quantised base model in GPU memory alongside the trainable adapters. Soup’s streaming mode instead stores the frozen base model in RAM or on disk. Each decoder layer is copied through a two-buffer VRAM pool during the forward pass and read again during backward recomputation; the full decoder stack is never resident in VRAM.
This is useful when a model narrowly exceeds your GPU’s capacity. It is not a free performance upgrade. PCIe transfers and disk reads add latency, so a GPU that can already hold the resident model should normally use standard QLoRA instead.
Before you start
- an NVIDIA GPU with a working CUDA-compatible PyTorch setup;
- 64-bit Python 3.10, 3.11 or 3.12;
- enough system RAM or fast NVMe storage for the frozen base model;
- access to Meta’s gated Llama 3.1 model and a Hugging Face token;
- a small, reviewed text dataset and plenty of time for a beta workflow.
Do not use Python 3.13 or newer. Soup 0.73 added an upper version bound because newer Python installations could resolve native dependencies that then failed at runtime.
If you would prefer a graphical workflow on better-equipped hardware, see the Unsloth Desktop installation guide. The older Llama 2 fine-tuning guide explains the broader dataset-and-adapter idea, but its commands should not be copied into this newer Soup workflow.
Step 1: Create an isolated environment
On Windows PowerShell, create a new folder and virtual environment:
mkdir soup-4gb-demo
cd soup-4gb-demo
py -3.12 -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip
mkdir data
On Linux, use the equivalent environment:
mkdir soup-4gb-demo && cd soup-4gb-demo
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
A separate environment makes it easier to pin the repaired Soup release and remove the experiment later.
Step 2: Install the current training package
Soup split its optional dependencies in version 0.71. Training now requires the train extra:
# Windows
.\.venv\Scripts\python.exe -m pip install --upgrade "soup-cli[train]>=0.73.2,<0.74"
.\.venv\Scripts\soup.exe version
.\.venv\Scripts\soup.exe doctor
On Linux, run the same commands as python -m pip, soup version and soup doctor after activating the environment.
If soup doctor does not report CUDA, use PyTorch’s official installation selector for your operating system and supported CUDA combination. Do not guess a wheel from the driver number.
Authenticate with Hugging Face before downloading the gated model:
hf auth loginEnter the token only at the protected prompt. Never save it in soup.yaml, a screenshot or a repository.
Do not follow an old tutorial that installs only soup-cli. Also avoid version 0.72.0: a key-naming bug could save adapters that loaded without actually applying their learned weights.
Step 3: Prepare a small training dataset
Start with a small, high-quality set that you have permission to use. Remove passwords, private messages, API keys, personal records and copyrighted material you cannot lawfully process.
An Alpaca-style JSONL file contains one JSON object per line:
{"instruction":"Rewrite the alert in plain English.","input":"HTTP 503 from api-2","output":"The second API server is temporarily unavailable."}
{"instruction":"Classify the incident priority.","input":"One test workstation cannot print","output":"Low priority"}
Save it as data/train.jsonl. Keep a separate validation set rather than judging the adapter on examples it has memorised. Validate the file with the format expected by your selected Soup recipe before starting a long run.
Step 4: Start from an official recipe
List the recipes included with your installed release instead of copying a stale configuration from a forum:
soup recipes list
soup recipes show llama3.1-8b-sft
soup recipes use llama3.1-8b-sft --output soup.yaml
If that exact recipe name has changed, choose the current Llama 3.1 8B SFT recipe shown by soup recipes list. Update its dataset path, output folder and Hugging Face model identifier while preserving the documented field structure.
Step 5: Enable 4-bit layer streaming
Layer streaming is configured in YAML; it is not a command-line switch. Use this conservative starting configuration for the published 4GB workflow:
base: meta-llama/Llama-3.1-8B-Instruct
task: sft
backend: transformers
data:
train: ./data/train.jsonl
format: alpaca
max_length: 512
val_split: 0.1
training:
epochs: 3
lr: 2e-4
batch_size: 1
gradient_accumulation_steps: 1
stream_layers: true
quantization: 4bit
stream_source: auto
stream_buffers: 2
stream_vram_probe: true
gradient_checkpointing: true
lora:
r: 16
alpha: 32
target_modules: auto
output: ./output
stream_source: auto lets Soup select RAM or NVMe according to the available resources. The documented streaming path currently supports text training through the Transformers backend and the listed Llama, Qwen, Mistral, Gemma and Phi-family architectures. It supports SFT plus several preference-training tasks, but it is not a universal path for every model or multimodal job.
Start with batch size 1 and a 512-token sequence length, matching the shape of Soup’s published 4GB measurement. This is still not a guarantee for another card, driver or dataset, so the measured VRAM gate remains enabled.
Step 6: Preflight before training
soup train --config soup.yaml --dry-run
This dry run validates the configuration and dataset. It does not perform Soup’s streaming memory test or architecture check. Close games, browsers with GPU acceleration and other CUDA processes before the real run. On Windows, leave extra headroom because display memory and driver allocation can reduce the VRAM available to training.
If validation fails, stop and fix the reported input or configuration problem. Moving the stream to NVMe may reduce RAM pressure, but it can make training much slower and creates sustained disk traffic. A larger GPU, shorter sequence length or smaller model may be the better choice.
Step 7: Train and keep the log
soup train --config soup.yaml
Trainer setup performs Soup’s real streaming-specific memory gate. With stream_vram_probe: true, Soup measures one SFT forward/backward step and refuses the run if it exceeds available VRAM. Do not disable that refusal to force an unsafe configuration.
Save the configuration, installed version and console log with the output adapter. They are essential if a future Soup update changes memory prediction or streaming behaviour. Do not compare your speed with an old headline number: the project’s 4GB throughput figure was produced before the later correctness repair and was not rerun on that same 4GB machine.
Step 8: Verify that the adapter really works
A completed command is not proof of a useful fine-tune. First confirm that the output folder contains adapter weights and configuration files of plausible size. Then compare the base model and tuned adapter on a held-out set of prompts.
- Use prompts that never appeared in training.
- Keep decoding settings and seeds the same where possible.
- Check task accuracy, not just whether the wording looks different.
- Test unrelated prompts for regressions and unsafe memorisation.
- Keep the original adapter until the comparison is complete.
Adapters created with Soup 0.72.0 should be treated as suspect and retrained with a current release. The project says that release could save inert adapter keys. The later NF4 repair was especially important for 32B and 72B models with very large layers; 8B and 14B paths were below the affected threshold, but using 0.73.2 or newer avoids mixing repaired and unrepaired environments.
Common problems
Python reports incompatible wheels
Check python --version. Use Python 3.10–3.12 in a fresh environment; do not try to repair a Python 3.13 environment by randomly downgrading native packages.
Training commands or dependencies are missing
Install soup-cli[train], not the base CLI alone, then rerun soup doctor.
Hugging Face returns 401 or 403
Accept Meta’s model licence on the official model page and authenticate with a token that has download access. Do not paste the token into the YAML file or a screenshot.
The 4GB GPU still runs out of memory
The published 3.32GB result is hardware- and configuration-specific. Reduce sequence length or batch size, close other GPU processes and profile again. If the model still does not fit, use a smaller model or more VRAM rather than disabling safety checks.
Training is extremely slow
Streaming trades speed for memory. NVMe backing, a laptop PCIe link, thermal throttling and small stream buffers can all add delay. If the quantised model fits normally, turn streaming off and use resident QLoRA.
The adapter changes nothing
Record the Soup version and inspect the adapter files. If it came from 0.72.0, retrain it. Otherwise verify that inference actually loads the adapter, then test held-out prompts with identical decoding settings.
Soup streaming versus the alternatives
| Method | Main advantage | Main trade-off |
|---|---|---|
| Soup layer streaming | Can train when the quantised base model does not fit in VRAM | Beta, slower and dependent on RAM/NVMe performance |
| Resident QLoRA | Simpler and usually faster | Needs enough VRAM for the quantised base model |
| Cloud fine-tuning | Access to larger GPUs without buying hardware | Ongoing cost and dataset privacy considerations |
FAQ
Can every 4GB GPU fine-tune Llama 3.1 8B?
No. Soup demonstrated one RTX 3050 Laptop setup at 3.32GB peak VRAM. Driver overhead, CUDA support, sequence length and other processes can push another system over the limit.
Does Soup train the full 8B model?
The low-memory workflow keeps the base model frozen and trains LoRA adapters. It is not a full-parameter fine-tune.
Can I use system RAM instead of a GPU?
System RAM or NVMe stores the frozen layers, but the documented workflow still transfers each active layer to a CUDA GPU for computation.
Should I use 4-bit or no quantisation?
For a 4GB target, the documented example uses 4-bit quantisation. Streaming currently documents 4bit and none; choose based on the profile rather than assuming a setting will fit.
Bottom line
Soup makes a previously impractical experiment possible on very small NVIDIA GPUs, but the honest description is “possible on a verified 4GB configuration,” not “fast on every 4GB card.” Use Python 3.10–3.12, install the training extra, pin the repaired 0.73.2 series, validate the data, enable the measured streaming gate, and let Soup refuse configurations that exceed the available VRAM. Judge the adapter on held-out prompts before relying on it.