Running Local LLMs on a Strix Halo Laptop
An AMD Ryzen AI Max+ 395 laptop with its integrated Radeon 8060S GPU and unified memory can run open-weight models from 4B parameters up to a 120B mixture-of-experts model, no cloud GPU needed. gpt-oss-120b generates around 35-40 tokens per second, and the 20B and 4B models are faster still. That's enough for the soviet.recipes project: this hardware can host a model large enough to attempt vision-language OCR locally.
Why I benchmarked this
The soviet.recipes project needs to pull text off 224 scanned pages of a Soviet cookbook. My Docling proof-of-concept showed that traditional OCR loses text wherever a page curves or tilts away from the camera, and that loss can't be recovered downstream. A vision-language model (VLM) that reads the page image directly, such as a Qwen VLM paired with Docling, is the next thing to try.
Page curvature isn't the only reason. Working on a separate project, my bank statement processor, I found Docling's table extraction dropped rows even on clean, digitally-generated PDFs that needed no OCR at all. Those are two different document types with different failure modes, and both led to the same conclusion: Docling on its own is inconsistent enough that I can't rely on it to hit the soviet.recipes quality bar.
Running a VLM through a cloud API is easy, but I didn't want to pay for API calls indefinitely if my own laptop could host a model large enough to be worth the effort. So before committing to that path, I measured what the machine can do with local models.
The models below are text-only. They tell me the ceiling on memory and compute, since a VLM adds image-encoding cost per page and will run slower still. If the laptop can't comfortably host a large text model, a VLM is a non-starter.
The machine
- CPU/GPU: AMD Ryzen AI Max+ 395 "Strix Halo" APU
- GPU: integrated Radeon 8060S (gfx1151, RDNA 3.5), which llama.cpp detects as both a ROCm and a Vulkan device
- Memory: unified memory shared between CPU and GPU. The same pool holds the weights regardless of where they run, which is what lets a 59 GiB model load at all. See AMD's writeup on variable graphics memory for how the split works.
Setup
Two separate GPU backends can drive the 8060S, and I wanted numbers for both rather than guessing which one to use.
ROCm is AMD's own GPU compute stack, built around the HIP runtime API. It's the path AMD documents for running gpt-oss on Ryzen AI hardware, and the one a card's own vendor tuning would normally be expected to win on. Vulkan is a cross-vendor graphics and compute API maintained by the Khronos Group, the same standards body behind OpenGL. It targets GPUs from any vendor rather than AMD hardware specifically, and llama.cpp supports it as an alternative to writing a backend per vendor.
Each llama.cpp release ships a separate prebuilt zip per backend for Windows: llama-<build>-bin-win-vulkan-x64.zip and llama-<build>-bin-win-hip-radeon-x64.zip, alongside CPU and CUDA builds for other hardware. There's no single binary that does both, and no runtime flag to switch. Whichever zip you download determines the backend, so I downloaded and ran both for this comparison:
- ROCm / HIP (the Radeon build)
- Vulkan
Before settling on llama.cpp for these numbers, I also tried gpt-oss-20b and gpt-oss-120b through Ollama, which picks a backend for you rather than shipping one binary per backend. ollama ps during those runs showed gpt-oss-20b fully on GPU (100%) and gpt-oss-120b split 81% GPU / 19% CPU, both at a 131072-token context, well beyond the 4096 tokens I used in llama-bench. gpt-oss-20b generated at 38.69 tokens/sec despite the full GPU offload, a bit over half of what either llama.cpp backend produced for the same model (65.35 ROCm, 68.45 Vulkan). gpt-oss-120b generated at 17.04 tokens/sec, but that number mixes in CPU inference from the partial offload, so it isn't a fair read on backend speed alone. Between the confirmed full-GPU gap on the 20b model and the extra confound on the 120b model, I stuck with llama.cpp and llama-bench for the rest of this benchmark instead of tuning Ollama further.
Models tested:
| Model | Quantization | On-disk size |
|---|---|---|
| Qwen3-4B | Q8_0 | 3.98 GiB |
| gpt-oss-20b | MXFP4 | 11.27 GiB |
| gpt-oss-120b | MXFP4 | 59.02 GiB |
Why the 120B model is even viable
The gpt-oss models are mixture-of-experts (MoE) and ship in OpenAI's native 4-bit MXFP4 format. gpt-oss-120b has 117B total parameters but only about 5.1B are active per token; gpt-oss-20b has 21B total with 3.6B active. Most of the weights sit idle for any given token, so generation cost tracks the small active set rather than the full parameter count, which is why a 59 GiB model still generates at laptop-usable speed. The full 59 GiB still has to fit in memory, which is where the unified memory matters.
Benchmark command
Every run used the same llama-bench invocation, changing only the model path:
llama-bench -m <model.gguf> -p 4096 -n 512 -b 512 -ub 512 -t 16 -ngl 99 -sm row --flash-attn 1
-p 4096measures prompt processing: how fast the model reads a 4096-token prompt.-n 512measures token generation: how fast it produces 512 new tokens.-ngl 99offloads all layers to the GPU.
Full raw output is in the appendix.
Results
Prompt processing (tokens/sec, higher is better)
| Model | ROCm | Vulkan |
|---|---|---|
| Qwen3-4B Q8_0 | 432.83 | 852.58 |
| gpt-oss-20b MXFP4 | 439.06 | 1233.31 |
| gpt-oss-120b MXFP4 | 142.01 | 469.66 |
Token generation (tokens/sec, higher is better)
| Model | ROCm | Vulkan |
|---|---|---|
| Qwen3-4B Q8_0 | 41.24 | 44.57 |
| gpt-oss-20b MXFP4 | 65.35 | 68.45 |
| gpt-oss-120b MXFP4 | 35.53 | 40.07 |
Vulkan beat ROCm in every test on this hardware, and by a wide margin on prompt processing (2-3x faster). I expected AMD's own ROCm stack to lead here, not Vulkan. Generation speeds are closer between the backends, though Vulkan is still ahead; the big gap shows up in prompt processing, which matters when feeding long inputs like full page transcripts. gpt-oss-20b is the sweet spot on this machine, with the fastest generation (68 t/s) and the fastest prompt processing of the three.
gpt-oss-120b generated at 40 t/s on Vulkan in llama-bench, even at 59 GiB.
Verdict: is local inference feasible?
Local inference is feasible on this laptop for the soviet.recipes work, and Vulkan is the backend to use since ROCm was slower in every test here. These are text-model numbers, so they mark a ceiling on memory and compute; a vision-language model adds per-page image encoding and will run slower. The memory headroom and compute here are enough to host a sizeable VLM locally.
What's next
The next soviet.recipes experiment is to run a Qwen VLM locally, feed the cookbook page images through Docling, and score the output against the same rubric used for the plain-OCR runs. This benchmark confirms the hardware can carry that experiment.
Appendix: Raw benchmark logs
Raw llama-bench output captured during the experiment. Local paths have been scrubbed to C:\Users\user.
ROCm results
(base) PS C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64> llama-bench -m "C:\Users\user\AppData\Local\llama.cpp\Qwen_Qwen3-4B-GGUF_Qwen3-4B-Q8_0.gguf" -p 4096 -n 512 -b 512 -ub 512 -t 16 -ngl 99 -sm row --flash-attn 1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon(TM) 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
load_backend: loaded ROCm backend from C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64\ggml-hip.dll
load_backend: loaded RPC backend from C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64\ggml-rpc.dll
load_backend: loaded CPU backend from C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64\ggml-cpu-icelake.dll
| model | size | params | backend | ngl | n_batch | sm | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ----: | -: | --------------: | -------------------: |
| qwen3 4B Q8_0 | 3.98 GiB | 4.02 B | ROCm | 99 | 512 | row | 1 | pp4096 | 432.83 ± 5.90 |
| qwen3 4B Q8_0 | 3.98 GiB | 4.02 B | ROCm | 99 | 512 | row | 1 | tg512 | 41.24 ± 2.88 |
build: 5ceed6242 (7248)
(base) PS C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64> llama-bench -m "C:\Users\user\.lmstudio\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf" -p 4096 -n 512 -b 512 -ub 512 -t 16 -ngl 99 -sm row --flash-attn 1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon(TM) 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
load_backend: loaded ROCm backend from C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64\ggml-hip.dll
load_backend: loaded RPC backend from C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64\ggml-rpc.dll
load_backend: loaded CPU backend from C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64\ggml-cpu-icelake.dll
| model | size | params | backend | ngl | n_batch | sm | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ----: | -: | --------------: | -------------------: |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | ROCm | 99 | 512 | row | 1 | pp4096 | 439.06 ± 7.22 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | ROCm | 99 | 512 | row | 1 | tg512 | 65.35 ± 1.08 |
build: 5ceed6242 (7248)
(base) PS C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64> .\llama-bench.exe `
>> -m "C:\Users\user\.lmstudio\models\lmstudio-community\gpt-oss-120b-GGUF\gpt-oss-120b-MXFP4-00001-of-00002.gguf" `
>> -p 4096 `
>> -n 512 `
>> -b 512 `
>> -ub 512 `
>> -t 16 `
>> -ngl 99 `
>> -sm row `
>> --flash-attn 1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon(TM) 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
load_backend: loaded ROCm backend from C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64\ggml-hip.dll
load_backend: loaded RPC backend from C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64\ggml-rpc.dll
load_backend: loaded CPU backend from C:\Users\user\Downloads\llama-b7248-bin-win-hip-radeon-x64\ggml-cpu-icelake.dll
| model | size | params | backend | ngl | n_batch | sm | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ----: | -: | --------------: | -------------------: |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | ROCm | 99 | 512 | row | 1 | pp4096 | 142.01 ± 12.81 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | ROCm | 99 | 512 | row | 1 | tg512 | 35.53 ± 3.88 |
build: 5ceed6242 (7248)
Vulkan results
(base) PS C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64> llama-bench -m "C:\Users\user\AppData\Local\llama.cpp\Qwen_Qwen3-4B-GGUF_Qwen3-4B-Q8_0.gguf" -p 4096 -n 512 -b 512 -ub 512 -t 16 -ngl 99 -sm row --flash-attn 1
load_backend: loaded RPC backend from C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon(TM) 8060S Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64\ggml-cpu-icelake.dll
| model | size | params | backend | ngl | n_batch | sm | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ----: | -: | --------------: | -------------------: |
| qwen3 4B Q8_0 | 3.98 GiB | 4.02 B | Vulkan | 99 | 512 | row | 1 | pp4096 | 852.58 ± 15.72 |
| qwen3 4B Q8_0 | 3.98 GiB | 4.02 B | Vulkan | 99 | 512 | row | 1 | tg512 | 44.57 ± 0.38 |
build: 5ceed6242 (7248)
(base) PS C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64> llama-bench -m "C:\Users\user\.lmstudio\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf" -p 4096 -n 512 -b 512 -ub 512 -t 16 -ngl 99 -sm row --flash-attn 1
load_backend: loaded RPC backend from C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon(TM) 8060S Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64\ggml-cpu-icelake.dll
| model | size | params | backend | ngl | n_batch | sm | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ----: | -: | --------------: | -------------------: |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | Vulkan | 99 | 512 | row | 1 | pp4096 | 1233.31 ± 26.61 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | Vulkan | 99 | 512 | row | 1 | tg512 | 68.45 ± 0.16 |
build: 5ceed6242 (7248)
(base) PS C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64> .\llama-bench.exe -m "C:\Users\user\.lmstudio\models\lmstudio-community\gpt-oss-120b-GGUF\gpt-oss-120b-MXFP4-00001-of-00002.gguf" -p 4096 -n 512 -b 512 -ub 512 -t 16 -ngl 99 -sm row --flash-attn 1
load_backend: loaded RPC backend from C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon(TM) 8060S Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\Users\user\Downloads\llama-b7248-bin-win-vulkan-x64\ggml-cpu-icelake.dll
| model | size | params | backend | ngl | n_batch | sm | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ----: | -: | --------------: | -------------------: |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | Vulkan | 99 | 512 | row | 1 | pp4096 | 469.66 ± 7.81 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | Vulkan | 99 | 512 | row | 1 | tg512 | 40.07 ± 5.70 |
build: 5ceed6242 (7248)