bitsandbytes Installation Errors: What's Actually Happening and How to Fix It

TL;DR

bitsandbytes is the library that makes QLoRA fine-tuning possible by enabling 4-bit and 8-bit quantization on NVIDIA GPUs. It is also one of the most reliably frustrating packages to install correctly. The errors it produces are cryptic, the GitHub issue queue has hundreds of open tickets, and the fixes depend on your specific combination of CUDA version, PyTorch version, and operating system. This post covers the four most common bitsandbytes errors, what each one actually means, and the specific fix for each.

There is a running joke in the fine-tuning community that any time you see bitsandbytes in a requirements file, you already know setup is going to be painful.

It is funny because it is true. And it is frustrating because bitsandbytes is not optional if you want to run QLoRA. It is the library that handles 4-bit and 8-bit quantization. Without it, a 7B model on a 24 GB GPU becomes a memory problem instead of a training job.

Here are the errors people hit most often and what to actually do about them.

First: Run the Built-In Bug Report

Before you do anything else, run this:

python -m bitsandbytes

This runs bitsandbytes's own diagnostic tool. It checks your CUDA version, your driver, your PyTorch installation, and whether GPU support was compiled in. It tells you what it found and what is wrong.

Read the output carefully. Most of the time it tells you exactly what the problem is. The fixes below correspond to what that diagnostic reveals.

Error One: Compiled Without GPU Support

What it looks like:

The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. CUDA is required but not available for bitsandbytes.

What caused it:

You installed bitsandbytes on a machine where CUDA was not detected at install time. This happens on CPU-only machines, in Docker containers where the GPU is not passed through during build, or in virtual environments where the CUDA toolkit is not on the PATH.

Fix option one: reinstall with CUDA available

Make sure your GPU is visible and CUDA is detected, then reinstall:

nvidia-smi  # confirm GPU is visible
pip uninstall bitsandbytes
pip install bitsandbytes

Fix option two: install the multi-backend build

If you are on an environment where the standard install keeps detecting CPU-only:

pip install bitsandbytes --prefer-binary --extra-index-url=https://jllllll.github.io/bitsandbytes-windows-webui

Fix option three: inside Docker

If you are building a Docker image and bitsandbytes is being installed during the build step where no GPU is available, the compiled binary will not have GPU support. Fix this by installing bitsandbytes at runtime, not at image build time, or by using an NVIDIA base image and the nvidia-docker runtime during build.

Error Two: CUDA Version Mismatch

What it looks like:

CUDA error: no kernel image is available for execution on the device

Or:

UserWarning: WARNING: bitsandbytes was compiled without GPU support, or the CUDA version is not supported.

What caused it:

bitsandbytes ships precompiled CUDA kernels for specific CUDA versions. If your CUDA version does not match any of the versions bitsandbytes was compiled against, it falls back to CPU-only mode or fails entirely.

Diagnose it:

python -c "import torch; print(torch.version.cuda)"
nvcc --version
nvidia-smi | head -3

Check all three. You want torch's CUDA version, the CUDA toolkit version, and the driver's supported CUDA version to be aligned.

Fix: install the version of bitsandbytes that matches your CUDA

# For CUDA 12.1
pip install bitsandbytes --index-url https://download.pytorch.org/whl/cu121

# For CUDA 11.8
pip install bitsandbytes --index-url https://download.pytorch.org/whl/cu118

Or install a specific bitsandbytes version known to work with your CUDA:

# Check compatibility at: https://github.com/bitsandbytes-foundation/bitsandbytes
pip install bitsandbytes==0.43.3  # replace with version matching your CUDA

Error Three: Version Too Old for Your Transformers

What it looks like:

Using `bitsandbytes` 8-bit quantization requires the latest version of bitsandbytes:

`pip install -U bitsandbytes`

What caused it:

Your installed version of bitsandbytes is too old for the version of Transformers you have. Transformers and bitsandbytes are tightly coupled. As Transformers adds new quantization features, it requires newer bitsandbytes APIs.

Fix:

pip install -U bitsandbytes

Then verify:

bash

python -c "import bitsandbytes; print(bitsandbytes.__version__)"

If upgrading bitsandbytes breaks something else in your environment, the alternative is to pin your Transformers version to one compatible with your bitsandbytes version. Check the bitsandbytes changelog on GitHub to find the right pairing.

Error Four: Windows-Specific Failures

bitsandbytes was originally Linux-only. Windows support was added later and is still rougher than the Linux experience.

What it looks like:

OSError: libcuda.so: cannot open shared object file

Or it installs but silently falls back to CPU mode.

Fix for Windows:

Use the Windows-specific build maintained by the community:

pip uninstall bitsandbytes
pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace the version number with the latest available at that repo for your Python version. Check github.com/jllllll/bitsandbytes-windows-webui for the current release list.

Alternatively, use WSL2. bitsandbytes on WSL2 behaves like Linux and the standard installation works reliably.

The Compatibility Matrix That Saves You Time

The fastest way to avoid bitsandbytes errors is to start from a known-working combination. Here is a stable starting point as of mid-2026:

Python: 3.11
PyTorch: 2.3.x with CUDA 12.1
bitsandbytes: 0.43.x
transformers: 4.41.x
accelerate: 0.30.x

Install in this order:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install bitsandbytes==0.43.3
pip install transformers accelerate

Installing PyTorch first, with the explicit CUDA index URL, ensures bitsandbytes installs against the right CUDA version when it comes next.

When Nothing Works

If you have tried everything above and bitsandbytes still will not install correctly, there are two reliable escape hatches.

Use a Docker image that has it pre-installed. The Axolotl Docker image and several Hugging Face images ship with bitsandbytes already configured against a specific CUDA version. Pull one of those and you skip the installation problem entirely.

Use Unsloth instead of bitsandbytes directly. Unsloth provides its own quantization implementation that handles a lot of the installation complexity. If your framework supports it, Unsloth can replace the bitsandbytes dependency for 4-bit quantization:

pip install unsloth

And if you want to skip all of this entirely and just run a fine-tuning job without managing dependencies, that is exactly what a managed platform handles for you. Your config runs in a pre-configured environment where bitsandbytes, CUDA, and PyTorch are already aligned.