CUDA Driver Version Mismatch: How to Fix It Without Reinstalling Everything

TL;DR

CUDA driver version mismatches are one of the most common environment errors in LLM fine-tuning. They usually appear as one of three specific error messages, each caused by something slightly different. Most guides tell you to reinstall the driver or the CUDA toolkit. In most cases you do not need to. This post explains the three types of CUDA version mismatch, how to diagnose which one you have, and the fastest fix for each one.

You run nvidia-smi or try to start a training job and get something like this:

Failed to initialize NVML: Driver/library version mismatch

Or this:

CUDA driver version is insufficient for CUDA runtime version

Or this:

NVRM: API mismatch: the client has version 535.104.12, but this kernel module has version 535.104.05

All three are version mismatch errors. But they are caused by different things and the fix for each one is different. If you apply the wrong fix you waste time and sometimes make things worse.

Here is how to figure out which one you have and what to do about it.

Step One: Run These Two Commands First

Before you do anything else, run these:

nvidia-smi

cat /proc/driver/nvidia/version

The first command shows the driver version your user-space tools think is running. The second shows the kernel module version that is actually loaded.

If those two numbers do not match, you have an NVML library mismatch. If nvidia-smi fails entirely, you have a different problem. If both show the same driver version but your training job is still failing, you have a CUDA runtime vs driver mismatch.

Write down what you see. The fix depends on which case you are in.

Case One: NVML Driver/Library Version Mismatch

What it looks like:

Failed to initialize NVML: Driver/library version mismatch

What caused it:

Your system updated the NVIDIA driver packages through unattended upgrades or a manual apt update, but the old kernel module is still loaded in memory. The user-space libraries are now a different version than the kernel module. They cannot talk to each other.

The fast fix: restart nvidia-persistenced

This resolves the mismatch in about 15 seconds for most cases:

sudo systemctl restart nvidia-persistenced

nvidia-smi

If nvidia-smi runs cleanly after this, you are done.

If that does not work: reload the kernel modules

sudo modprobe -r nvidia_uvm

sudo modprobe -r nvidia

sudo modprobe nvidia

nvidia-smi

This unloads the old kernel module and reloads the new one. You do not need to reboot. You do not need to reinstall.

If you are on a server you cannot reboot:

This is the case where the modprobe approach above is your path. Rebooting would also fix it but if that is not an option, unloading and reloading the modules handles it.

How to prevent it from happening again:

Unattended upgrades silently update NVIDIA driver packages without reloading the kernel module. Disable them for NVIDIA packages:

sudo dpkg-reconfigure unattended-upgrades

Choose no when prompted. After any manual driver update, always reboot or reload the modules before running a training job.

Case Two: CUDA Runtime Version Is Newer Than Your Driver

What it looks like:

CUDA driver version is insufficient for CUDA runtime version cudaErrorInsufficientDriver (error code 35)

What caused it:

Your NVIDIA driver is too old for the version of CUDA that PyTorch or your training framework was compiled against. CUDA has a one-way compatibility rule: newer drivers support older CUDA toolkits, but older drivers cannot run newer CUDA code.

This is the most common mismatch for engineers who installed PyTorch with pip without checking their driver version first.

Diagnose it:

nvidia-smi | head -3

python -c "import torch; print(torch.version.cuda)"

The first command shows your driver version and the maximum CUDA version it supports. The second shows what CUDA version PyTorch is compiled against.

If PyTorch's CUDA version is higher than what your driver supports, that is your mismatch.

The compatibility table you need:

CUDA 12.4 requires driver 550.xx or higher

CUDA 12.3 requires driver 545.xx or higher

CUDA 12.1 requires driver 530.xx or higher

CUDA 12.0 requires driver 525.xx or higher

CUDA 11.8 requires driver 520.xx or higher

CUDA 11.7 requires driver 515.xx or higher

Fix option one: install a PyTorch version that matches your current driver

If your driver supports CUDA 11.8 and you installed PyTorch for CUDA 12.1, reinstall PyTorch for the right version:

pip install torch --index-url https://download.pytorch.org/whl/cu118

This is the faster fix if you want to avoid touching your driver.

Fix option two: update your NVIDIA driver

If you want the latest PyTorch, update your driver instead:

sudo apt update sudo apt install nvidia-driver-550 sudo reboot

Replace 550 with the minimum version required for your CUDA runtime. Always reboot after a driver update.

Case Three: NVRM API Mismatch

What it looks like:

NVRM: API mismatch: the client has version 535.104.12, but this kernel module has version 535.104.05

What caused it:

The NVIDIA userspace libraries and the kernel module are from two different minor versions of the same driver. This usually happens when a driver update installed new userspace libraries but the kernel module did not get rebuilt, or vice versa.

The fix:

sudo apt purge nvidia* libnvidia*

sudo apt install nvidia-driver-535

sudo reboot

Replace 535 with your target driver version. The purge removes all NVIDIA packages including the mismatched kernel module. The reinstall builds everything from scratch and in sync. The reboot loads the new kernel module.

This is the one case where a clean reinstall of the driver package is genuinely the right answer. But note: you are reinstalling the driver, not the entire CUDA toolkit and not your Python environment.

The Diagnostic Cheat Sheet

Not sure which case you are in? Work through these in order.

# Step 1: Can nvidia-smi run at all?

nvidia-smi

# Step 2: Do the driver versions match?

cat /proc/driver/nvidia/version

# Step 3: What CUDA version does PyTorch expect?

python -c "import torch; print(torch.version.cuda)"

# Step 4: What does your driver actually support?

nvidia-smi | grep "CUDA Version"

If step 1 fails with NVML mismatch: reload the kernel module. Case one. If step 1 succeeds but step 3 shows a higher CUDA version than step 4: update the driver or downgrade PyTorch. Case two. If step 2 shows mismatched minor versions: purge and reinstall the driver. Case three.

The One Thing Worth Doing After Any Fix

Once your environment is working, pin your dependencies so they do not drift again.

If you are in a Docker container, specify your base image with an exact CUDA version tag rather than latest. If you are on a bare instance, note the exact driver version and PyTorch version that worked. If you are using a requirements file, pin torch to the exact version including the CUDA suffix.

The mismatch that took you two hours to fix today will take you two minutes next time if you know exactly what was working before.