# Why LightGBM permutation importance is slow, and what you can do about it without changing the result

Permutation importance asks a fitted model a direct question: how much worse does it score when one feature
is scrambled? In scikit-learn it is one call, `sklearn.inspection.permutation_importance`, and on a LightGBM
model it is often the slowest step of an analysis. Almost all of its time goes into one thing. Some settings
change only that time and leave every bit of the result the same; others are faster because they compute
something different. Below are measurements of both kinds and a script to repeat them.

## Where the time goes

`permutation_importance(model, X, y, n_repeats=R)` scores the model once on the evaluation set. Then, for each
of the F features and each of the R repeats, it shuffles that feature's column and scores the model again.
Every score is a full prediction over all N rows, and each prediction walks every one of the T trees from the
root to a leaf for every row:

```text
(1 + F × R) predict calls × N rows × T trees
```

Our example, from [pi_cost.py](lightgbm-permutation-importance/pi_cost.py), is an `LGBMClassifier` with 300
trees of up to 63 leaves, trained on synthetic data (`make_classification`). It is evaluated on 10,000 rows ×
50 features with `n_repeats=5` and `scoring="roc_auc"`. That makes 251 predict calls and 753 million tree
walks. One `predict_proba` over the 10,000 rows took 234 ms on one core of an AMD Threadripper PRO 5975WX
(Zen 3) and 179 ms on one core of an AMD Ryzen 9 7950X3D (Zen 4). 251 × 234 ms is 58.8 s, and the whole call
took 59.2 s: prediction is about 99% of the time, and shuffling, copying and scoring are the rest.

So the time is proportional to each of the four factors. Only the trees belong to the model; reducing F, R or
N changes what is measured.

## Same result, less time

Every setting in this table gave bit-identical importances: the same SHA-256 of the full features × repeats
array, on both machines. Times are medians of three runs of the `permutation_importance` call, with the
process restricted to 8 CPUs on the Zen 3 machine and 4 on the Zen 4 machine.

| Setting | Zen 3, 8 CPUs | Zen 4, 4 CPUs |
|---|---:|---:|
| LightGBM on one thread (`n_jobs=1` on the estimator), pandas input | 59.6 s | 46.0 s |
| the same with a numpy array | 59.2 s | 45.9 s |
| out of the box: pandas input, LightGBM's default threads | 9.1 s | 13.0 s |
| LightGBM on all CPUs, numpy input | **8.6 s** | **12.5 s** |
| `permutation_importance(n_jobs=<all CPUs>)`, LightGBM on one thread per worker | 10.4 s | 13.8 s |
| `n_jobs=2`, LightGBM on half the CPUs per worker | 10.4 s | 14.5 s |
| `n_jobs=<all CPUs>` with LightGBM's default threads | 11.6 s | 15.2 s |

**Use every core, once.** LightGBM predicts rows in parallel. In its scikit-learn API the default
(`n_jobs=None`) is the number of physical cores, or of the CPUs the process may use if that is fewer, so the
out-of-the-box call already uses them: 9.1 s instead of 59.6 s. If the estimator was created with `n_jobs=1`,
which is common in cross-validation code, permutation importance runs on one core. Set it back with
`model.set_params(n_jobs=None)` (LightGBM's default) or the number of cores before the call.

**Threads or processes, not both.** `permutation_importance(n_jobs=k)` spreads the features over k worker
processes, and each worker's LightGBM still starts its own threads. With `n_jobs` equal to the number of CPUs
and LightGBM's default, that is 64 threads on 8 CPUs: 11.6 s, 36% slower than letting LightGBM use the CPUs
alone. If you parallelise over features, give the estimator one thread per worker (`n_jobs=1`), or keep
workers × threads equal to the number of CPUs. On these machines LightGBM's own threads were faster than
worker processes (8.6 s against 10.4 s, and 12.5 s against 13.8 s). 50 features do not divide evenly over 8
workers, so some sit idle in the last round, and starting the workers and sending them the model and the data
takes time. Which way wins depends on the number of rows and features, so time both on your data.

**Why the result does not change.** scikit-learn draws one seed from `random_state` and shuffles every feature
from it, whatever `n_jobs` is. LightGBM computes each row's prediction on one thread, adding the trees in the
same order, whatever the thread count. Give `random_state` an integer: with the default `None`, every run
shuffles differently and no two runs match anyway.

**pandas costs little unless prediction is cheap.** LightGBM converts a DataFrame to a numpy array in every
predict call, 251 times here. With 300 trees on one thread that is 0.6% of the time (59.6 s against 59.2 s);
with LightGBM on all CPUs, 6% on Zen 3 (9.1 s against 8.6 s) and 4% on Zen 4. For a 30-tree model on 8 CPUs,
the DataFrame took 1.75 s and the numpy array 1.40 s. Passing `X.to_numpy()` gave the same result here, where
all columns are numeric and in the training order. With pandas categorical columns keep the DataFrame:
LightGBM maps the categories itself.

## Faster, but a different result

| Setting (workers on all CPUs, LightGBM on one thread each) | Zen 3 | Zen 4 | Result |
|---|---:|---:|---|
| all rows, 5 repeats (the reference above) | 10.4 s | 13.8 s | reference |
| `max_samples=0.5` | 6.1 s | 8.0 s | different |
| `n_repeats=2` | 5.3 s | 6.8 s | different |
| input cast to float32 | 10.3 s | 13.9 s | the same here, not guaranteed; no faster |

- `max_samples` scores each feature on a random subset of the rows: a different, noisier estimate.
- Fewer repeats move the mean, and a standard deviation from two repeats says little.
- float32 input gave the same result and no gain here. LightGBM compares the input with thresholds stored as
  64-bit numbers, so a value that lies within float32 rounding of a threshold can go to the other branch.
- These change the result by design (we did not time them): predicting with fewer trees (`num_iteration`),
  prediction early stopping, a smaller or different evaluation set, or a different importance measure such as
  split or gain counts or SHAP values.

They can be the right choice for a quick look. They are just not the same numbers.

## A faster library with the same results: lightgbm-takt

[lightgbm-takt](https://pypi.org/project/lightgbm-takt/) is our free build of LightGBM 4.7.0 for Linux
x86-64. It is the official Python package with the library replaced, so `import lightgbm` and the API stay
the same:

```sh
pip uninstall -y lightgbm && pip install lightgbm-takt
```

Its predictions are bit-identical to the official wheel's on the same machine. The free package speeds up
single-threaded repeated-call analyses, such as permutation importance, partial dependence and what-if sweeps,
on models of up to 300 trees. Measured in our [LightGBM showcase](/showcase/lightgbm/README.html), whole Python
process on one core, factor in CPU cycles over the official 4.7.0 wheel:

| Workload | Free, Zen 3 | Free, Zen 4 | Full version, Zen 3 | Full version, Zen 4 |
|---|---:|---:|---:|---:|
| Permutation importance, 158 quant features, classifier | 3.30× | 2.74× | 16.75× | 18.35× |
| Permutation importance, OpenML kick (categorical, missing values) | 2.63× | 2.82× | 12.97× | 14.64× |
| Partial dependence + ICE, 20 features | 3.67× | 3.12× | 16.77× | 18.31× |
| What-if scenarios (30) | 2.50× | 2.60× | — | — |

"Free" was measured on 300-tree models, the free package's limit. "Full version" is the commercial licence,
measured on 1,000-tree (quant features) and 500-tree (kick) models; the free package runs models of that size
at the official speed.

On this note's workload, with the same result hash as the official wheel in every run:

| Setting | Zen 3, official | Zen 3, lightgbm-takt | Zen 4, official | Zen 4, lightgbm-takt |
|---|---:|---:|---:|---:|
| one core | 59.2 s | 22.7 s | 45.9 s | 18.0 s |
| `n_jobs=<all CPUs>`, LightGBM on one thread each | 10.4 s | **7.8 s** | 13.8 s | **10.2 s** |
| LightGBM on all CPUs | 8.6 s | 9.4 s | 12.5 s | 10.9 s |
| out of the box (pandas, default threads) | 9.1 s | 9.2 s | 13.0 s | 11.0 s |

With the free package, use worker processes (`n_jobs` on `permutation_importance`) and one LightGBM thread per
worker: that was the fastest exact setting on both machines. With LightGBM's own threads, alternating runs gave
a mixed picture: on Zen 4 the free package was faster than the official wheel (median 10.96 s against
12.41 s), on Zen 3 it was slower (9.35 s against 8.66 s).

## Run it yourself

```sh
python -m venv a && a/bin/pip install lightgbm==4.7.0 scikit-learn pandas
a/bin/python pi_cost.py prep work      # data and model, about 2 s
a/bin/python pi_cost.py count work     # predict calls and the time of one call
a/bin/python pi_cost.py all work 3     # every setting, 3 runs each, about 12 minutes on 8 CPUs
```

Prefix the commands with `taskset -c 0-7` (or another CPU list) to fix the CPUs. To try lightgbm-takt, install
it in a second environment and run `pi_cost.py all work 3` there; equal hashes mean identical results. Our
readings are in [zen3-official.txt](lightgbm-permutation-importance/zen3-official.txt),
[zen4-official.txt](lightgbm-permutation-importance/zen4-official.txt),
[zen3-lightgbm-takt.txt](lightgbm-permutation-importance/zen3-lightgbm-takt.txt) and
[zen4-lightgbm-takt.txt](lightgbm-permutation-importance/zen4-lightgbm-takt.txt).

Measured on 12 October 2026: LightGBM 4.7.0, scikit-learn 1.9.1, numpy 2.5.4, pandas 3.0.6, joblib 1.6.0,
Python 3.12. Zen 3: Ubuntu 24.04, 8 CPUs on 8 different cores. Zen 4: Debian 13, 4 cores.

Disclosure: TAKT sells faster builds of libraries like LightGBM whose output is bit-identical to the original
on the same system; see the [LightGBM package](/showcase/lightgbm/README.html) and
[lightgbm-takt on PyPI](https://pypi.org/project/lightgbm-takt/).

This note was drafted with an AI assistant from our measurements; the scripts were run on Linux
as described above. Corrections are welcome at hello@taktcycles.com.