Why LightGBM permutation importance is slow, and what you can do about it without changing the result

12 October 2026 · Читать по-русски

Permutation importance asks a fitted model a direct question: how much worse does it score when one feature is scrambled? In scikit-learn it is one call, sklearn.inspection.permutation_importance, and on a LightGBM model it is often the slowest step of an analysis. Almost all of its time goes into one thing. Some settings change only that time and leave every bit of the result the same; others are faster because they compute something different. Below are measurements of both kinds and a script to repeat them.

Where the time goes

permutation_importance(model, X, y, n_repeats=R) scores the model once on the evaluation set. Then, for each of the F features and each of the R repeats, it shuffles that feature's column and scores the model again. Every score is a full prediction over all N rows, and each prediction walks every one of the T trees from the root to a leaf for every row:

(1 + F × R) predict calls × N rows × T trees

Our example, from pi_cost.py, is an LGBMClassifier with 300 trees of up to 63 leaves, trained on synthetic data (make_classification). It is evaluated on 10,000 rows × 50 features with n_repeats=5 and scoring="roc_auc". That makes 251 predict calls and 753 million tree walks. One predict_proba over the 10,000 rows took 234 ms on one core of an AMD Threadripper PRO 5975WX (Zen 3) and 179 ms on one core of an AMD Ryzen 9 7950X3D (Zen 4). 251 × 234 ms is 58.8 s, and the whole call took 59.2 s: prediction is about 99% of the time, and shuffling, copying and scoring are the rest.

So the time is proportional to each of the four factors. Only the trees belong to the model; reducing F, R or N changes what is measured.

Same result, less time

Every setting in this table gave bit-identical importances: the same SHA-256 of the full features × repeats array, on both machines. Times are medians of three runs of the permutation_importance call, with the process restricted to 8 CPUs on the Zen 3 machine and 4 on the Zen 4 machine.

Use every core, once. LightGBM predicts rows in parallel. In its scikit-learn API the default (n_jobs=None) is the number of physical cores, or of the CPUs the process may use if that is fewer, so the out-of-the-box call already uses them: 9.1 s instead of 59.6 s. If the estimator was created with n_jobs=1, which is common in cross-validation code, permutation importance runs on one core. Set it back with model.set_params(n_jobs=None) (LightGBM's default) or the number of cores before the call.

Threads or processes, not both. permutation_importance(n_jobs=k) spreads the features over k worker processes, and each worker's LightGBM still starts its own threads. With n_jobs equal to the number of CPUs and LightGBM's default, that is 64 threads on 8 CPUs: 11.6 s, 36% slower than letting LightGBM use the CPUs alone. If you parallelise over features, give the estimator one thread per worker (n_jobs=1), or keep workers × threads equal to the number of CPUs. On these machines LightGBM's own threads were faster than worker processes (8.6 s against 10.4 s, and 12.5 s against 13.8 s). 50 features do not divide evenly over 8 workers, so some sit idle in the last round, and starting the workers and sending them the model and the data takes time. Which way wins depends on the number of rows and features, so time both on your data.

Why the result does not change. scikit-learn draws one seed from random_state and shuffles every feature from it, whatever n_jobs is. LightGBM computes each row's prediction on one thread, adding the trees in the same order, whatever the thread count. Give random_state an integer: with the default None, every run shuffles differently and no two runs match anyway.

pandas costs little unless prediction is cheap. LightGBM converts a DataFrame to a numpy array in every predict call, 251 times here. With 300 trees on one thread that is 0.6% of the time (59.6 s against 59.2 s); with LightGBM on all CPUs, 6% on Zen 3 (9.1 s against 8.6 s) and 4% on Zen 4. For a 30-tree model on 8 CPUs, the DataFrame took 1.75 s and the numpy array 1.40 s. Passing X.to_numpy() gave the same result here, where all columns are numeric and in the training order. With pandas categorical columns keep the DataFrame: LightGBM maps the categories itself.

Faster, but a different result

• max_samples scores each feature on a random subset of the rows: a different, noisier estimate.
• Fewer repeats move the mean, and a standard deviation from two repeats says little.
• float32 input gave the same result and no gain here. LightGBM compares the input with thresholds stored as 64-bit numbers, so a value that lies within float32 rounding of a threshold can go to the other branch.
• These change the result by design (we did not time them): predicting with fewer trees (num_iteration), prediction early stopping, a smaller or different evaluation set, or a different importance measure such as split or gain counts or SHAP values.

They can be the right choice for a quick look. They are just not the same numbers.

A faster library with the same results: lightgbm-takt

lightgbm-takt is our free build of LightGBM 4.7.0 for Linux x86-64. It is the official Python package with the library replaced, so import lightgbm and the API stay the same:

pip uninstall -y lightgbm && pip install lightgbm-takt

Its predictions are bit-identical to the official wheel's on the same machine. The free package speeds up single-threaded repeated-call analyses, such as permutation importance, partial dependence and what-if sweeps, on models of up to 300 trees. Measured in our LightGBM showcase, whole Python process on one core, factor in CPU cycles over the official 4.7.0 wheel:

"Free" was measured on 300-tree models, the free package's limit. "Full version" is the commercial licence, measured on 1,000-tree (quant features) and 500-tree (kick) models; the free package runs models of that size at the official speed.

On this note's workload, with the same result hash as the official wheel in every run:

With the free package, use worker processes (n_jobs on permutation_importance) and one LightGBM thread per worker: that was the fastest exact setting on both machines. With LightGBM's own threads, alternating runs gave a mixed picture: on Zen 4 the free package was faster than the official wheel (median 10.96 s against 12.41 s), on Zen 3 it was slower (9.35 s against 8.66 s).

Run it yourself

python -m venv a && a/bin/pip install lightgbm==4.7.0 scikit-learn pandas
a/bin/python pi_cost.py prep work      # data and model, about 2 s
a/bin/python pi_cost.py count work     # predict calls and the time of one call
a/bin/python pi_cost.py all work 3     # every setting, 3 runs each, about 12 minutes on 8 CPUs

Prefix the commands with taskset -c 0-7 (or another CPU list) to fix the CPUs. To try lightgbm-takt, install it in a second environment and run pi_cost.py all work 3 there; equal hashes mean identical results. Our readings are in zen3-official.txt, zen4-official.txt, zen3-lightgbm-takt.txt and zen4-lightgbm-takt.txt.

Measured on 12 October 2026: LightGBM 4.7.0, scikit-learn 1.9.1, numpy 2.5.4, pandas 3.0.6, joblib 1.6.0, Python 3.12. Zen 3: Ubuntu 24.04, 8 CPUs on 8 different cores. Zen 4: Debian 13, 4 cores.

Disclosure: TAKT sells faster builds of libraries like LightGBM whose output is bit-identical to the original on the same system; see the LightGBM package and lightgbm-takt on PyPI.

This note was drafted with an AI assistant from our measurements; the scripts were run on Linux as described above. Corrections are welcome at [email protected].