[Windows] [Pyinstaller] Process exit when I use "Pyinstaller" to package, the executable cannot detect XPU.

[tjtanaa]

Describe the issue

I have no issue when running before packaging.

I encounter the same issue with pyinstaller compiled executable. The program exited within AutoModelForCausalLM without any error. I can observed that the progress bar of loading the weight is complete.

        self.model = AutoModelForCausalLM.from_pretrained(
            model_path, trust_remote_code=True
        ).to(self.device)

There is also this error when launching the intel_extension_for_pytorch.

intel_extension_for_pytorch\xpu\lazy_init.py when loading extension _C._initExtension() is causing the following error. _C is _C.cp310-win_amd64.pyd

 intel_extension_for_pytorch\xpu\lazy_init.py:80: UserWarning: XPU Device count is zero! (Triggered internally at C:/Users/arc/ruijie/2.1_RC3/python310/frameworks.ai.pytorch.ipex-gpu/csrc/gpu/runtime/Device.cpp:127.)
  _C._initExtension()

[huiyan2021]

Hi @tjtanaa, could you share a reproducer step by step? Thanks!

[tjtanaa]

Hi @huiyan2021 . The following is the steps to reproduce the issue on Windows 11. Conda environment is required.

Requisite

1. Download a LLM model. e.g. meta-llama/Meta-Llama-3-8B-Instruct

Steps to Reproduce

1. Install embeddedllm.
2. git clone https://github.com/EmbeddedLLM/embeddedllm.git --branch ipex-llm-win-tj
3. cd embeddedllm
4. conda create -n ellm python=3.10 libuv; conda activate ellm
5. ELLM_TARGET_DEVICE='ipex' python setup.py develop
6. Go to C:\Users\<user>\anaconda3\envs\ellmifp\Lib\site-packages\ipex_llm\transformers\convert.py of your conda environment to change the following line 56: _IS_VLLM_AVAILABLE = None to _IS_VLLM_AVAILABLE = False.
7. Install PyInstaller: pip install pyinstaller==6.9.0.
8. Compile Windows Executable: pyinstaller .\ellm_api_server.spec.
9. You can find the executable in the dist\ellm_api_server.
10. Run the following in Windows
    .\ellm_api_server.exe --model_path 'C:\Users\path\to\meta-llama_Meta-Llama-3-8B-Instruct' --backend 'ipex' --device 'xpu'
11. Expected stdout logs:

pydantic\_internal\_fields.py:160: UserWarning: Field "model_path" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ('settings_',)`.
The system is Windows.
torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: ''If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
intel_extension_for_pytorch\xpu\lazy_init.py:80: UserWarning: XPU Device count is zero! (Triggered internally at C:/Users/arc/ruijie/2.1_RC3/python310/frameworks.ai.pytorch.ipex-gpu/csrc/gpu/runtime/Device.cpp:127.)
  _C._initExtension()
2024-07-15 17:48:09,901 - INFO - intel_extension_for_pytorch auto imported
2024-07-15 17:48:10.057 | INFO     | embeddedllm.backend.ipex_engine:__init__:43 - Model Context Lenght: 8192
2024-07-15 17:48:10.058 | INFO     | embeddedllm.backend.ipex_engine:__init__:46 - Attempt to load fast tokenizer
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  9.82it/s]
2024-07-15 17:48:11,547 - INFO - Converting the current model to sym_int4 format......

[huiyan2021]

Hi @tjtanaa , We don't support PyInstaller and no bandwidth at the time being.

[tjtanaa]

Hi @tjtanaa , We don't support PyInstaller and no bandwidth at the time being.

Hi, @huiyan2021 , could you give me some pointers as to which part of the code that is generates _C.cp310-win_amd64.pyd, so that in the meantime, I could try to look into it on my end.

[xiguiw]

Steps to Reproduce

1. Install embeddedllm.
2. git clone https://github.com/EmbeddedLLM/embeddedllm.git --branch ipex-llm-win-tj
3. cd embeddedllm
4. conda create -n ellm python=3.10 libuv; conda activate ellm
5. ELLM_TARGET_DEVICE='ipex' python setup.py develop
6. Go to C:\Users\<user>\anaconda3\envs\ellmifp\Lib\site-packages\ipex_llm\transformers\convert.py of your conda environment to change the following line 56: _IS_VLLM_AVAILABLE = None to _IS_VLLM_AVAILABLE = False.

@tjtanaa What's your environment to reproduce this issue. The Intel extension-for-pytorch version, oneAPI version, driver version?
What's the ipex_llm is?

[xiguiw]

There is also this error when launching the intel_extension_for_pytorch.

intel_extension_for_pytorch\xpu\lazy_init.py when loading extension _C._initExtension() is causing the following error. _C is _C.cp310-win_amd64.pyd

 intel_extension_for_pytorch\xpu\lazy_init.py:80: UserWarning: XPU Device count is zero! (Triggered internally at C:/Users/arc/ruijie/2.1_RC3/python310/frameworks.ai.pytorch.ipex-gpu/csrc/gpu/runtime/Device.cpp:127.)
  _C._initExtension()

@tjtanaa
How did you install IPEX?
What are these: "C:/Users/arc/ruijie/2.1_RC3/python310/frameworks.ai.pytorch.ipex-gpu/csrc/gpu/runtime/Device.cpp:127".
I did not find any 'gpu' in IPEX source files (it's 'xpu'). And I did find out the file 'Device.cpp' in IPEX public source code. This is internal innersource code.

[jingxu10]

Hi @tjtanaa , We don't support PyInstaller and no bandwidth at the time being.

Hi, @huiyan2021 , could you give me some pointers as to which part of the code that is generates _C.cp310-win_amd64.pyd, so that in the meantime, I could try to look into it on my end.

@ashokei @min-jean-cho any insights?

[min-jean-cho]

UserWarning: XPU Device count is zero!

Could you please run command sycl-ls after activating oneAPI environment to verify your GPU device is being detected?

[tjtanaa]

@xiguiw
I installed ipex-llm from pypi package. The steps is based on this Ipex-llm: Install Windows GPU
which runs the command pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

When installing embeddedllm it will run this same command pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/.

The following is my python environment packages

Package                     Version            Editable project location
--------------------------- ------------------ -------------------------------------------
accelerate                  0.23.0
aiohttp                     3.9.5
aiosignal                   1.3.1
altgraph                    0.17.4
annotated-types             0.7.0
anyio                       4.4.0
async-timeout               4.0.3
attrs                       23.2.0
bigdl-core-xe-21            2.5.0b20240714
bigdl-core-xe-addons-21     2.5.0b20240714
bigdl-core-xe-batch-21      2.5.0b20240714
certifi                     2024.7.4
charset-normalizer          3.3.2
click                       8.1.7
colorama                    0.4.6
datasets                    2.20.0
dill                        0.3.8
distro                      1.9.0
docstring_parser            0.16
dpcpp-cpp-rt                2024.0.2
embeddedllm                 0.0.1+ipex         C:\Users\hpintel\VDrive\git\embeddedllm-ifp
exceptiongroup              1.2.2
fastapi                     0.110.3
filelock                    3.15.4
filetype                    1.2.0
frozenlist                  1.4.1
fsspec                      2024.5.0
gunicorn                    22.0.0
h11                         0.14.0
httpcore                    1.0.5
httpx                       0.27.0
huggingface-hub             0.23.4
idna                        3.7
inquirerpy                  0.3.4
intel-cmplr-lib-rt          2024.0.2
intel-cmplr-lic-rt          2024.0.2
intel-extension-for-pytorch 2.1.10+xpu
intel-opencl-rt             2024.0.2
intel-openmp                2024.0.2
ipex-llm                    2.1.0b20240714
Jinja2                      3.1.4
loguru                      0.7.2
markdown-it-py              3.0.0
MarkupSafe                  2.1.5
mdurl                       0.1.2
mkl                         2024.0.0
mkl-dpcpp                   2024.0.0
mpmath                      1.3.0
multidict                   6.0.5
multiprocess                0.70.16
networkx                    3.3
numpy                       1.26.4
onednn                      2024.0.0
onemkl-sycl-blas            2024.0.0
onemkl-sycl-datafitting     2024.0.0
onemkl-sycl-dft             2024.0.0
onemkl-sycl-lapack          2024.0.0
onemkl-sycl-rng             2024.0.0
onemkl-sycl-sparse          2024.0.0
onemkl-sycl-stats           2024.0.0
onemkl-sycl-vm              2024.0.0
openai                      1.35.13
packaging                   24.1
pandas                      2.2.2
pefile                      2023.2.7
pfzy                        0.3.4
pillow                      10.3.0
pip                         24.0
prompt_toolkit              3.0.47
protobuf                    5.28.0rc1
psutil                      6.0.0
py-cpuinfo                  9.0.0
pyarrow                     16.1.0
pyarrow-hotfix              0.6
pydantic                    2.7.4
pydantic_core               2.18.4
pydantic-settings           2.3.4
Pygments                    2.18.0
pyinstaller                 6.9.0
pyinstaller-hooks-contrib   2024.7
python-dateutil             2.9.0.post0
python-dotenv               1.0.1
pytz                        2024.1
pywin32-ctypes              0.2.2
PyYAML                      6.0.1
regex                       2024.5.15
requests                    2.32.3
rich                        13.7.1
safetensors                 0.4.3
sentencepiece               0.2.0
setuptools                  69.5.1
shtab                       1.7.1
six                         1.16.0
sniffio                     1.3.1
starlette                   0.37.2
sympy                       1.13.0
tabulate                    0.9.0
tbb                         2021.13.0
tokenizers                  0.15.2
torch                       2.1.0a0+cxx11.abi
torchvision                 0.16.0a0+cxx11.abi
tqdm                        4.66.4
transformers                4.36.2
trl                         0.9.6
typing_extensions           4.12.2
tyro                        0.8.5
tzdata                      2024.1
urllib3                     2.2.2
uvicorn                     0.30.1
wcwidth                     0.2.13
wheel                       0.43.0
win32-setctime              1.1.0
xxhash                      3.4.1
yarl                        1.9.4

[tjtanaa]

UserWarning: XPU Device count is zero!

Could you please run command sycl-ls after activating oneAPI environment to verify your GPU device is being detected?

@min-jean-cho
I forgot to share my hardware setup. It is a Windows 11 PC with Hardware: Intel(R) Core(TM) Ultra 7 165H

The ipex-llm[xpu] supports the use of iGPU of Intel(R) Core(TM) Ultra 7 165H.
The ipex-llm[xpu] is able to detect and utilized the iGPU when I run through python command.
However, after packaging it into executable using pyInstaller, the executable cannot detect and utilized the iGPU and fail to run.
After tracing the error, I found that the only different in log message is when loading the _C library of intel-extension-for-pytorch and it says XPU device is not found.

[MaksimDanilov]

@tjtanaa, Hello. I use pyinstaller once to make exe and succeed. You need to ensure that all dll including

you have in PATH or in your dist package. And of course, pyinstaller knows nothing about ipex package and its dependencies, so you need to do everything manually.