The Kernel crashed while executing code on M1

[Aisuko]

Applies To

• Notebooks (.ipynb files)
• Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?

Cannot execute code in specific cell

Other cells

!pip install diffusers["torch",'flax'] transformers

import os, platform

torch_device = 'cpu'

if 'kaggle' in os.environ.get('KAGGLE_URL_BASE','localhost'):
    torch_device = 'cuda'
else:
    torch_device = 'mps' if platform.system() == 'Darwin' else 'cpu'

torch_device

os.environ['PYTORCH_ENABLE_MPS_FALLBACK'] = '1'

import torch
from diffusers import DiffusionPipeline


model_id="runwayml/stable-diffusion-v1-5"
pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
prompt = "portrait photo of master sword in Zelda: Breath of the Wild"

The cell cause Jupyter kernel crashed

pipeline = pipeline.to(torch_device)
generator = torch.Generator(device=torch_device).manual_seed(0)
image = pipeline(prompt, generator=generator).images[0]
image

The error message

Cannot execute code, session has been disposed. Please try restarting the Kernel.

VS Code Version

Version: 1.79.2 (Universal) Commit: 695af097c7bd098fbf017ce3ac85e09bbc5dda06 Date: 2023-06-14T08:58:52.392Z Electron: 22.5.7 Chromium: 108.0.5359.215 Node.js: 16.17.1 V8: 10.8.168.25-electron.0 OS: Darwin arm64 22.5.0

Jupyter Extension Version

v2023.5.1101742258

Jupyter logs

21:49:10.256 [info] Starting Kernel startUsingPythonInterpreter, .jvsc74a57bd0804b1e140bc08c445406fa6cdc5c6ee092d082a20e0122dbdb9e65f0ef0b1b5b.~/homebrew/Caskroom/miniconda/base/envs/diffusers/python.~/homebrew/Caskroom/miniconda/base/envs/diffusers/python.-m#ipykernel_launcher  (Python Path: ~/homebrew/Caskroom/miniconda/base/envs/diffusers/bin/python, Conda, diffusers, 3.11.3) for '~/Downloads/workspace/notebooks/diffusion/diffusers/generating_by_f16.ipynb' (disableUI=true)
21:49:10.258 [info] Process Execution: ~/homebrew/Caskroom/miniconda/base/envs/diffusers/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
21:49:10.264 [info] Process Execution: ~/homebrew/Caskroom/miniconda/base/envs/diffusers/bin/python -m ipykernel_launcher --ip=127.0.0.1 --stdin=9008 --control=9006 --hb=9005 --Session.signature_scheme="hmac-sha256" --Session.key=b"cde269ce-2333-48b1-af2b-f6aeea3b5e34" --shell=9007 --transport="tcp" --iopub=9009 --f=~/Library/Jupyter/runtime/kernel-v2-3512dxCZk3UwTFSr.json
    > cwd: ~/Downloads/workspace/notebooks/diffusion/diffusers
21:49:10.381 [info] ipykernel version & path 6.19.2, ~/homebrew/Caskroom/miniconda/base/envs/diffusers/lib/python3.11/site-packages/ipykernel/__init__.py for ~/homebrew/Caskroom/miniconda/base/envs/diffusers/bin/python
21:49:10.531 [warn] StdErr from Kernel Process This version of python seems to be incorrectly compiled
(internal generated filenames are not absolute).
This may make the debugger miss breakpoints.
Related bug: http://bugs.python.org/issue1666807
21:49:10.676 [info] Started Kernel diffusers (Python 3.11.3) (pid: 4344)
21:49:10.676 [info] Started new session 4f760d9a-fd40-4f88-b9d5-0edaf16d9674
21:49:13.135 [info] Handle Execution of Cells 1 for ~/Downloads/workspace/notebooks/diffusion/diffusers/generating_by_f16.ipynb
21:49:13.139 [info] Kernel acknowledged execution of cell 1 @ 1688125753137
21:49:13.140 [info] End cell 1 execution @ 1688125753139, started @ 1688125753137, elapsed time = 0.002s
21:49:14.753 [info] Handle Execution of Cells 2 for ~/Downloads/workspace/notebooks/diffusion/diffusers/generating_by_f16.ipynb
21:49:14.758 [info] Kernel acknowledged execution of cell 2 @ 1688125754755
21:49:14.760 [info] End cell 2 execution @ 1688125754758, started @ 1688125754755, elapsed time = 0.003s
21:49:16.554 [info] Handle Execution of Cells 3 for ~/Downloads/workspace/notebooks/diffusion/diffusers/generating_by_f16.ipynb
21:49:16.558 [info] Kernel acknowledged execution of cell 3 @ 1688125756556
21:49:16.560 [info] End cell 3 execution @ 1688125756558, started @ 1688125756556, elapsed time = 0.002s
21:49:18.571 [info] Handle Execution of Cells 4 for ~/Downloads/workspace/notebooks/diffusion/diffusers/generating_by_f16.ipynb
21:49:18.575 [info] Kernel acknowledged execution of cell 4 @ 1688125758573
21:49:23.365 [info] End cell 4 execution @ 1688125763363, started @ 1688125758573, elapsed time = 4.79s
21:49:26.616 [info] Handle Execution of Cells 5 for ~/Downloads/workspace/notebooks/diffusion/diffusers/generating_by_f16.ipynb
21:49:26.620 [info] Kernel acknowledged execution of cell 5 @ 1688125766618
21:49:26.621 [info] End cell 5 execution @ 1688125766620, started @ 1688125766618, elapsed time = 0.002s
21:49:28.734 [info] Handle Execution of Cells 6 for ~/Downloads/workspace/notebooks/diffusion/diffusers/generating_by_f16.ipynb
21:49:28.738 [info] Kernel acknowledged execution of cell 6 @ 1688125768735
21:49:29.826 [error] Disposing session as kernel process died ExitCode: undefined, Reason: This version of python seems to be incorrectly compiled
(internal generated filenames are not absolute).
This may make the debugger miss breakpoints.
Related bug: http://bugs.python.org/issue1666807

21:49:29.826 [info] Dispose Kernel process 4344.
21:49:29.827 [error] Raw kernel process exited code: undefined
21:49:29.828 [error] Error in waiting for cell to complete Error: Canceled future for execute_request message before replies were done
    at t.KernelShellFutureHandler.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1101742258-darwin-arm64/out/extension.node.js:2:32375)
    at ~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1101742258-darwin-arm64/out/extension.node.js:2:51427
    at Map.forEach (<anonymous>)
    at y._clearKernelState (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1101742258-darwin-arm64/out/extension.node.js:2:51412)
    at y.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1101742258-darwin-arm64/out/extension.node.js:2:44894)
    at ~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1101742258-darwin-arm64/out/extension.node.js:24:112498
    at ne (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1101742258-darwin-arm64/out/extension.node.js:2:1586779)
    at cy.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1101742258-darwin-arm64/out/extension.node.js:24:112474)
    at uy.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2023.5.1101742258-darwin-arm64/out/extension.node.js:24:119757)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
21:49:29.828 [warn] Cell completed with errors {
  message: 'Canceled future for execute_request message before replies were done'
}
21:49:29.828 [info] End cell 6 execution @ 1688125769828, started @ 1688125768735, elapsed time = 1.093s
21:49:29.829 [warn] Cancel all remaining cells due to cancellation or failure in execution
21:49:29.859 [info] End cell 6 execution @ undefined, started @ undefined, elapsed time = 0s

Coding Language and Runtime Version

Python v3.11.3

Language Extension Version (if applicable)

v2023.10.1

Anaconda Version (if applicable)

Miniconda 23.5.0

Running Jupyter locally or remotely?

Local

[Aisuko]

Additional information may be useful:

If I change pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16) to float32 precision pipeline = DiffusionPipeline.from_pretrained(model_id), this issue will gone.

Below is the code of float32 and it works well under the same environment.
https://github.com/hololandscape/notebooks/blob/main/diffusion/diffusers/effective_and_efficient_diffusion.ipynb

[cipri-tom]

My intuition would be that the operating system is killing the kernel due to some incompatibility with float16, and this has nothing to do with VSCode . You can easily check this by running the same code in a regular Jupyter Notebook, launched from the command line with jupyter notebook and accessed in the browser with localhost:8080

[frbelotto]

I am taking the same error here!

[Aisuko]

My intuition would be that the operating system is killing the kernel due to some incompatibility with float16, and this has nothing to do with VSCode . You can easily check this by running the same code in a regular Jupyter Notebook, launched from the command line with jupyter notebook and accessed in the browser with localhost:8080

Hi @cipri-tom , good point. However, all the environment components were installed automatically by the Jupyter extension, and it uses the IPython kernel to run the .ipynb file. Here I capture the log below:

And according to your point("operating system"), I remember that I hit the issue 'f16 was not supported by mps' on M1 several days ago with pytorch. Maybe the error in IPython.

14:18:58.755 [info] Started Kernel diffusers (Python 3.11.3) (pid: 7250)
14:18:58.755 [info] Started new session 81f024f5-92c2-4a66-bc25-5aa767d3128e
14:19:28.989 [info] Starting Kernel startUsingPythonInterpreter, .jvsc74a57bd0804b1e140bc08c445406fa6cdc5c6ee092d082a20e0122dbdb9e65f0ef0b1b5b.~/homebrew/Caskroom/miniconda/base/envs/diffusers/python.~/homebrew/Caskroom/miniconda/base/envs/diffusers/python.-m#ipykernel_launcher  (Python Path: ~/homebrew/Caskroom/miniconda/base/envs/diffusers/bin/python, Conda, diffusers, 3.11.3) for '~/Downloads/workspace/notebooks/diffusion/diffusers/generating_by_f16.ipynb' (disableUI=true)
14:19:28.992 [info] Process Execution: ~/homebrew/Caskroom/miniconda/base/envs/diffusers/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
14:19:29.001 [info] Process Execution: ~/homebrew/Caskroom/miniconda/base/envs/diffusers/bin/python -m ipykernel_launcher --ip=127.0.0.1 --stdin=9008 --control=9006 --hb=9005 --Session.signature_scheme="hmac-sha256" --Session.key=b"b0a6470d-9caf-41a5-a59f-4d7855fbee10" --shell=9007 --transport="tcp" --iopub=9009 --f=~/Library/Jupyter/runtime/kernel-v2-7133QddSgSlnMFFw.json
    > cwd: ~/Downloads/workspace/notebooks/diffusion/diffusers
14:19:29.105 [info] ipykernel version & path 6.19.2, ~/homebrew/Caskroom/miniconda/base/envs/diffusers/lib/python3.11/site-packages/ipykernel/__init__.py for ~/homebrew/Caskroom/miniconda/base/envs/diffusers/bin/python
14:19:29.256 [warn] StdErr from Kernel Process This version of python seems to be incorrectly compiled
(internal generated filenames are not absolute).
This may make the debugger miss breakpoints.
Related bug: http://bugs.python.org/issue1666807
14:19:29.409 [info] Started Kernel diffusers (Python 3.11.3) (pid: 7264)
14:19:29.409 [info] Started new session 843a08e4-5eeb-4a44-9dcc-ef45658ef2ba
14:55:06.073 [info] Handle Execution of Cells 4 for ~/Downloads/workspace/notebooks/diffusion/diffusers/generating_by_f16.ipynb
14:55:06.081 [info] Kernel acknowledged execution of cell 4 @ 1688187306077
14:55:06.082 [info] End cell 4 execution @ 1688187306080, started @ 1688187306077, elapsed time = 0.003s

[cipri-tom]

@Aisuko my point was that if the error also happens in the regular notebook, not just in VSCode, then you should look elsewhere for a solution, as it is not related to VSCode :(. Don't mean to sound off putting, but you won't find experts in M1 and fp16 here. Maybe ask on StackOverflow ?

[Aisuko]

@Aisuko my point was that if the error also happens in the regular notebook, not just in VSCode, then you should look elsewhere for a solution, as it is not related to VSCode :(. Don't mean to sound off putting, but you won't find experts in M1 and fp16 here. Maybe ask on StackOverflow ?

Hi @cipri-tom , I got your point. Thank you. I will post to StackOverflow.

[DonJayamanne]

thanks @cipri-tom for your input and helping with this issue.
@Aisuko, I agree with @cipri-tom, if you are running into this issue outside VS Code as well, then this is most likely an issue with the setup, we've come across a number of issues related to pytorch and similar packages on M1s and other machines as well.

Closing this issue for now.

[Aisuko]

Got it, thank you.