Failed to change kernel [NoPermissions (FileSystemError): Unable to write file

[korrosivesec]

We have a shared server in which different kernel configurations are centrally managed under /usr/local/share/jupyter/kernels where write access is restricted. The Jupyter extension is unable to use these kernels due to a file write permissions error in which it appears to be trying to edit the kernel.json files.

Environment data

• VS Code version: 1.56.0-insider
• Jupyter Extension version (available under the Extensions sidebar): v2021.6.740555125
• Python Extension version (available under the Extensions sidebar): v2021.5.734046506-dev
• OS (Windows | Mac | Linux distro) and version: OSX Client, Linux SSH Host
• Jupyter server running: Local via RemoteSSH

Expected behaviour

1. Connect to server via RemoteSSH
2. Select kernel from list presented
3. Kernel starts and operates normally

Actual behaviour

1. Connect to server via RemoteSSH
2. Select kernel from list presented
3. Kernel fails to start due to file write error.

Logs

Info 2021-04-12 18:07:33: Execute Cells request 1 1
Info 2021-04-12 18:07:33: Execute Cell 1 file:///foobar.ipynb in kernelWithMetadata.ts
Info 2021-04-12 18:07:33: Starting Notebook in kernel.ts id = .pyspark24_python3_s./usr/local/anaconda3.7/bin/python3./usr/bin/python3.(S) PySpark 2.4 w/ Python 3.7
Info 2021-04-12 18:07:33: Cell Index:1, state:1, exec: undefined. User queued cell for execution
Info 2021-04-12 18:07:33: Cell Index:1, state:1, exec: undefined. Update cell state as it was enqueued
Info 2021-04-12 18:07:33: Compute Launch Info uri = /foobar.ipynb, kernelConnection id = .pyspark24_python3_s./usr/local/anaconda3.7/bin/python3./usr/bin/python3.(S) PySpark 2.4 w/ Python 3.7
Info 2021-04-12 18:07:33: Compute Launch Info uri = /foobar.ipynb, changed true, .pyspark24_python3_s./usr/local/anaconda3.7/bin/python3./usr/bin/python3.(S) PySpark 2.4 w/ Python 3.7
Info 2021-04-12 18:07:33: Python Daemon (pid: 67991): write to stderr: [I 18:07:33.519 NotebookApp] Creating new notebook in /notebooks/foobar
[I 18:07:33.519 NotebookApp] Creating new notebook in /notebooks/foobarInfo 2021-04-12 18:07:33: Python Daemon (pid: 67991): write to stderr: 


Info 2021-04-12 18:07:33: installMissingDependencies /usr/bin/python3
Info 2021-04-12 18:07:34: Cached data exists getEnvironmentVariables, <No Resource>
Info 2021-04-12 18:07:34: Process Execution: > /usr/bin/python3 ~/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2021.6.740555125/pythonFiles/pyvsc-run-isolated.py -c "import ipykernel"
> /usr/bin/python3 ~/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2021.6.740555125/pythonFiles/pyvsc-run-isolated.py -c "import ipykernel"
Info 2021-04-12 18:07:34: Process Execution: > /usr/bin/python3 ~/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2021.6.740555125/pythonFiles/pyvsc-run-isolated.py -c "import ipykernel"
> /usr/bin/python3 ~/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2021.6.740555125/pythonFiles/pyvsc-run-isolated.py -c "import ipykernel"
Info 2021-04-12 18:07:35: Install Missing Dependencies, Class name = v, completed in 1386ms, has a falsy return value, Return Value: undefined

Error 2021-04-12 18:07:40: Failed to change kernel [NoPermissions (FileSystemError): Unable to write file 'vscode-remote://ssh-remote+REMOTESSHSERVER/usr/local/share/jupyter/kernels/pyspark24_python3_s/kernel.json' (NoPermissions (FileSystemError): Error: EACCES: permission denied, open '/usr/local/share/jupyter/kernels/pyspark24_python3_s/kernel.json')
	at _handleError (/home/USERNAME/.vscode-server-insiders/bin/01466cf692c3e54f0eeb82ef0539a42391f185de/out/vs/server/remoteExtensionHostProcess.js:86:155628)
	at async b.updateKernelEnvironment (/home/USERNAME/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2021.6.740555125/out/client/extension.js:32:751398)
	at async b.ensureKernelIsUsable (/home/USERNAME/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2021.6.740555125/out/client/extension.js:32:749540)] {
  code: 'NoPermissions'
}
Info 2021-04-12 18:07:40: Shutdown session -- complete

Kernel configs stored in /home/USERNAME/.local/share/jupyter/kernels/ open as expected and show file modified timestamps that align with kernel execution, though the actual content of the kernel.json doesn't appear to have changed.

Is there a way to bypass the attempt to write to the kernel.json files?

[joyceerhl]

This is the spot where we try to write the file:

vscode-jupyter/src/client/datascience/jupyter/kernels/jupyterKernelService.ts

Lines 235 to 238 in f536e12

	// Update the kernel.json with our new stuff.
	if (shouldUpdate) {
	await this.fs.writeLocalFile(kernelSpecFilePath, JSON.stringify(specModel, undefined, 2));
	}

We could instead do

if (shouldUpdate) { 
     this.fs.writeLocalFile(kernelSpecFilePath, JSON.stringify(specModel, undefined, 2)).catch((e) => traceError('Failed to write kernelspec', e));
} 

Not sure if there would be any adverse consequences resulting from us failing to write the kernel.json and continuing to use the new kernel. I think the worst thing that might happen is we might fail to find a matching kernel for a given notebook later on, since the kernel.json wouldn't match the notebook metadata.

[DonJayamanne]

t sure if there would be any adverse consequences resulting from us failing to writ

When using Jupyter (non-raw) this will cause issues (basically the conda environment will not start properly).

I think we might want to create a separate kernel spec in the /.local/share/jupyter/kernels that points to this kernel spec, and then update that one and start that.

I.e. kernel a kernel spec named xyz with all of the information from the original kernel spec.
Update that with the env information & then start kernel spec xyz.

[DonJayamanne]

This should be fixed with #9141

[rchiodo]

/verified

Well at least that it gets written next to the extension. We need write access to this folder for other stuff to work so I assume it's going to work: