Update failed, leader and workers on different versions

Hey everyone,

i’ve tried to update a cluster consisting of a leader and three workers via the ui. The leader updated fine to version 24.10.1, the workers are still stuck on version 24.8.22. I’ve tried to update the workers manually via ssh, the update fails each time. According to the logs, the update downloads fine, only to fail with “DEBUG error: externally-managed-environment”. Is there any way to fix this without reimaging the pioreactors?

Cheers,

Hi @EBTPio,

Sorry about that! The error: externally-managed-environment is interesting, and should be an easy solution, but can you show me the entire logs when you run pio update app on a worker?

Hey,
of course, and thank you.

2024-10-24T16:03:31+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/create_diskcache.sh https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/create_diskcache.sh
2024-10-24T16:03:33+0200 [update_app] DEBUG
2024-10-24T16:03:33+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/huey.service https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/huey.service
2024-10-24T16:03:34+0200 [update_app] DEBUG
2024-10-24T16:03:34+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/lighttpd.conf https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/lighttpd.conf
2024-10-24T16:03:35+0200 [update_app] DEBUG
2024-10-24T16:03:35+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/lighttpd.service https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/lighttpd.service
2024-10-24T16:03:36+0200 [update_app] DEBUG
2024-10-24T16:03:36+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/lighttpd_packages.zip https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/lighttpd_packages.zip
2024-10-24T16:03:59+0200 [update_app] DEBUG
2024-10-24T16:03:59+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/pioreactorui_24.9.20.tar.gz https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/pioreactorui_24.9.20.tar.gz
2024-10-24T16:04:02+0200 [update_app] DEBUG
2024-10-24T16:04:02+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/release_24.9.19.zip https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/release_24.9.19.zip
2024-10-24T16:04:33+0200 [update_app] DEBUG
2024-10-24T16:04:33+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/release_24.9.19.zip.sha256 https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/release_24.9.19.zip.sha256
2024-10-24T16:04:34+0200 [update_app] DEBUG
2024-10-24T16:04:34+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/update_ui.sh https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/update_ui.sh
2024-10-24T16:04:35+0200 [update_app] DEBUG
2024-10-24T16:04:35+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/wheels_24.9.19.zip https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/wheels_24.9.19.zip
2024-10-24T16:04:37+0200 [update_app] DEBUG
2024-10-24T16:04:37+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/pre_update.sh https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/pre_update.sh
2024-10-24T16:04:39+0200 [update_app] DEBUG
2024-10-24T16:04:39+0200 [update_app] DEBUG sudo bash /tmp/release_24.9.19/pre_update.sh
2024-10-24T16:04:40+0200 [update_app] DEBUG Version check passed: 24.8.22

2024-10-24T16:04:40+0200 [update_app] DEBUG sudo pip3 install “pioreactor @ https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/pioreactor-24.9.19-py3-none-any.whl
2024-10-24T16:04:43+0200 [update_app] DEBUG error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

For more information visit http://rptl.io/venv

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

2024-10-24T16:04:43+0200 [update_app] ERROR Update failed. See logs.
2024-10-24T16:05:18+0200 [update_app] DEBUG rm -rf /tmp/release_24.9.19
2024-10-24T16:05:18+0200 [update_app] DEBUG
2024-10-24T16:05:18+0200 [update_app] DEBUG mkdir /tmp/release_24.9.19
2024-10-24T16:05:18+0200 [update_app] DEBUG
2024-10-24T16:05:18+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/50-pioreactorui.conf https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/50-pioreactorui.conf
2024-10-24T16:05:19+0200 [update_app] DEBUG
2024-10-24T16:05:19+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/52-api-only.conf https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/52-api-only.conf
2024-10-24T16:05:20+0200 [update_app] DEBUG
2024-10-24T16:05:20+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/CHANGELOG.md https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/CHANGELOG.md
2024-10-24T16:05:22+0200 [update_app] DEBUG
2024-10-24T16:05:22+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/create_diskcache.service https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/create_diskcache.service
2024-10-24T16:05:23+0200 [update_app] DEBUG
2024-10-24T16:05:23+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/create_diskcache.sh https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/create_diskcache.sh
2024-10-24T16:05:25+0200 [update_app] DEBUG
2024-10-24T16:05:25+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/huey.service https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/huey.service
2024-10-24T16:05:25+0200 [update_app] DEBUG
2024-10-24T16:05:25+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/lighttpd.conf https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/lighttpd.conf
2024-10-24T16:05:26+0200 [update_app] DEBUG
2024-10-24T16:05:26+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/lighttpd.service https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/lighttpd.service
2024-10-24T16:05:27+0200 [update_app] DEBUG
2024-10-24T16:05:27+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/lighttpd_packages.zip https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/lighttpd_packages.zip
2024-10-24T16:05:51+0200 [update_app] DEBUG
2024-10-24T16:05:51+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/pioreactorui_24.9.20.tar.gz https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/pioreactorui_24.9.20.tar.gz
2024-10-24T16:05:54+0200 [update_app] DEBUG
2024-10-24T16:05:54+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/release_24.9.19.zip https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/release_24.9.19.zip
2024-10-24T16:06:21+0200 [update_app] DEBUG
2024-10-24T16:06:21+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/release_24.9.19.zip.sha256 https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/release_24.9.19.zip.sha256
2024-10-24T16:06:21+0200 [update_app] DEBUG
2024-10-24T16:06:21+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/update_ui.sh https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/update_ui.sh
2024-10-24T16:06:22+0200 [update_app] DEBUG
2024-10-24T16:06:22+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/wheels_24.9.19.zip https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/wheels_24.9.19.zip
2024-10-24T16:06:24+0200 [update_app] DEBUG
2024-10-24T16:06:24+0200 [update_app] DEBUG wget -O /tmp/release_24.9.19/pre_update.sh https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/pre_update.sh
2024-10-24T16:06:25+0200 [update_app] DEBUG
2024-10-24T16:06:25+0200 [update_app] DEBUG sudo bash /tmp/release_24.9.19/pre_update.sh
2024-10-24T16:06:26+0200 [update_app] DEBUG Version check passed: 24.8.22

2024-10-24T16:06:26+0200 [update_app] DEBUG sudo pip3 install “pioreactor @ https://github.com/Pioreactor/pioreactor/releases/download/24.9.19/pioreactor-24.9.19-py3-none-any.whl
2024-10-24T16:06:29+0200 [update_app] DEBUG error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

For more information visit http://rptl.io/venv

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

2024-10-24T16:06:29+0200 [update_app] ERROR Update failed. See logs.
tail: cannot open ‘’ for reading: No such file or directory

Yea that’s weird, I thought we fixed this in our images. Anyways, try this solution:

sudo pip3 config set global.break-system-packages true

and then run the pio update app command again and see it if works.

1 Like

Hey,

this worked, i needed to run pio update app twice.
break-system-packages should probably be to false again, or have i understood something wrong?
As usual, thank you so much for your help.

Cheers,

1 Like

No, leave it as true. It sounds like a scary config, but it’s not.