While trying to update the software, I accidentally used one of the development versions. After this, I tried to update the software back to the latest version, however I ended up with the following error in the command line interface of the leader pioreactor:
2024-09-02T13:25:19+0200 DEBUG [update_app] rm -rf /tmp/release_24.8.22
2024-09-02T13:25:19+0200 DEBUG [update_app]
2024-09-02T13:25:19+0200 DEBUG [update_app] mkdir /tmp/release_24.8.22
2024-09-02T13:25:19+0200 DEBUG [update_app]
2024-09-02T13:25:19+0200 DEBUG [update_app] wget -O /tmp/release_24.8.22/CHANGELOG.md https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/CHANGELOG.md
2024-09-02T13:25:20+0200 DEBUG [update_app]
2024-09-02T13:25:20+0200 DEBUG [update_app] wget -O /tmp/release_24.8.22/main.elf https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/main.elf
2024-09-02T13:25:21+0200 DEBUG [update_app]
2024-09-02T13:25:21+0200 DEBUG [update_app] wget -O /tmp/release_24.8.22/pioreactorui_24.8.21.tar.gz https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactorui_24.8.21.tar.gz
2024-09-02T13:25:23+0200 DEBUG [update_app]
2024-09-02T13:25:23+0200 DEBUG [update_app] wget -O /tmp/release_24.8.22/release_24.8.22.zip https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/release_24.8.22.zip
2024-09-02T13:25:26+0200 DEBUG [update_app]
2024-09-02T13:25:26+0200 DEBUG [update_app] wget -O /tmp/release_24.8.22/release_24.8.22.zip.sha256 https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/release_24.8.22.zip.sha256
2024-09-02T13:25:27+0200 DEBUG [update_app]
2024-09-02T13:25:27+0200 DEBUG [update_app] wget -O /tmp/release_24.8.22/wheels_24.8.22.zip https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/wheels_24.8.22.zip
2024-09-02T13:25:29+0200 DEBUG [update_app]
2024-09-02T13:25:29+0200 DEBUG [update_app] wget -O /tmp/release_24.8.22/pre_update.sh https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pre_update.sh
2024-09-02T13:25:30+0200 DEBUG [update_app]
2024-09-02T13:25:30+0200 DEBUG [update_app] sudo bash /tmp/release_24.8.22/pre_update.sh
2024-09-02T13:25:31+0200 DEBUG [update_app] Version check passed: 24.8.22
2024-09-02T13:25:31+0200 DEBUG [update_app] sudo pip3 install "pioreactor @ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl"
2024-09-02T13:25:39+0200 DEBUG [update_app] Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pioreactor@ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl
Downloading https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl (232 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 232.7/232.7 kB 1.3 MB/s eta 0:00:00
Requirement already satisfied: click==8.1.7 in /usr/local/lib/python3.11/dist-packages (from pioreactor@ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl) (8.1.7)
Requirement already satisfied: paho-mqtt==2.1.0 in /usr/local/lib/python3.11/dist-packages (from pioreactor@ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl) (2.1.0)
Requirement already satisfied: sh==2.0.6 in /usr/local/lib/python3.11/dist-packages (from pioreactor@ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl) (2.0.6)
Requirement already satisfied: JSON-log-formatter==0.5.1 in /usr/local/lib/python3.11/dist-packages (from pioreactor@ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl) (0.5.1)
Requirement already satisfied: colorlog==6.7.0 in /usr/local/lib/python3.11/dist-packages (from pioreactor@ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl) (6.7.0)
Requirement already satisfied: msgspec==0.18.5 in /usr/local/lib/python3.11/dist-packages (from pioreactor@ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl) (0.18.5)
Requirement already satisfied: diskcache==5.6.3 in /usr/local/lib/python3.11/dist-packages (from pioreactor@ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl) (5.6.3)
Requirement already satisfied: crudini==0.9.5 in /usr/local/lib/python3.11/dist-packages (from pioreactor@ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl) (0.9.5)
Requirement already satisfied: iniparse==0.5 in /usr/local/lib/python3.11/dist-packages (from pioreactor@ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl) (0.5)
Requirement already satisfied: six==1.16.0 in /usr/local/lib/python3.11/dist-packages (from pioreactor@ https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/pioreactor-24.8.22-py3-none-any.whl) (1.16.0)
2024-09-02T13:25:39+0200 DEBUG [update_app] wget -O /tmp/release_24.8.22/update.sh https://github.com/Pioreactor/pioreactor/releases/download/24.8.22/update.sh
2024-09-02T13:25:40+0200 DEBUG [update_app]
2024-09-02T13:25:40+0200 DEBUG [update_app] sudo bash /tmp/release_24.8.22/update.sh
2024-09-02T13:25:44+0200 DEBUG [update_app] + export LC_ALL=C
+ LC_ALL=C
+ PIO_DIR=/home/pioreactor/.pioreactor
+++ dirname /tmp/release_24.8.22/update.sh
++ cd /tmp/release_24.8.22
++ pwd
+ SCRIPT_DIR=/tmp/release_24.8.22
++ hostname
+ HOSTNAME=L02
++ crudini --get /home/pioreactor/.pioreactor/config.ini cluster.topology leader_hostname
+ LEADER_HOSTNAME=L02
+ '[' L02 = L02 ']'
+ touch /home/pioreactor/.pioreactor/config_L02.ini
+ crudini --ini-options=nospace --set /home/pioreactor/.pioreactor/config_L02.ini cluster.topology leader_address 127.0.0.1
+ crudini --ini-options=nospace --set /home/pioreactor/.pioreactor/config_L02.ini mqtt broker_address 127.0.0.1
+ for ini_file in "$PIO_DIR"/config*.ini
+ echo 'Processing file: /home/pioreactor/.pioreactor/config.ini'
+ crudini --get /home/pioreactor/.pioreactor/config.ini stirring
+ echo 'No [stirring] section found in /home/pioreactor/.pioreactor/config.ini. Skipping.'
+ for ini_file in "$PIO_DIR"/config*.ini
+ echo 'Processing file: /home/pioreactor/.pioreactor/config_L02.ini'
+ crudini --get /home/pioreactor/.pioreactor/config_L02.ini stirring
+ echo 'No [stirring] section found in /home/pioreactor/.pioreactor/config_L02.ini. Skipping.'
+ for ini_file in "$PIO_DIR"/config*.ini
+ echo 'Processing file: /home/pioreactor/.pioreactor/config.ini'
+ crudini --get /home/pioreactor/.pioreactor/config.ini od_config
+ echo 'No [od_config] section found in /home/pioreactor/.pioreactor/config.ini. Skipping.'
+ for ini_file in "$PIO_DIR"/config*.ini
+ echo 'Processing file: /home/pioreactor/.pioreactor/config_L02.ini'
+ crudini --get /home/pioreactor/.pioreactor/config_L02.ini od_config
+ echo 'No [od_config] section found in /home/pioreactor/.pioreactor/config_L02.ini. Skipping.'
+ sudo -u pioreactor pios sync-configs
Traceback (most recent call last):
File "/usr/local/bin/pios", line 8, in <module>
sys.exit(pios())
^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pioreactor/cli/pios.py", line 425, in sync_configs
units = add_leader(universal_identifier_to_all_workers(units))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pioreactor/cli/pios.py", line 61, in universal_identifier_to_all_workers
units = get_workers_in_inventory()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pioreactor/cluster_management/__init__.py", line 29, in get_workers_in_inventory
return tuple(worker["pioreactor_unit"] for worker in result.json())
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pioreactor/mureq.py", line 234, in json
return loads(self.body)
^^^^^^^^^^^^^^^^
msgspec.DecodeError: JSON is malformed: invalid character (byte 0)
2024-09-02T13:25:44+0200 ERROR [update_app] Update failed. See logs.
Additional information:
I can run all the leader commands except for commands regarding software update and worker pioeractors (e.g., pio/pios update, pio workers, pios sync-configs, etc.)
I cannot update the software from the zip file either for some reason.
Does anyone know what could be the potential problem and how to fix it?
This will revert what I think is the problem: an error in the web backend on the development branch. You can check this by running pio version -v to check that the pioreactor-ui version is something like 24.8.21.
and then try:
pios update
This will turn all your Pioreactors back to the most recent version (24.8.21).
Thanks for the suggestion but I already tried that and I can update the ui but that does not fix the problem to update the leader.
Also, when I open the UI the website also seems to be down because I cannot see anything in the inventory and I get html code errors in some of the sections
Can you try updating the leader in isolation? On the leader: pio update app - does that work? From there, can you run pio version -v to confirm the leader’s software is up to date.
I also tried that already both using the zip file and also by the pio update app in the command line interface. Both end up with the same error log I attached above.
I cannot check the leader command at the moment but I assume it is stuck in the development software version ad trying to update it to any version ends up with the same error log. I will check that and inform the version if I can see.
Also just to be clear we do not have problem with worked as I transferred those to another cluster and uodated to the latest version. The problem is more in the leader only pioreactor.
Then your UI should work (i.e. no more html stuff). Then try the pio commands. I strongly suggest running pio update app and pio update ui on your leader (even if you already did), and then pios update.
I’ve also removed the “update to development” from the UI - it is too much of a foot gun.
I think generally everything worked out, but it’s trying to sync configs to p12 (which is not available?), and failing here. This is the last step, so I wouldn’t worry about it failing.
However, I transferred all the workers and removed the config files…but it seems the leader still tries to sync configs.
Like you removed all the workers from the inventory? But it still thinks p12 is in the inventory?
I removed the workers through the CLI pio workers remove. However, even after removing their config files remained so I manually deleted them on the CLI.
Nevertheless, after running the sudo commands you recommended above, the workers were added back to the leader as running pio workers status illustrated all the workers remaining in the leader. Hence, I removed the workers again by the command pio workers remove --unit and manage to update the leader app and ui through the CLI.
