We have been using our Pioreactors non-stop lately and i kind of fell behind with the updates. I tried today to update to a later version. I was 24.6.10 i think (i am not sure about the last two digits ). I was being lazy and selected âupdated to next release over the internetâ. But it was taking forever and since then all the Pioreactors have disappeared from the UI. When i try to update using a .zip file i get this message:
âUnexpected token â<â, â<?xml versâ⌠is not valid JSONâ
Maybe relevant. When i started the first update one of the workers was not online. And the UI showed briefly the green message saying leader.1 updated to a 24.7 version but then there was a red window that leader1 cannot find worker7. Since then UI shows no messages.
I can still ssh into the leader and i tried updating again but no luck.
I restarted the leader and i entered
pios update -v 24.7.7 -y
and i got this:
Traceback (most recent call last):
File â/usr/local/bin/piosâ, line 8, in
sys.exit(pios())
^^^^^^
File â/usr/local/lib/python3.11/dist-packages/click/core.pyâ, line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File â/usr/local/lib/python3.11/dist-packages/click/core.pyâ, line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File â/usr/local/lib/python3.11/dist-packages/click/core.pyâ, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File â/usr/local/lib/python3.11/dist-packages/click/core.pyâ, line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File â/usr/local/lib/python3.11/dist-packages/click/core.pyâ, line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File â/usr/local/lib/python3.11/dist-packages/pioreactor/cli/pios.pyâ, line 262, in update
units = universal_identifier_to_all_workers(units)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File â/usr/local/lib/python3.11/dist-packages/pioreactor/cli/pios.pyâ, line 61, in universal_identifier_to_all_workers
units = get_workers_in_inventory()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File â/usr/local/lib/python3.11/dist-packages/pioreactor/cluster_management/init.pyâ, line 28, in get_workers_in_inventory
return tuple(worker[âpioreactor_unitâ] for worker in result.json())
^^^^^^^^^^^^^
File â/usr/local/lib/python3.11/dist-packages/pioreactor/mureq.pyâ, line 234, in json
return loads(self.body)
^^^^^^^^^^^^^^^^
msgspec.DecodeError: JSON is malformed: invalid character (byte 0)
Itâs getting better, now the leader is updated but the workers are stuck in version 24.8.22 and the temperature, dosing and led automations are not responding (both in leader and workers).
So you should be able to SSH to your leader, and run:
pios update app -v 24.9.19
And that will update all your workers at once (or you can do it one-by-one by SSHing into each worker and using the original command).
Iâm guessing other jobs like stirring still work - and itâs just automations that fail? So we did change how automations work in an inbetween release.
Can you show me the output of:
ls /var/www/pioreactorui/contrib/jobs | grep temperature
The leader doesnât update the workers unfortunatelyâŚbut it also doesnât give an error message in the UI logs.
Here is what i get:
pioreactor@leader1:~ $ ls /var/www/pioreactorui/contrib/jobs | grep temperature
03_temperature_automation.yaml
03_temperature_control.yaml
But does that mean that my leader has lost connection to the workers? Is it something we can fix or its going to stay like this?
No, itâs temporary. This is a consequence of the new way we do updates on the workers. Your workers need to be on a later version (24.9.x) before the leader can communicate successfully.
I run it but It didnât fix the automations.
Make sure your (hard) refresh the UI first. You should only see one ârowâ of âTemperature Automationâ and other automations in the job list (previously, you may have seen two!):
Ah okay, I thought leader was up to date. Thatâs okay. I think the goal now is to try to get your leader and your workers up to 24.9.26. First bring them to 24.9.19, and then to 24.9.26, using the commands I mentioned above.
For the automations: is it still only the automations that arenât responding from the UI, or is it now any activity?
One thing to look at is: on a machine like worker1 or leader1, open up the logs with pio logs, and then try running the activity from the UI. You should see something like:
INFO Executing pio run temperature_automation --automation-name thermostat --skip-first-run 0 --target-temperature 30
(make sure it says temperature_automation and not temperature_control)
All Reactors are up to date now. And yes it is only the automations that are not working. Stirring, OD etc run normally.
I tried what you said, this what i get:
2024-09-27T22:10:24+0200 [monitor] DEBUG Running JOB_SOURCE=user EXPERIMENT='pH control SC002' nohup pio run temperature_automation >/dev/null 2>&1 & from monitor job.
That command looks like itâs missing arguments. For example, it needs a --automation-name argument:
pio run temperature_automation --automation-name thermostat
So why is it missing commands? Those commands should come from the UI when you click âSTARTâ, go to the leaderâs web server, and then be sent to a worker from there.
Iâm kinda stumped. Your leader is version 24.9.26? Try rebooting the cluster (leader and workers) maybe? Iâll have to think more about this and get back to you shortly!
). I was being lazy and selected âupdated to next release over the internetâ. But it was taking forever and since then all the Pioreactors have disappeared from the UI. When i try to update using a .zip file i get this message:
