Pioreactors can be added to cluster but do not show up as online

I had to redo my Leader Pioreactor and set-up the wifi (local access point) as with the one before. The existing workers do connect to the wifi and get recognized by the watchdog. Adding them to the inventory also works, but they never show up as online. If the identify button on one of the added pioreactors is pressed the according pop-up in the web-ui shows up and the according messages in the mqtt log on the leader (pio mqtt) can be observed. I am really confused on what is going on. Best, Kai

And the MQTT Log from the leader regarding biohalo-worker01:

2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/computer_statistics | {“disk_usage_percent”:11,“cpu_temperature_celcius”:46,“timestamp”: “2024-10-30T02:17:20.662726Z”}
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/computer_statistics/$settable | False
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/computer_statistics/$datatype | json
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/button_down | False
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/button_down/$settable | False
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/button_down/$datatype | boolean
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/versions | {“app”:“24.8.22”,“hat”:“1.2”,“firmware”:“0.3”,“hat_serial”:“”,“rpi_machine”:“Raspberry Pi 3 Model A Plus Rev 1.0”,“timestamp”:“2024-10-30T02:17:20.576159Z”,“pioreactor_version”:“1.1”,“pioreactor_mode l”:“pioreactor_20ml”}
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/versions/$settable | True
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/versions/$datatype | json
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/voltage_on_pwm_rail/$settable | False
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/voltage_on_pwm_rail/$datatype | Voltage
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/ipv4 | 10.42.0.155
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/ipv4/$settable | False
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/ipv4/$datatype | string
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/wlan_mac_address |
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/wlan_mac_address/$settable | False
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/wlan_mac_address/$datatype | string
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/eth_mac_address | Not available
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/eth_mac_address/$settable | False
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/eth_mac_address/$datatype | string
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/state/$settable | True
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/state/$datatype | string
2024-10-30T03:17:55 | pioreactor/BioHalo-worker01/$experiment/monitor/$state | ready
2024-10-30T03:17:58 | pioreactor/BioHalo-worker01/$experiment/monitor/button_down | True
2024-10-30T03:17:58 | pioreactor/BioHalo-worker01/$experiment/monitor/$state | ready
2024-10-30T03:17:59 | pioreactor/BioHalo-worker01/$experiment/monitor/button_down | False
2024-10-30T03:18:00 | pioreactor/BioHalo-worker01/$experiment/monitor/button_down | True
2024-10-30T03:18:00 | pioreactor/BioHalo-worker01/$experiment/monitor/$state | ready
2024-10-30T03:18:02 | pioreactor/BioHalo-worker01/$experiment/monitor/button_down | False
2024-10-30T03:18:02 | pioreactor/BioHalo-worker01/$experiment/monitor/button_down | False

odd, thats the signal the UI is looking for. Does a browser hard refresh help? Do actions still start? Any errors in the browser’s js console?

Hey Cameron, no a browser hard refresh sadly did not help. I can not start actions like stirring for example. If I press the button on the WebUI I do not see any corresponding entries in the MQTT log. For the Browser Console I will quickly check if there is anything…

Browser console also does not show any errors, what is shown is a log entry from MQTTContext.js:111 that it “connected to MQTT broker for global” and “connected to MQTT broker for pioreactor”.

Can you tell me what software versions you Pioreactors have (on the Inventory page, or sshing in and running pio version)? Specifically, what the leader version is vs the workers?

So there is a version mismatch:

Leader has 24.10.29
Workers have 24.08.22

Yea you definetly want to update the workers. Fastest way is to rewrite the worker images, or if they access to the internet, use pio update app. Otherwise, you can upload the release_xxx.zip files from prerelease:false · Releases · Pioreactor/pioreactor · GitHub and run

pio update app --source /path/to/release_xxx.zip

Okay, will do that and let you know! Thanks for the help :slight_smile:

I updated all workers to 24.10.29 but still have the same problem with the same symptoms. During the Update I could also follow along in the log on the WebUI, so there is definitely Data flowing between workers and leader, just the control from the WebUI (Identify, Stirring etc ) and the Status Overview of the workers in the WebUI (Online, Version Data, IP, …) are not working.

You may need to power-cycle the cluster (the workers atleast) - try that first.

If you’re still having problems, ssh into the workers and try running stirring locally:

pio run stirring

Does that run, and show up in the UI as “On”?

I powercycled the cluster (leader and workers) after the update. I tried to start stirring on worker01 but got an error exclaiming that the worker is not assigned to an experiment.

Right, you’ll need to assign the worker to an experiment (in the UI). Then try again?

Is their data in the UI on the Inventory page?

Assigning the worker through the UI works (in the UI) but does not actually assign the worker. The inventory page does not show any information regarding the worker (Online Status, IP, Software Version,… is all blank)

Hm. Can you try re-adding the worker(s) to the cluster?

Did that, sadly the same outcome :confused:

How strange. Let’s see. The worker’s get their assinged experiment from the leader’s API, so on a worker, try:

curl http://<leader_address or hostname>/api/workers/<worker_hostname>/experiment

Ex:

 curl http://pioleader.local/api/workers/pio01/experiment

Does that work and return the correct assignment?


Since there is no data in the UI, it does sound like a connection problem between the workers and leader.

Will try that :slight_smile:

I thought so too at first but the logs in the WebUI and the ident pop-up (also WebUI) prove that the WebUI and the workers have access to the MQTT Topic, right? The communication from the worker side I can also see in the MQTT log on the leader. Could it be that the WebUI has somehow only read but no write access to the MQTT Thread?

When I curl the api from the worker I get the correct experiment assignment back, trying to run “pio run stirring” on the worker nevertheless leads to an error for not being assigned to an experiment.

Ahhhh the only other thing I can think of right now is that the leader_address is wrong in the config.ini?

On the worker, does the following return the correct address for your leader?

crudini --get ~/.pioreactor/config.ini cluster.topology leader_address

Here’s the code that gets the assigned experiment, and it runs when a job like stirring starts: pioreactor/pioreactor/whoami.py at master · Pioreactor/pioreactor · GitHub

The command returns the correct leader address. I can also edit the config successfully from the leader webUI (shared as well as worker specific).

I will have a look at the code you sent :slight_smile: