I tried flashing the SD cards with older versions of the worker image (e.g., 25.5.22 to 25.6.25). However, none of the flashed SD cards gave blue-light on the HAT nor did they get recognized by the watchdog. I have been troubleshooting this issue for 3 days now, and just realized that this does not happen with version 25.7.2 (most recent version - 24.07.2025).
I tested many things, but using the 24.07.2025 version was the only trial that gave positive results. Has anyone else experienced this?
Hi @sharknaro,
Yes there was a problem in those worker images (fresh installs) that was fixed in 25.7.2. If you need those versions, 25.5.22 to 25.6.25, you may need to install a previous version, and perform an upgrade(s) to get to the version you need.
Cameron
Thank you for the hint! However, the workers are still not getting recognized by the watchdog, even after flushing with the most recent software version and having constant blue light on the HAT. Could this still be caused by an existing problem?
Hm, thats odd then. It’s possible your network has turned off “mdns”, which is the communication protocol the watchdog uses. Turning off mdns is common in large networks. Are you able to find the workers via your local computer? If you remove the worker’s SD card (after powering off), place it into a computer again, look for the bootfs dir, and find a network_info.txt file there - is there an IP address assigned to the Pioreactor (should be near the top).
Would this happen randomly? The network has not been changed from the intial time it was setup. We just flashed the SD cards and could not add them back to the network.
No IP address is assigned. But it makes sense, assuming that the IP is set after the worker is added to the cluster, right?
The IP address is assigned when the Pi connects to the network, independent of being part of a cluster.
So if there is no IP address, it’s not connecting to the network. If you’re using a local-access-point, confirm that the wifi details are all correct and you’re not adding too many Pioreactors (~8 is max). If it’s your institution’s network, is this being connected over ethernet or wifi?
Yes, we are using local access point (LAP). We have changed the firmware through sudo update-alternatives --config cyfmac43455-sdio.bin as suggested. We had 12 pioreactors in the LAP. After flashing, I removed the worker(s) from the LAP, so there should be available space for them to be added.
The LAB SSID and password details must be correct because I have entered them multiple times. In addition, I also tried to add the worker to a fresh LAP with no workers. However, the same issue, the worker did not get recognized.
It shows constant blue light as described in the The blue LED is constantly on?.
The link suggests hardware issue or missing config.ini. It cannot be hardware error because the HAT behaves perfectly fine with other SD cards that is already in the cluster, and the config.ini is missing because the worker is not added to the cluster yet. I have tested all this with 25.5.22. Maybe this is why I am not being able to add to the cluster - because the leader is 25.5.22? I can try to make clean flash of leader and worker with the most recent software version.
Yes, try the recent 25.7.x software instead.
On that network_info.txt, there’s also a dump of the network logs. Can you share that (or send it to me privately at cam@pioreactor.com)?
I tried on flashing 25.7.2 on worker (alone) and worker + leadere. Both situation ended with the same outcome - constant blue light on the worker and no recognition on the leader.
I have sent the network_info.txt with the dump of the network logs to the sated email.
Looking at your logs, it looks like wifi is never started on the worker Pi. Can you tell me the Pi models of your leader and worker? There is an issue with some older Pi models and the LAP.
We’ll try replicating the issue here.
Worker: RPi 3A+
Leader: RPi 5
I do not have any hint to aid replicating the issue. Though, I think it is not Pioreactor related as I have added many RPi3A+ to RPi5 leaders without any problem before.
I will try really old software versions and share my update incase I find something.
Update:
Using 24.10.1 worker image solved the issue. I have been able to add workers to both 25.5.22 and 25.7.2 leader images.
Once think I realized with 25.7.2 leader image - the notification of “the new worker being available” appeared only in the “Invenstory > Leader > Recent logs”. There was no notification in the “Overview > Recent event logs” is this new update?
Ah! That’s true (and a recent change)! I wonder if I can make that more clear…
24.10.1
That’s interesting to know. Nothing after that worked, or did you not try?
I think stating the update in the change log and introducing back the pop-up window informing the presence of a new worker would be sufficient to keep the user informed to take out updates logged in the “Recent event logs”.
Quick answer, I did not try the ones between 24.10.1-25.5.22. However, most recent updates: I think the problem was not on the softwares but rather on our leaders. Flashing a new leader SD card with leader (only) image - 25.5.22 and 25.7.2 allowed to add workers images - 25.5.22 and 25.7.2. On the other hand, current leaders (25.5.22) did not recognize any of the worker versions, despite getting blue light on the HAT unit. Perhaps, the leaders are too busy running the experiment (one having 10 Pios running for >600 hours). I will make a clean start after the experiments and start with the most recent leader/worker versions. This should fix the ongoing issue.