Hey all,
we are experiencing problems when setting up new clusters at the moment.
What we see is that if we try to set up clusters with versions newer than 24.10.29 on both workers and leaders, the workers show a steady blue led but do not register on the leader and can not be added to the inventory. They are all connected on the same network, SSH connections to the leader are possible, SSH connections to the workers were working on the first cluster, on setting up the second cluster SSH to the workers came back with a “SSH connection refused” message.
Setting up the leader with the newest version (25.2.11) and the workers with 24.10.29 did also not work for initial connection.
When we set up the cluster (leader and workers) with version 24.10.29, addition to the inventory is not a problem and subsequent updates to the newest version also do not pose any problems (updates are done manually via the command line).
We noticed this behavior on two different clusters of three Pios each now.
If you need more information, please let me know.
We did find an issue in the latest 25.2.11 worker images that was breaking SSH connections (same result: SSH connection refused.).
We haven’t seen the issue prior though, but we will keep investigating. We know more about the cause of SSH connection problems now - can you tell us about how you are connecting workers to leaders (i.e. over wifi, a local-access-point, ethernet, local-link)?
the first cluster we set-up was trying to connect via a local access point hosted by the leader, the second cluster we were trying to set-up today is using a separately hosted wifi. Hope this helps
There was a OS update in releases after 24.10.28. We updated the base Raspberry Pi OS to version 2024-11-19, and there were some change to wifi in that OS update. Notably,
* rfkill and NetworkManager defaults to blocking all devices by default
- On-board adapters devices are whitelisted
- WLAN devices are unblocked after the regulatory domain is set
This means that the “Wireless LAN country” must be set in the Imager before writing the image, or wifi won’t turn on:
