No devices connecting 25.2.11

DonalC · February 12, 2025, 7:04pm

Following our discussion in Error with storage.persistent_cache not found in config when trying to update to 25.1.21, I decided to wipe my cluster and start from scratch on the latest update to have a fresh install to avoid my previous issues.

After flashing my leader, I switched the Local Access Point to wlan1 and changed proto to wpa in the shared config.ini (and confirmed this was accurate by checking via /home/pioreactor/.pioreactor on the leader) for my USB WiFi hub as we discussed a while ago. However, I now am unable to connect any pioreactors to my cluster (or connect via a different computer).

The blue light on each worker is solidly illuminated (not flashing) and in the logs I am regularly get the prompts “Pioreactor work, workerXX is available to be added to your cluster”. When I try to add workers via the UI, it says the worker is added successfully, but shows up greyed out in the inventory and if I reload the page the added worker no longer is displayed.

If I try adding workers with pio workers add workerXX.local, I consistently get ‘workerXX.local not found on network - checking again.’ despite the name being spelled correctly and no matter which of my available workers I try to connect. Similarly as before, when I try to connect a different computer to the Local Access Point, I get a pop up asking for the WPA2/WPA3 password for the WiFi network “pioreactor”, which will infinitely loop with a connection failed warning. I’ve double checked in the config.ini that the ssid and password are spelled correctly (and similarly for when I am entering the password). Any ideas?

Here’s the logs output (nothing I’m noticing that stands out):

2025-02-12T13:38:17-0500 [pioreactorui-leader] DEBUG Starting pioreactorui-leader=25.2.10 on leader...
2025-02-12T13:38:17-0500 [pioreactorui-leader] DEBUG .env={'DOT_PIOREACTOR': '/home/pioreactor/.pioreactor/', 'WWW': '/var/www/pioreactorui/', 'PIO_EXECUTABLE': '', 'PIOS_EXECUTABLE': ''}
2025-02-12T13:38:17-0500 [pioreactorui-leader] DEBUG Starting MQTT client
2025-02-12T13:38:21-0500 [add_pioreactor] INFO Adding new pioreactor worker01.local to cluster.
2025-02-12T13:38:22-0500 [monitor] NOTICE Pioreactor worker, worker04, is available to be added to your cluster.
2025-02-12T13:38:22-0500 [monitor] NOTICE Pioreactor worker, worker03, is available to be added to your cluster.
2025-02-12T13:38:22-0500 [monitor] NOTICE Pioreactor worker, worker02, is available to be added to your cluster.
2025-02-12T13:38:22-0500 [monitor] NOTICE Pioreactor worker, worker01, is available to be added to your cluster.
2025-02-12T13:38:41-0500 [add_pioreactor] INFO Adding new pioreactor worker02.local to cluster.
2025-02-12T13:38:52-0500 [add_pioreactor] INFO Adding new pioreactor worker02.local to cluster.
2025-02-12T13:50:19-0500 [huey.consumer] INFO Adding new pioreactor worker01, pioreactor_20ml 1.1
2025-02-12T13:50:20-0500 [add_pioreactor] INFO Adding new pioreactor worker01 to cluster.
2025-02-12T13:51:03-0500 [add_pioreactor] ERROR `worker01` not found on network after 43 seconds. Check that you provided the right i) the name is correct, ii) worker is powered on, iii) any WiFi credentials to the network are correct.
2025-02-12T13:54:39-0500 [add_pioreactor] INFO Adding new pioreactor worker02.local to cluster.
2025-02-12T13:56:36-0500 [add_pioreactor] INFO Adding new pioreactor worker04.local to cluster.

CamDavidsonPilon · February 12, 2025, 9:21pm

Hi @DonalC, we are trying to reproduce this now.

Interesting that ^ shows up, but you can’t connect them.

it seems like they are on the network
blue light shows up on the workers => workers are online and waiting

From your leader, can you ping them at all, ex: ping worker02.local. And pio workers discover “finds” the workers I’m guessing? Can you SSH into them? ssh worker02.local?

Possibly to connect your computer to the leader’s wifi, you might try to “forget” the network and reconnect.

it says the worker is added successfully, but shows up greyed out in the inventory and if I reload the page the added worker no longer is displayed.

this sounds like a bug, we’ll investigate.

CamDavidsonPilon · February 12, 2025, 9:22pm

Wait - you should do pio workers add workerXX - drop the .local

CamDavidsonPilon · February 12, 2025, 10:17pm

@DonalC actually we are seeing the same problem independently (LAP or no LAP). We are investigating now.

One potential solution is to use the 25.1.21 worker images - we think the problem is isolated to the latest worker images only.

Some notes while we investigate:

Seems like the SSH daemon is down => possible parts of network stack are down (ping works, but web server + ssh don’t)
Can’t SSH over ethernet
LAP or not - same issue.
Exists in the nightly image => recent software issue.

DonalC · February 13, 2025, 5:34pm

This (plus updating the workers to 25.2.11) worked, but I still cannot connect with a second computer. I have one legacy connection, but it’s still concerning to me that I am unable to join the local access point on another non-RPi device. Any thoughts?

CamDavidsonPilon · February 14, 2025, 1:24am

FYI for others reading: the worker image issue is fixed.

@DonalC I’m unsure. You’ve tried a few other devices (phones, etc)? Are you trying to connect a windows machine? We’ll try to replicate tomorrow.

DonalC · February 14, 2025, 4:29pm

I’ve tried both a macbook and an iPhone and restarted both and tried a second time in case that would help. The space I’m in doesn’t have any windows machines with WiFi cards, so I haven’t tried that.

CamDavidsonPilon · February 14, 2025, 4:36pm

Have you tried manually adding it with the right security too (this is on a Mac, under the “other” in the wifi dropdown). I definitely suggest “forgetting” the pioreactor wifi network first, too (under “Wifi settings”.

DonalC · February 14, 2025, 4:38pm

Yes I’ve tried manually adding the connection and forgetting it as a known network; when I’ve tried that, it infinitely loops on this screen (after the first time entering the ssid and password):

CamDavidsonPilon · February 14, 2025, 6:36pm

Looking at a similar situation here: macos - M1 MacBook Pro can't join Linux hotspot when other devices can - Super User

Try:

nmcli con modify pioreactor wifi-sec.pmf 1

DonalC · February 14, 2025, 7:23pm

This worked, thank you so much!