Config ini error. Cannot ssh "push" new to workers as "unable to get workers"

Software
1

Several issues emerged, and I think we made some errors.

Initially the data was not stored, it was not possible to control workers from the GUI and the workers did not connect to the leader. We constantly got the pup-up error: “Failed to connect to MQTT, is MQTT.broker_address correct it’s currently xxxx” the xxx value did not match the one in the leader file. updating the config mqtt broker address to the leader didn’t work. We changed the config file back to the original file by ssh into the leader and nano write the config. Now the issue is that the command pios sync-configs doesn’t work. error: “Unable to get workers from the inventory. Is the webserver down?” And when trying to access the config from the GUI there is not a list of previous version but only this:

<?xml version="1.0" encoding="iso-8859-1"?> 500 Internal Server Error

500 Internal Server Error

Is it possible to somehow reboot the config files in another way.
or how to continue from here?

2

Hi @Stkral,

Let’s first get your leader’s webserver working. Do any errors show up in the logs with pio logs (on the leader)? If so, what do they say?

If no errors, can you check sudo systemctl status lighttpd.service for errors?

3

Hey Cameron,

I updated the UI version which restored the config.ini. But it seems like the workers have lost connection to the leader again. I think the issue is again the way the updates are performed. I was on holiday so I don’t know exactly what happened. I’ll try to find out!

4

Failed to connect to MQTT. Is configuration for mqtt.broker_address correct? Currently set to 100.84.187.10

This is what I get as notification in the UI if I try to control a worker

5

Okay, it could just be a local problem. A few things to try:

  1. Did your leader’s ipv4 change at all? SSH into the leader, and try hostname -I - that should show you the IPv4 address(es). Is one of them 100.84.187.10?
  2. Check that MQTT is indeed working: on your leader, try pio mqtt - do you see data, or is there an error.
  3. If you suspect the update, let’s try the following:
    • SSH into a worker and run pio logs - any errors come up?
    • Is it on the correct version? Try pio version -v - it should have the same version as the leader.
6

  1. Hostname -I gives: 192.168.0.190 10.42.0.1 (not the 100… address)

  2. MQTT seems fine

  3. For the worker, i get a couple of weird messages in the logs. Not sure if anything to do with an update:
    2025-02-02T03:32:12+0100 [monitor] DEBUG IPv4 address: 10.42.0.112
    2025-02-02T03:32:12+0100 [monitor] DEBUG WLAN MAC address: d8:3a:dd:61:04:a8
    2025-02-02T03:32:12+0100 [monitor] DEBUG Ethernet MAC address: Not available
    2025-02-02T03:32:12+0100 [monitor] DEBUG Disk space at 11%.
    2025-02-02T03:32:12+0100 [monitor] DEBUG CPU temperature at 47 ℃.
    2025-02-02T03:32:12+0100 [monitor] WARNING Webserver isn’t online.
    2025-02-02T03:32:23+0100 [monitor] DEBUG Pioreactor UI version: Unknown

My guess to what has been giving our recent problems is the way we have the system setup. With the leader raspberry pi broadcasting the network that all the workers connect to. Seems like most people just connect their entire cluster to an existing wifi network. But maybe you have another opinion on this :slight_smile:

7

So I see two things:

  1. I think I recall your team setting up a web portal to the Pioreactor UI though? Are you accessing the UI still through that portal? Generally, the broker_address field in your [mqtt] configuration should be one of those two IPs (probably the 192. one). However, the 100. address might be a proxy set up by your IT team.

    With the leader raspberry pi broadcasting the network that all the workers connect to.

    This is a fine solution, but there are some checks to do:

    • in the workers configuration, they should have their own mqtt section with broker_address=10.42.0.1, or broker_address=<leader's hostname>.local.

  2. It looks like your worker’s webserver is down. Can you try sudo systemctl status lighttpd.service on that worker?

I’m happy to jump on call to help get this sorted too!

8

Feb 04 16:03:59 PioBCFO01 lighttpd[18305]: During handling of the above exception, another exception occurred:
Feb 04 16:03:59 PioBCFO01 lighttpd[18305]: Traceback (most recent call last):
Feb 04 16:03:59 PioBCFO01 lighttpd[18305]: File “/var/www/pioreactorui/main.fcgi”, line 7, in
Feb 04 16:03:59 PioBCFO01 lighttpd[18305]: import pioreactorui.tasks # noqa: F401
Feb 04 16:03:59 PioBCFO01 lighttpd[18305]: ^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 04 16:03:59 PioBCFO01 lighttpd[18305]: File “/var/www/pioreactorui/pioreactorui/init.py”, line 26, in
Feb 04 16:03:59 PioBCFO01 lighttpd[18305]: from .config import env
Feb 04 16:03:59 PioBCFO01 lighttpd[18305]: File “/var/www/pioreactorui/pioreactorui/config.py”, line 29, in
Feb 04 16:03:59 PioBCFO01 lighttpd[18305]: raise IOError(f’Unable to open huey.db at {CACHE_DIR / “huey.db”}')
Feb 04 16:03:59 PioBCFO01 lighttpd[18305]: OSError: Unable to open huey.db at /tmp/pioreactorui_cache/huey.db

9

The 100. address is how we used to access the leader via the tailscale remotely.

10

They don’t have this configuration. The only info in the worker config file is the pio version

11

I’m also happy to jump on a call if you think that is better, I am free for the next 3 hours :slight_smile: