Leader not distributing the config file

Hello, i have a problem with my leader’s communication with the workers. the problem started after i reinstalled the 25.9.18 image on the leader and the workers. (i installed the 25.11.19 and then i went back to 25.9.18).

Is it possible that the root of the problem is that the app and the ui are at different versions. My leader and workers are at 25.9.18 and the ui is at 25.11.19. I dont really know how this happened.
now i rebooted the leader and all the workers are gone and both the mqtt_to_db_streaming and monitor jobs are off.

Hm, did you rewrite the sd card when you did this? That’s probably the only way to “revert” (the 9.18 → 11.19 is a large jump).

yes i rewrote the SD with the 11.19 and then again with the 9.18

Can you try a hard-refresh in the browser, ex: cmd-R or Ctrl + Shift + R? It could just be a caching issue.

Otherwise, there shouldn’t be any mention of 25.11.x on the pioreactor cluster.

the hard-refresh didn’t solve it. do you think that the mismatch of the app and the ui version is the issue here? if so is there a way to revert the ui to 9.18?

So, the ui-version doesn’t really exist - not since the summer. There’s really one a single version now, and the app version === ui version. But you’re seeing differences? Where are you seeing this? (Honestly this kinda sounds like what @gendor was seeing…)

it was there next to the Version installed on leader:

Now its gone because i restarted the leader and everything is gone.

This part sounds like a web server issue now. Can you ssh into your leader and try:

sudo systemctl status lighttpd

i get this:

lighttpd.service - Lighttpd Daemon
Loaded: loaded (/etc/systemd/system/lighttpd.service; enabled; preset: enabled)
Active: active (running) since Thu 2025-11-27 17:18:36 CET; 5h 32min ago
Process: 580 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
Main PID: 587 (lighttpd)
Tasks: 2 (limit: 380)
CPU: 5h 31min 27.577s
CGroup: /system.slice/lighttpd.service
├─ 587 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
└─15678 /usr/bin/python3 /var/www/pioreactorui/main.fcgi

Nov 27 22:50:35 leader lighttpd[15677]: import pioreactorui.tasks # noqa: F401
Nov 27 22:50:35 leader lighttpd[15677]: ^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 27 22:50:35 leader lighttpd[15677]: File “/var/www/pioreactorui/pioreactorui/init.py”, line 24, in
Nov 27 22:50:35 leader lighttpd[15677]: from pioreactor.logging import create_logger
Nov 27 22:50:35 leader lighttpd[15677]: File “/usr/local/lib/python3.11/dist-packages/pioreactor/logging.py”, line 10>
Nov 27 22:50:35 leader lighttpd[15677]: from pioreactor import types as pt
Nov 27 22:50:35 leader lighttpd[15677]: File “/usr/local/lib/python3.11/dist-packages/pioreactor/types.py”, line 14
Nov 27 22:50:35 leader lighttpd[15677]: type Unit = str
Nov 27 22:50:35 leader lighttpd[15677]: ^^^^
Nov 27 22:50:35 leader lighttpd[15677]: SyntaxError: invalid syntax

okay! So something is up! Somehow, there is a pioreactor version on your machine that shouldn’t be. Maybe I messed up a leader-worker fix I did in October, and somehow a newer version was installed. I am checking now

I’m not really sure what’s up. Is this a totally fresh leader-worker (like nothing was installed - plugins, updates - maybe only workers were added)? Does your cluster have access to the internet?

Ah thats a good point! After flashing i added all my workers with no issues. The cluster seemed fine, the problems started when i tried to install the air-bubbler plugin.

ok i reproduced it. I flashed my leader again, added worker1 and worker2 to my cluster. they seemed to be working fine with the OD and stirring automations working. In the updates it says “Version installed on leader:25.9.18”. I made changes in the config file and that didnt change the version. But when i try to install the air-bubbler plugin the version installed on leader jumped to 25.11.19.

Okay, I see the problem. I’ve fixed the air-bubbler package to version 0.10.0, which shouldn’t force the latest pioreactor to be installed. Sorry for wasting your time =(

1 Like

Amazing! Indeed it is fixed now. Thank you for looking into it :smiley: