Unable to control specific pioreactor jobs through UI

Software
1

Hello! I’ve had this problem in the UI where I am unable to control certain jobs in my pioreactors.

This problem started when I ran a self-test on pio06, where I accidentally pressed the self-test button twice, resulting in most of the jobs (such as temperature control, starting OD readings etc) to stop appearing. At the same time, the self-tests keep flashing between the “-” and check mark which indicates that a self-test was passed. This only occurs in pio06 though and the bug is still there even after reflashing the SD card.

I can still control all pios through the “Control All Pioreactors” interface, just not individually anymore. This has affected all pios except the leader. Any ideas how to fix this? Thanks!

2

bug
bug898×853 69 KB

bug 2
bug 21504×507 50 KB

I uploaded some pictures of what I see in the UI (I couldn’t take a video of the alternating minus and check marks in the self-test section sadly)

3

I have a theory as to what is happening. Like other bugs, this is a caching error. SSH into your leader, and try:

python3 - <<'PY'
from pioreactor.utils import local_intermittent_storage
from pioreactor.web.cache import LEADER_MULTICAST_GET_CACHE

with local_intermittent_storage(LEADER_MULTICAST_GET_CACHE) as c:
    c.empty()
PY
4

I’ll try to repro this too

5

Actually I didn’t have to do anything, I see it broken atm!

6

I’ll fix this shortly. for now, you can view the self-test results on other pages (like the pioreactor-specific page, or inventory)

7

Hello! Sorry for the late reply! I tried the command and sadly it didn’t work, I get this as a reply from the leader

“2026-04-14T16:28:04+0100 DEBUG [pioreactor-pio01-api] Starting pioreactor-pio01-api=26.4.0 on pio01…”

8

okay that message is expected, but thats too bad it didn’t work. So you’re still seeing very few activities available in the UI for that worker?

9

Yup, the only job available is Dosing Automation.

I also see two extra jobs in pio06 (Spectrometer reading and air bubbler) from the plugins I had to re-install to the pioreactor after re-flashing it… maybe this has something to do with it?

10

Ah, okay, there’s a bug in the current worker image. I’ll create a new release today. Here’s a fix for you for now:

  1. SSH into your leader
  2. Run
scp -r ~/.pioreactor/ui/jobs ~/.pioreactor/ui/automations WORKER.local:~/.pioreactor/ui/

but replace WORKER with a specific worker, like pio05:

scp -r ~/.pioreactor/ui/jobs ~/.pioreactor/ui/automations pio05.local:~/.pioreactor/ui/
  1. Do this for each worker having the problem
  2. If the problem persists, try restarting the leader, too
1 Like
11

This was the fix!! Thank you so much!

12

New minor release is coming out shortly with some other UI issues + a fix for the worker images.

2 Likes
13

I had the same problem before and it was solved by the command you mentioned above. Now I just updated to the new version v26.4.3 (20ml, v1.5), but there is another problem. The worker’s “Temp. automation”, “Dosing automation” and “LED automation” do not have the full function as the leader. See a screenshot. (Note that I didn’t check if this was also the case before updating from v26.4.3 + ssh command.)

image
image710×293 10.7 KB

14

Hi @hai1234,

hm, the most likely cause is that the worker is missing some files. Can you ssh into the worker, and try:

ls -al .pioreactor/ui/automations/dosing

and show me the result?

15

The output is “No such file or directory” and it seems that there is not directory of “automations” under ui

16

And I ignored that I got message previously
scp: stat remote: No such file or directory
scp: failed to upload directory /home/pioreactor/.pioreactor/ui/automations to ~/.pioreactor/ui/

17

@hai1234 try this on your leader:

rsync -a ~/.pioreactor/ui/automations WORKER.local:~/.pioreactor/ui/
18

Got error message of

rsync: [generator] recv_generator: mkdir "/home/pioreactor/.pioreactor/ui/dosing" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: [generator] recv_generator: mkdir "/home/pioreactor/.pioreactor/ui/led" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: [generator] recv_generator: mkdir "/home/pioreactor/.pioreactor/ui/temperature" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: [receiver] mkstemp "/home/pioreactor/.pioreactor/ui/.automation.yaml.example.b3MU5s" failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]```
19

Actually, this one

rsync: [generator] recv_generator: mkdir "/home/pioreactor/.pioreactor/ui/automations" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]```
20

hm, @hai1234

  1. try this on the worker:
sudo chown -R pioreactor:www-data /home/pioreactor/.pioreactor/ui
  1. Then try this on the leader:
rsync -a ~/.pioreactor/ui/jobs ~/.pioreactor/ui/automations WORKER.local:/home/pioreactor/.pioreactor/ui/