Update broke UI + Huey failed

Hi guys,

This will be quite a long one, but I appreciate any help you can give me! Please excuse the formatting and other errors, I’m new to Pioreactors and programming/coding in general. I’m a uni student who had recently taken on 2 pioreactor units from the postgrad student before me and was working on updating them.

After using internet sharing to get my reactors online, I used the ‘update to next release button’. Before long, the two Pioreactors disappeared from the UI. They were v. 24.3.10, and now I’m not sure what version they are.

I used the steps Cam explained on the 'Updating made the UI lose the Pioreactors’ but tweaked it to suit my version.

  1. I tried pios update app -v 24.4.3 -y, but it told me app was an unexpected argument.
  2. I then tried pios update -v 24.4.3 -y which gave the following (which I believe means it updated):

2024-10-10T16:32:27+0100 DEBUG [update] Executing pio update app -v 24.4.3 on pioreactor1…
2024-10-10T16:32:27+0100 DEBUG [update] Executing pio update app -v 24.4.3 on pioreactor2…
client_loop: send disconnect: Connection reset

  1. My issue was different from Eleni’s at this point, as I did not get the error she had. When using sudo systemctl status lighttpd.service everything was normal.

lighttpd.service - Lighttpd Daemon
Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; preset: enabled)
Active: active (running) since Thu 2024-10-10 16:34:27 BST; 10min ago
● lighttpd.service - Lighttpd Daemon
Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; preset: enabled)
Active: active (running) since Thu 2024-10-10 16:34:27 BST; 10min ago
Process: 831 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
Main PID: 838 (lighttpd)
Tasks: 2 (limit: 3980)
CPU: 9min 24.535s
CGroup: /system.slice/lighttpd.service
├─ 838 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
└─1337 /usr/bin/python3 /var/www/pioreactorui/main.fcgi

  1. I tried sudo systemctl status huey.service and got this, the huey service failed.

× huey.service - Huey service for PioreactorUI
Loaded: loaded (/lib/systemd/system/huey.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Thu 2024-10-1016:34:26 BST; 17min ago
Process: 828 ExecStart=huey_consumer tasks.huey -n -b 1.0 -w 2 -f -C (code=exited, status=1/FAILURE)
Process: 829 ExecStartPost=pio log -m huey.service successful -n systemd --local-only (code=exited, status=0/SUCCESS)
Main PID: 828 (code=exited, status=1/FAILURE)
CPU: 2.011s

  1. I then tried wget -o /tmp/pioreactorui_24.4.4.tar.gz https://github.com/Pioreactor/pioreactor/releases/download/24.4.3/pioreactorui_24.4.4.tar.gz which aborted and stated the file was not in gzip format.
  2. I then realised the Huey service failure was probably an issue I needed to fix to get the UI and updates working. So I started following the steps outlined on ‘Huey service fail’.
    sudo systemctl status huey.service gave me:

× huey.service - Huey service for PioreactorUI
Loaded: loaded (/lib/systemd/system/huey.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Thu 2024-10-1016:34:26 BST; 17min ago
Process: 828 ExecStart=huey_consumer tasks.huey -n -b 1.0 -w 2 -f -C (code=exited, status=1/FAILURE)
Process: 829 ExecStartPost=pio log -m huey.service successful -n systemd --local-only (code=exited, status=

and journalctl -xeu huey.service gave me:

Hint: You are currently not seeing messages from other usersand the system.
Users in groups ‘adm’, ‘systemd-journal’ can see all messages.
Pass -q to turn off this notice.
~
~
– No entries –

  1. cd /var/www/pioreactorui && huey_consumer tasks.huey -n -b 1.0 -w 2 -f -C gave:

Error importing tasks.huey
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/huey/bin/huey_consumer.py”, line 20, in load_huey
return load_class(path)
^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/huey/utils.py”, line 34, in load_class
import(path)
ModuleNotFoundError: No module named ‘tasks’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/local/bin/huey_consumer”, line 8, in
sys.exit(consumer_main())
^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/huey/bin/huey_consumer.py”, line 49, in consumer_main
huey_instance = load_huey(args[0])
^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/huey/bin/huey_consumer.py”, line 25, in load_huey
return load_huey(path)
^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/huey/bin/huey_consumer.py”, line 20, in load_huey
return load_class(path)
^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/huey/utils.py”, line 34, in load_class
import(path)
ModuleNotFoundError: No module named ‘tasks’

  1. Using nano /var/www/pioreactorui/.env, I had an extra bar show up on the file but I did have DOT, WWW and UI_LOG. I did not screenshot this extra line of text and on returning to the lab today I am unable to connect to the leader.

My error on Huey was a module error (No module named ‘tasks’) and I have no clue how to go about this.

I went into the lab today to gather some output I forgot to screenshot, but could no longer connect to the pioreactor and the blue light flashed 8 times indicating the webserver is offline. I can’t access the lab, and so the pioreactor, over the weekend so I may not respond immediately. But I’m honestly at a loss with what I should do next.

I would appreciate any help or advice you can give me!

Hi @Grace_h, honestly you’ve done some great investigation work here.

I think the problem is that your UI is on the wrong version. Can you try

pio update ui -v 24.4.4

And see if that restores huey (and the version in the UI).


Going forward, I’d like you to use the release archive method: Updating the Pioreactor software | Pioreactor Docs

and update from one version to the next.

or

since your pioreactors are so far behind, you might want to consider restarting the cluster by reimaging the SD cards with the latest images.

Hey @CamDavidsonPilon, thank you for getting back to me so quickly!

As the Pioreactors are offline I’m not able to contact them using the ssh command prompt.

I’m guessing I should go ahead and reimage the SD card as I can’t try pio update ui -v 24.4.4?

Thanks!

Ok, I have since reimaged both SD cards and reset the cluster and success!

Both leader and worker are up to date, the local wifi is working, and the UI exists and is working as it should. I’m going to make sure I stay on top of updates so I don’t lose data after I get started.

Thank you for your help @CamDavidsonPilon

Hi @Grace_h, I’m glad to hear. We’re always trying to make the update process easier, and hearing feedback like this is important!

1 Like