Huey service fail

sharknaro · July 25, 2024, 9:58am

Hi everyone,

We are having problem with accessing the user interface (UI) through our local access point.

The local access point can still be accessed through the command line, however, the UI is remains to be down and cannot be accessed.

Upon troubleshooting, running pio update ui on the leader results with the error log:

Job for huey.service failed because the control process exited with error code.
See “systemctl status huey.service” and “journalctl -xeu huey.service” for details.

2024-07-25T11:46:36+0200 DEBUG  [update_ui] wget https://github.com/pioreactor/pioreactorui/archive/refs/tags/24.7.19.tar.gz -O /tmp/pioreactorui.tar.gz
2024-07-25T11:46:39+0200 DEBUG  [update_ui] bash /usr/local/bin/update_ui.sh /tmp/pioreactorui.tar.gz
2024-07-25T11:46:41+0200 ERROR  [update_ui] + set -e
+ export LC_ALL=C
+ LC_ALL=C
+ SRC_TAR=/tmp/pioreactorui.tar.gz
++ mktemp -d -t pioreactorui_XXXX
+ TEMP_DIR=/tmp/pioreactorui_8lQz
+ UI_FOLDER=/var/www/pioreactorui
+ trap finish EXIT
+ tar -xzf /tmp/pioreactorui.tar.gz -C /tmp/pioreactorui_8lQz
++ find /tmp/pioreactorui_8lQz -mindepth 1 -maxdepth 1 -type d
+ WORK_DIR=/tmp/pioreactorui_8lQz/pioreactorui-24.7.19
+ echo /tmp/pioreactorui_8lQz/pioreactorui-24.7.19
+ [[ -z /tmp/pioreactorui_8lQz/pioreactorui-24.7.19 ]]
+ rsync -ap --ignore-existing /var/www/pioreactorui/contrib/ /tmp/pioreactorui_8lQz/pioreactorui-24.7.19/contrib/
+ cp -p /var/www/pioreactorui/.env /tmp/pioreactorui_8lQz/pioreactorui-24.7.19
+ rm -rf /var/www/pioreactorui
+ mkdir /var/www/pioreactorui
+ cp -rp /tmp/pioreactorui_8lQz/pioreactorui-24.7.19/. /var/www/pioreactorui
+ chgrp -R www-data /var/www/pioreactorui
+ ENV=/var/www/pioreactorui/.env
+ '[' -f /var/www/pioreactorui/.env ']'
+ echo '/var/www/pioreactorui/.env exists.'
+ finish
+ rm -rf /tmp/pioreactorui_8lQz
+ sudo systemctl restart lighttpd.service
+ sudo systemctl restart huey.service
Job for huey.service failed because the control process exited with error code.
See "systemctl status huey.service" and "journalctl -xeu huey.service" for details.

Traceback (most recent call last):
  File "/usr/local/bin/pio", line 8, in <module>
    sys.exit(pio())
             ^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/cli/pio.py", line 686, in update_ui
    raise exc.BashScriptError(p.stderr)
pioreactor.exc.BashScriptError: + set -e
+ export LC_ALL=C
+ LC_ALL=C
+ SRC_TAR=/tmp/pioreactorui.tar.gz
++ mktemp -d -t pioreactorui_XXXX
+ TEMP_DIR=/tmp/pioreactorui_8lQz
+ UI_FOLDER=/var/www/pioreactorui
+ trap finish EXIT
+ tar -xzf /tmp/pioreactorui.tar.gz -C /tmp/pioreactorui_8lQz
++ find /tmp/pioreactorui_8lQz -mindepth 1 -maxdepth 1 -type d
+ WORK_DIR=/tmp/pioreactorui_8lQz/pioreactorui-24.7.19
+ echo /tmp/pioreactorui_8lQz/pioreactorui-24.7.19
+ [[ -z /tmp/pioreactorui_8lQz/pioreactorui-24.7.19 ]]
+ rsync -ap --ignore-existing /var/www/pioreactorui/contrib/ /tmp/pioreactorui_8lQz/pioreactorui-24.7.19/contrib/
+ cp -p /var/www/pioreactorui/.env /tmp/pioreactorui_8lQz/pioreactorui-24.7.19
+ rm -rf /var/www/pioreactorui
+ mkdir /var/www/pioreactorui
+ cp -rp /tmp/pioreactorui_8lQz/pioreactorui-24.7.19/. /var/www/pioreactorui
+ chgrp -R www-data /var/www/pioreactorui
+ ENV=/var/www/pioreactorui/.env
+ '[' -f /var/www/pioreactorui/.env ']'
+ echo '/var/www/pioreactorui/.env exists.'
+ finish
+ rm -rf /tmp/pioreactorui_8lQz
+ sudo systemctl restart lighttpd.service
+ sudo systemctl restart huey.service
Job for huey.service failed because the control process exited with error code.
See "systemctl status huey.service" and "journalctl -xeu huey.service" for details.

We tried:

Rebooting the leader
Updating the leader app with lower software version - i.e., 24.6.10
Reinstalling huey.whl through wheels_24.7.18.zip

However, none of the attempts solved the problem.

Did anyone encountered a similar issue or have an idea of a potential solution for this problem?

CamDavidsonPilon · July 25, 2024, 1:05pm

Hi @sharknaro,

Can you tell me the outputs of the following:

sudo systemctl status huey.service

and

sudo journalctl -xeu huey.service

The latter might be long output, we just need the last bit of it.

It’s unusual for huey to fail. I’ll do some tests this morning to try to replicate, too!

sharknaro · July 25, 2024, 1:29pm

Hi @CamDavidsonPilon!

The output of the first command should have been included in the previous post but did not get edited for some reason sorry for that!

Output: sudo systemctl status huey.service

× huey.service - Huey service for PioreactorUI
     Loaded: loaded (/lib/systemd/system/huey.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-07-25 11:46:41 CEST; 3h 39min ago    Process: 1044 ExecStart=huey_consumer tasks.huey -n -b 1.0 -w 2 -f -C (code=exited,>    Process: 1045 ExecStartPost=pio log -m huey.service successful -n systemd --local-o>   Main PID: 1044 (code=exited, status=1/FAILURE)
        CPU: 1.495s

Notice: journal has been rotated since unit was started, output may be incomplete.
lines 1-9/9 (END)...skipping...
× huey.service - Huey service for PioreactorUI
     Loaded: loaded (/lib/systemd/system/huey.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-07-25 11:46:41 CEST; 3h 39min ago
    Process: 1044 ExecStart=huey_consumer tasks.huey -n -b 1.0 -w 2 -f -C (code=exited, status=1/FAILURE)
    Process: 1045 ExecStartPost=pio log -m huey.service successful -n systemd --local-only (code=exited, status=0/SUCCESS)
   Main PID: 1044 (code=exited, status=1/FAILURE)
        CPU: 1.495s

Notice: journal has been rotated since unit was started, output may be incomplete.

Output: sudo journalctl -xeu huey.service

~
~
~
~
-- No entries --

Unfortunately, I was away when this situation happened so I cannot provide the exact cause to replicate, but I am available to provide any additional information needed!

CamDavidsonPilon · July 25, 2024, 1:35pm

Hm, what about if you ran the following:

cd /var/www/pioreactorui && huey_consumer tasks.huey -n -b 1.0 -w 2 -f -C

sharknaro · July 25, 2024, 1:42pm

Output:

Error importing tasks.huey
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/huey/bin/huey_consumer.py", line 20, in load_huey
    return load_class(path)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/huey/utils.py", line 34, in load_class
    __import__(path)
ModuleNotFoundError: No module named 'tasks'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/huey_consumer", line 8, in <module>
    sys.exit(consumer_main())
             ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/huey/bin/huey_consumer.py", line 49, in consumer_main
    huey_instance = load_huey(args[0])
                    ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/huey/bin/huey_consumer.py", line 25, in load_huey
    return load_huey(path)
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/huey/bin/huey_consumer.py", line 20, in load_huey
    return load_class(path)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/huey/utils.py", line 34, in load_class
    __import__(path)
  File "/var/www/pioreactorui/tasks.py", line 10, in <module>
    from config import cache
  File "/var/www/pioreactorui/config.py", line 39, in <module>
    config.read(Path(env["DOT_PIOREACTOR"]) / config_filename)
                     ~~~^^^^^^^^^^^^^^^^^^
KeyError: 'DOT_PIOREACTOR'

CamDavidsonPilon · July 25, 2024, 2:06pm

Oh okay, this is an easy one. I’m not sure how this happened, I’ll have to think about it, but try this:

run: nano /var/www/pioreactorui/.env
The contents of that file should look like:

DOT_PIOREACTOR=/home/pioreactor/.pioreactor/

WWW=/var/www/pioreactorui/

UI_LOG_LOCATION=/var/log/pioreactorui.log

I’m guessing you’re missing all of them, or just the DOT_PIOREACTOR one?

Save, and reboot your server:

sudo systemctl restart lighttpd.service
sudo systemctl restart huey.service

sharknaro · July 25, 2024, 2:19pm

I actually got close to this conclusion but was not sure to take any actions.

Running the command sudo nano /lib/systemd/system/huey.service resulted with the following output:

[Unit]
Description=Huey service for PioreactorUI
After=network.target firstboot.service am_i_leader.service
Requires=am_i_leader.service
Before=lighttpd.service

[Service]
User=pioreactor
WorkingDirectory=/var/www/pioreactorui
ExecStart=huey_consumer tasks.huey -n -b 1.0 -w 2 -f -C
ExecStartPost=pio log -m "%n successful" -n systemd --local-only
StandardOutput=null
StandardError=null

[Install]
WantedBy=multi-user.target

So yes, I was missing the entire .env for some reason.

After writing in the script, the UI become accessible, thank you very much!

CamDavidsonPilon · July 25, 2024, 2:28pm

Interesting. I think a bad pio update ui maybe occurred. I’ll review the code and look for edge cases. Thanks for letting us know!