Updating made leader doesn't work

Sansanee · October 11, 2024, 3:19pm

Hi here,

I updated the software from 24.8.22 to 24.9.19, follwed by 24.9.26 and 24.10.1.
All of my pioreactors don’t start working when I pressed start after the updates.

It happened only with leader when they were updated to 24.9.26. I decided to further update to the latest version as I hoped it would help.

Apparently, no. All of them didn’t work afterwards.

I then downgraded the software to 24.9.26, but they still didn’t work.

I did the update by sshing to pioreactor. Firstly, to leader and then contributed the zip file to workers, followed by sshing to each worker.

The result after running the command showed ‘successfully uninstalled-24.10.1’ and ‘successfully installed pioreactor-24.9.26’.

Has anyone experined this and known how to sort this out, please let me know?

Thank you so much,
Sansanee

Sansanee · October 11, 2024, 4:18pm

Additionally, I downgraded the software to previous version (24.8.22). They still cannot start operating any jobs (i.e. stirriring, OD, and so on). They cant even start sefltest. When I run a command of
pio workers status
The result showed that all pioreactors has been updated to 24.8.22, and are active.

CamDavidsonPilon · October 11, 2024, 5:29pm

Hi @Sansanee, if they are all on 24.8.22 after downgrading, try doing a reboot (power-cycle) of the workers, and see if that works.

I suggest trying to update again, though! I think the release archive method is best (and again, go one version at a time).

Sansanee · October 14, 2024, 10:59am

Hi@Cameron,

I did. I shutdown the pioreactor after downgraded to 24.8.22, but the UI did not respond after turning on again. I can connect to the pioreactors and can ssh to them, but cannot work on any experiments.
The configuration shows message as below;

<?xml version="1.0" encoding="iso-8859-1"?> 503 Service Unavailable

503 Service Unavailable

I’m thinking it might be because of the previous problem about air_bubbler plugin installation. So I checked the active plugins on my pioreactors and there was the output as below;
pioreactor@Pio01:~ $ pio plugins list
/home/pioreactor/.pioreactor/plugins/setup.py encountered plugin load error: [Errno 2] No such file or directory: ‘README.md’
/home/pioreactor/.pioreactor/plugins/setup.py encountered plugin load error: [Errno 2] No such file or directory: ‘README.md’
spectrometer-reading-plugin==0.2.8
pioreactor-air-bubbler==0.3.1
init==Unknown
air_bubbler==Unknown

I am not sure if the problems of software update and the plugin installation are related.
Could you please give advice how to solve this issue?

Best regards,
Sansanee

CamDavidsonPilon · October 15, 2024, 12:41am

I believe the plugin issue is unrelated, but we can clean that up:

pios rm /home/pioreactor/.pioreactor/plugins/setup.py -y 
pios rm /home/pioreactor/.pioreactor/plugins/__init__.py -y

The web server looks down, let’s find out why. Can you report back the output of the following?

sudo systemctl status lighttpd.service

LeoKleym · October 15, 2024, 8:32am

Hey all, so for me its the other way around: i updated all my pioreactors and the ui to the newest version and also update from pioreactor version 1.0 to 1.1 and update that as well in the config ini. i just wanted to start a selftest but it only works on the leader but on none of the workers. anyone has an idea? they are all updated and online, but no job is starting on the workers…

edit: i can ssh into the workers and perform the selftest but i cannot start it from the UI.

Sansanee · October 15, 2024, 5:03pm

Hi Cameron,

I did not run the command to test the server, Sorry! Because I decided to reflash the card (as I have only 4 reactors and already downloaded the important data).
However, the same problem happens after I reflashed the SD card (installed the sofware version 24.10.1). My leader cannot start jobs (stirring, OD, growth rate, and so on). But all workers are working well.
An additional problem is about connection between leader and a worker. Even I can start running an experiment in all workers, but it always shows an error after changing configuration that:
‘Could not transfer config to Pio02. Is it online?’

It is actually online because I can ssh to it and can start jobs.

Additonal problem about plugin installation, I tried to install the air_bubbler plugin again by adding .whl file in /home/pioreactor/.pioreactor/plugins, and using a command:

pio plugins install pioreactor_air_bubbler-0.3.1-py3-none-any.whl

The output was same as before (as I asked in the other topic)
https://forum.pioreactor.com/t/how-to-install-plugin-for-air-bubbler-for-offline-pioreactor/487

I added additional configuration as you suggested, but it still cannot be started.

Best regards,
Sansanee

CamDavidsonPilon · October 15, 2024, 5:20pm

This sounds like your worker’s web server (that’s a new addition in the recent updates) isn’t working. Can you try:

sudo systemctl status lighttpd.service

on a worker and see what it reports back?

CamDavidsonPilon · October 15, 2024, 5:22pm

This sounds similar to @LeoKleym remark above. Can you try:

sudo systemctl status lighttpd.service

on a worker and see what it reports back?

For plugins, try this:

pio plugins install pioreactor-air-bubbler --source ~/pioreactor_air_bubbler-0.3.1-py3-none-any.whl

LeoKleym · October 16, 2024, 7:42am

sure! so this is what i get back from a worker:

pioreactor@pioreactor3:~ $ sudo systemctl status lighttpd.service
● lighttpd.service - Lighttpd Daemon
Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-10-15 10:27:53 CEST; 23h ago
Process: 753 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
Main PID: 775 (lighttpd)
Tasks: 7 (limit: 1690)
CPU: 11.010s
CGroup: /system.slice/lighttpd.service
├─775 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
└─781 /usr/bin/python3 /var/www/pioreactorui/main.fcgi

Oct 15 09:17:12 pioreactor3 systemd[1]: Starting lighttpd.service - Lighttpd Daemon…
Oct 15 10:27:53 pioreactor3 systemd[1]: Started lighttpd.service - Lighttpd Daemon.

this is what i get back from my leader:

pioreactor@pioreactor2:~ $ sudo systemctl status lighttpd.service
● lighttpd.service - Lighttpd Daemon
Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-10-15 14:10:07 CEST; 19h ago
Process: 4932 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
Main PID: 4938 (lighttpd)
Tasks: 8 (limit: 1690)
CPU: 25.931s
CGroup: /system.slice/lighttpd.service
├─4938 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
└─4954 /usr/bin/python3 /var/www/pioreactorui/main.fcgi

Oct 15 14:10:06 pioreactor2 systemd[1]: Starting lighttpd.service - Lighttpd Daemon…
Oct 15 14:10:07 pioreactor2 systemd[1]: Started lighttpd.service - Lighttpd Daemon.

Sansanee · October 16, 2024, 12:07pm

The output from worker was;
pioreactor@Pio02:~ $ sudo systemctl status lighttpd.service
● lighttpd.service - Lighttpd Daemon
Loaded: loaded (/etc/systemd/system/lighttpd.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-10-01 22:41:14 BST; 1min 7s ago
Process: 769 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
Main PID: 779 (lighttpd)
Tasks: 7 (limit: 3977)
CPU: 1.394s
CGroup: /system.slice/lighttpd.service
├─779 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
└─797 /usr/bin/python3 /var/www/pioreactorui/main.fcgi

Oct 01 22:41:14 Pio02 systemd[1]: Starting lighttpd.service - Lighttpd Daemon…
Oct 01 22:41:14 Pio02 systemd[1]: Started lighttpd.service - Lighttpd Daemon.

I run the same command on leader and the output was;
pioreactor@Pio01:~ $ sudo systemctl status lighttpd.service
● lighttpd.service - Lighttpd Daemon
Loaded: loaded (/etc/systemd/system/lighttpd.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-10-01 22:31:44 BST; 2min 59s ago
Process: 753 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
Main PID: 826 (lighttpd)
Tasks: 8 (limit: 3977)
CPU: 2.111s
CGroup: /system.slice/lighttpd.service
├─826 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
└─838 /usr/bin/python3 /var/www/pioreactorui/main.fcgi

Oct 01 22:31:43 Pio01 systemd[1]: Starting lighttpd.service - Lighttpd Daemon…
Oct 01 22:31:44 Pio01 systemd[1]: Started lighttpd.service - Lighttpd Daemon.

The problem of sync and connection within the cluster was sorted out by removing the problem worker from cluster and adding it back again.
However, my leader pioreactor still cannot start its jobs.

For the plugin, I run the command on leader and the output was;
pioreactor@Pio01:~ $ pio plugins install pioreactor-air-bubbler --source ~/pioreactor_air_bubbler-0.3.1-py3-none-any.whl
2024-10-01T23:41:04+0100 DEBUG [install_plugin] Installing plugin pioreactor-air-bubbler.
2024-10-01T23:41:04+0100 DEBUG [install_plugin] bash /usr/local/bin/install_pioreactor_plugin.sh pioreactor-air-bubbler /home/pioreactor/pioreactor_air_bubbler-0.3.1-py3-none-any.whl
2024-10-01T23:41:08+0100 NOTICE [install_plugin] Successfully installed plugin pioreactor-air-bubbler.

But when I run this command on workers, the output was;
pioreactor@Pio02:~ $ pio plugins install pioreactor-air-bubbler --source ~/pioreactor_air_bubbler-0.3.1-py3-none-any.whl
2024-10-01T23:41:30+0100 DEBUG [install_plugin] Installing plugin pioreactor-air-bubbler.
2024-10-01T23:41:30+0100 DEBUG [install_plugin] bash /usr/local/bin/install_pioreactor_plugin.sh pioreactor-air-bubbler /home/pioreactor/pioreactor_air_bubbler-0.3.1-py3-none-any.whl
2024-10-01T23:41:32+0100 ERROR [install_plugin] Failed to install plugin pioreactor-air-bubbler. See logs.
2024-10-01T23:41:32+0100 DEBUG [install_plugin] b’Processing ./pioreactor_air_bubbler-0.3.1-py3-none-any.whl\n’
2024-10-01T23:41:32+0100 DEBUG [install_plugin] b"+ export LC_ALL=C\n+ LC_ALL=C\n+ plugin_name=pioreactor-air-bubbler\n+ source=/home/pioreactor/pioreactor_air_bubbler-0.3.1-py3-none-any.whl\n+ clean_plugin_name=pioreactor-air-bubbler\n+ clean_plugin_name_with_dashes=pioreactor-air-bubbler\n+ clean_plugin_name_with_underscores=pioreactor_air_bubbler\n++ python3 -c ‘import site; print(site.getsitepackages()[0])’\n+ install_folder=/usr/local/lib/python3.11/dist-packages/pioreactor_air_bubbler\n++ crudini --get /home/pioreactor/.pioreactor/config.ini cluster.topology leader_hostname\n+ leader_hostname=Pio01\n++ hostname\n+ ‘[’ Pio01 = Pio02 ‘]’\n+ am_i_leader=false\n+ ‘[’ -n /home/pioreactor/pioreactor_air_bubbler-0.3.1-py3-none-any.whl ‘]’\n+ sudo pip3 install --force-reinstall --no-index /home/pioreactor/pioreactor_air_bubbler-0.3.1-py3-none-any.whl\nWARNING: Requirement ‘/home/pioreactor/pioreactor_air_bubbler-0.3.1-py3-none-any.whl’ looks like a filename, but the file does not exist\nERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: ‘/home/pioreactor/pioreactor_air_bubbler-0.3.1-py3-none-any.whl’\n\n"
Traceback (most recent call last):
File “/usr/local/bin/pio”, line 8, in
sys.exit(pio())
^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/plugin_management/install_plugin.py”, line 44, in click_install_plugin
install_plugin(name_of_plugin, source)
File “/usr/local/lib/python3.11/dist-packages/pioreactor/plugin_management/install_plugin.py”, line 33, in install_plugin
raise BashScriptError(f"Failed to install plugin {name_of_plugin}. See logs.")
pioreactor.exc.BashScriptError: Failed to install plugin pioreactor-air-bubbler. See logs.

EDITED: All problems were sorted out.
I deleted a serie of unnecessary files that I put on the leader when installed the air bubbler plugin. Those files are <init.py>, <setup.py>, <README.md>,<air_bubbler.py>, and <additional_config>.
This made my leader can do its jobs and the air bubbler pump works!

For workers, I put the file name <pioreactor_air_bubbler-0.3.1-py3-none-any.whl. to each of worker via WinSCP. And run the command the you suggested
pio plugins install pioreactor-air-bubbler --source ~/pioreactor_air_bubbler-0.3.1-py3-none-any.whl

The output showed the successfully installation of the plugin.
And now, the air bubbler pumps are working in all of my workers.

Still, I’m a bit curious why existing of some files affect the pioreactor.

Anyway, thank you so much for your help, Cameron

CamDavidsonPilon · October 17, 2024, 1:06am

So I’ll give some advice on how to determine what the error might be when a Pioreactor (leader or worker) isn’t starting a job when the job is triggered from the UI, which I think is the case for both of you. Below is only relevant for version >= 24.9.25.

With the UI open, click the START button for the worker (or leader) and job that refuses to start. Let’s say it’s stirring on worker pio01.
The UI will likely spin, and nothing occurs.
Next, SSH into the worker pio01.
Open up the logs with pio logs. Do you see a line that says something like INFO Executing nohup /usr/local/bin/pio run stirring?
- if not, then likely there’s a problem on the webserver or huey. Try to check if it’s responding by running the following:
```
curl localhost/unit_api/jobs/running
```
  Does that return some metadata about jobs, or just fail to connect? If the latter, then the problem is with lighttpd, the web server. Try to learn more from sudo systemctl status lighttpd.service. If the former, huey may not be working, look into sudo systemctl status huey.service.
- You saw Executing nohup /usr/local/bin/pio run stirring but stirring isn’t starting. Something might be wrong with the start sequence of jobs. Try running pio run stirring on the worker’s command line. Does it fail? If so, how?

You’ve gone through the above, and it’s still not helping? Maybe there is a problem communicating between the leader and worker. On the leader now, try pio logs and click START from the UI on the problem job. Any error messages come up?
Is the UI working? If you navigate to the Edit Config page, does it look expected to you, or is their HTML/an error message where there should be configuration? This might be a leader web server problem. Try running sudo systemctl status lighttpd.server on the leader to learn more.

Try these things, and let me know what you find! Specifically we are looking for error messages.

LeoKleym · October 21, 2024, 11:42am

hey! alright, so i tried your suggestions and this came up:

so starting the job with ssh connection is working, but still not via the ui and i didn’t see any error messages.

pio logs and my leader does this:

pioreactor@pioreactor2:~ $ pio logs
2024-10-20T20:40:26+0200 [monitor] DEBUG Heating PCB temperature at 24 ℃.
2024-10-21T00:00:02+0200 [backup_database] DEBUG Starting backup of database to /home/pioreactor/.pioreactor/storage/pioreactor.sqlite.backup
2024-10-21T00:03:23+0200 [backup_database] INFO Completed backup of database.
2024-10-21T00:03:23+0200 [backup_database] DEBUG Attempting backing up database to pioreactor5.
2024-10-21T00:03:40+0200 [backup_database] DEBUG Backed up database to pioreacto r5:/home/pioreactor/.pioreactor/storage/pioreactor.sqlite.backup.
2024-10-21T00:03:41+0200 [backup_database] DEBUG Attempting backing up database to pioreactor4.
2024-10-21T00:03:53+0200 [backup_database] DEBUG Backed up database to pioreacto r4:/home/pioreactor/.pioreactor/storage/pioreactor.sqlite.backup.
2024-10-21T00:40:21+0200 [monitor] DEBUG IPv4 address: 10.131.125.67
2024-10-21T00:40:21+0200 [monitor] DEBUG WLAN MAC address: b8:27:eb:85:eb:da
2024-10-21T00:40:21+0200 [monitor] DEBUG Ethernet MAC address: b8:27:eb:d0:be:8f
2024-10-21T00:40:21+0200 [monitor] DEBUG Disk space at 10%.
2024-10-21T00:40:21+0200 [monitor] DEBUG CPU temperature at 54 ℃.
2024-10-21T00:40:26+0200 [monitor] DEBUG Pioreactor UI version: 24.10.1
2024-10-21T00:40:26+0200 [monitor] DEBUG PWM power supply at ~5.20V.
2024-10-21T00:40:26+0200 [monitor] DEBUG Power status okay.
2024-10-21T00:40:26+0200 [monitor] DEBUG Heating PCB temperature at 24 ℃.
2024-10-21T04:40:21+0200 [monitor] DEBUG IPv4 address: 10.131.125.67
2024-10-21T04:40:21+0200 [monitor] DEBUG WLAN MAC address: b8:27:eb:85:eb:da
2024-10-21T04:40:21+0200 [monitor] DEBUG Ethernet MAC address: b8:27:eb:d0:be:8f
2024-10-21T04:40:21+0200 [monitor] DEBUG Disk space at 10%.
2024-10-21T04:40:21+0200 [monitor] DEBUG CPU temperature at 54 ℃.
2024-10-21T04:40:26+0200 [monitor] DEBUG Pioreactor UI version: 24.10.1
2024-10-21T04:40:26+0200 [monitor] DEBUG PWM power supply at ~5.20V.
2024-10-21T04:40:26+0200 [monitor] DEBUG Power status okay.
2024-10-21T04:40:26+0200 [monitor] DEBUG Heating PCB temperature at 24 ℃.
2024-10-21T08:40:21+0200 [monitor] DEBUG IPv4 address: 10.131.125.67
2024-10-21T08:40:21+0200 [monitor] DEBUG WLAN MAC address: b8:27:eb:85:eb:da
2024-10-21T08:40:21+0200 [monitor] DEBUG Ethernet MAC address: b8:27:eb:d0:be:8f
2024-10-21T08:40:21+0200 [monitor] DEBUG Disk space at 10%.
2024-10-21T08:40:21+0200 [monitor] DEBUG CPU temperature at 54 ℃.
2024-10-21T08:40:26+0200 [monitor] DEBUG Pioreactor UI version: 24.10.1
2024-10-21T08:40:26+0200 [monitor] DEBUG PWM power supply at ~5.20V.
2024-10-21T08:40:26+0200 [monitor] DEBUG Power status okay.
2024-10-21T08:40:26+0200 [monitor] DEBUG Heating PCB temperature at 25 ℃.
2024-10-21T12:40:21+0200 [monitor] DEBUG IPv4 address: 10.131.125.67
2024-10-21T12:40:21+0200 [monitor] DEBUG WLAN MAC address: b8:27:eb:85:eb:da
2024-10-21T12:40:21+0200 [monitor] DEBUG Ethernet MAC address: b8:27:eb:d0:be:8f
2024-10-21T12:40:21+0200 [monitor] DEBUG Disk space at 10%.
2024-10-21T12:40:21+0200 [monitor] DEBUG CPU temperature at 53 ℃.
2024-10-21T12:40:26+0200 [monitor] DEBUG Pioreactor UI version: 24.10.1
2024-10-21T12:40:26+0200 [monitor] DEBUG PWM power supply at ~5.20V.
2024-10-21T12:40:26+0200 [monitor] DEBUG Power status okay.
2024-10-21T12:40:26+0200 [monitor] DEBUG Heating PCB temperature at 25 ℃.
2024-10-21T13:30:37+0200 [pioreactorui-pioreactor2] ERROR Exception on /api/work ers/pioreactor1/blink [POST]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1463, in wsg i_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 873, in full _dispatch_request
return self.finalize_request(rv)
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 892, in fina lize_request
response = self.make_response(rv)
^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1164, in mak e_response
raise TypeError(
TypeError: The view function for ‘api.blink_worker’ did not return a valid respo nse. The function either returned None or ended without a return statement.
2024-10-21T13:30:39+0200 [pioreactorui-pioreactor2] ERROR Exception on /api/work ers/pioreactor2/blink [POST]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1463, in wsg i_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 873, in full _dispatch_request
return self.finalize_request(rv)
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 892, in fina lize_request
response = self.make_response(rv)
^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1164, in mak e_response
raise TypeError(
TypeError: The view function for ‘api.blink_worker’ did not return a valid respo nse. The function either returned None or ended without a return statement.
2024-10-21T13:30:41+0200 [pioreactorui-pioreactor2] ERROR Exception on /api/work ers/pioreactor3/blink [POST]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1463, in wsg i_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 873, in full _dispatch_request
return self.finalize_request(rv)
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 892, in fina lize_request
response = self.make_response(rv)
^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1164, in mak e_response
raise TypeError(
TypeError: The view function for ‘api.blink_worker’ did not return a valid respo nse. The function either returned None or ended without a return statement.
2024-10-21T13:34:07+0200 [huey.consumer] INFO Executing pio plugins list --json
2024-10-21T13:35:49+0200 [pioreactorui-pioreactor2] ERROR Exception on /api/work ers/pioreactor3/blink [POST]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1463, in wsg i_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 873, in full _dispatch_request
return self.finalize_request(rv)
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 892, in fina lize_request
response = self.make_response(rv)
^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1164, in mak e_response
raise TypeError(
TypeError: The view function for ‘api.blink_worker’ did not return a valid respo nse. The function either returned None or ended without a return statement.

so there is an error message.

also interesting: i can stop the job from the ui but i cant start it.

CamDavidsonPilon · October 21, 2024, 1:31pm

So the error first presents itself in the first command:

curl localhost/unit_api/jobs/running

In your screenshot, it looks like it this errors (it returns the wrong thing), and likely what is causing jobs not to start from the UI.

Hm, does this following command show any errors

flask --app /var/www/pioreactorui/main.py run

(cltr-c to exit)

LeoKleym · October 22, 2024, 6:22am

pioreactor@pioreactor3:~ $ flask --app /var/www/pioreactorui/main.py run

Serving Flask app ‘/var/www/pioreactorui/main.py’
Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Running on http://127.0.0.1:5000
Press CTRL+C to quit

this is what comes up when i try the command with the worker

CamDavidsonPilon · October 23, 2024, 12:26pm

Yea that looks fine. Hm hm hm, I’m still unsure why your server isn’t responding correctly. Can you try

curl -v localhost/unit_api/jobs/running

and

cat /etc/lighttpd/conf-enabled/50-pioreactorui.conf

LeoKleym · October 28, 2024, 9:25am

so this is what comes up then:

pioreactor@pioreactor3:~ $ curl -v localhost/unit_api/jobs/running

Trying 127.0.0.1:80…
Connected to localhost (127.0.0.1) port 80 (#0)

GET /unit_api/jobs/running HTTP/1.1
Host: localhost
User-Agent: curl/7.88.1
Accept: /

< HTTP/1.1 200 OK
< Content-Type: text/html
< ETag: “2922167695”
< Last-Modified: Sun, 20 Oct 2024 23:48:19 GMT
< Content-Length: 525
< Accept-Ranges: bytes
< Date: Mon, 28 Oct 2024 09:19:07 GMT
< Server: lighttpd/1.4.69
<
</he ad>

Connection #0 to host localhost left intact

^C
pioreactor@pioreactor3:~ $ cat /etc/lighttpd/conf-enabled/50-pioreactorui.conf

url.rewrite-once = (
“^(/static($|/.))$" => “$1”,
"^(/api/.)$” => “/main.fcgi$1”
)
alias.url += (
“/static/” => “/var/www/pioreactorui/static/”
)
server.error-handler-404 = “/index.html”

fastcgi.server += (“.fcgi” =>
((
“socket” => “/tmp/main-fcgi.sock”,
“bin-path” => “/var/www/pioreactorui/main.fcgi”,
“check-local” => “disable”,
“max-procs” => 1,
))
)

CamDavidsonPilon · October 28, 2024, 6:59pm

I found it! Ok, some part of your updating process didn’t apply. Try this on each Pioreactor thats not working:

sudo sed -i '/^url.rewrite-once = (/a\  "^(/unit_api/.*)$" => "/main.fcgi$1",' /etc/lighttpd/conf-available/50-pioreactorui.conf

Your final 50-pioreactorui.conf file should look like: CustoPiZer/workspace/scripts/files/system/lighttpd/50-pioreactorui.conf at pioreactor · Pioreactor/CustoPiZer · GitHub

Then restart your server:

sudo systemctl restart lighttpd.service

And confirm it works with:

curl localhost/unit_api/jobs/running

LeoKleym · October 29, 2024, 1:46pm

heyhey!
so this is what comes up for most of my workers:
pioreactor@pioreactor4:~ $ sudo sed -i ‘/^url.rewrite-once = (/a\ “^(/unit_api/.*)$” => “/main.fcgi$1”,’ /etc/lighttpd/conf-available/50-pioreactorui.conf
pioreactor@pioreactor4:~ $ sudo systemctl restart lighttpd.service
pioreactor@pioreactor4:~ $ curl localhost/unit_api/jobs/running
[{“ended_at”:null,“experiment”:“$experiment”,“id”:1,“is_running”:1,“job_source”:“user”,“leader”:“pioreactor2”,“name”:“monitor”,“pid”:740,“started_at”:“2024-10-08T10:27:41.377000Z”,“unit”:“pioreactor4”}]
pioreactor@pioreactor4:~ $

for one though this happens:
pioreactor@pioreactor1:~ $ sudo sed -i ‘/^url.rewrite-once = (/a\ “^(/unit_api/.*)$” => “/main.fcgi$1”,’ /etc/lighttpd/conf-available/50-pioreactorui.conf
pioreactor@pioreactor1:~ $ sudo systemctl restart lighttpd.service
Job for lighttpd.service failed because the control process exited with error code.
See “systemctl status lighttpd.service” and “journalctl -xeu lighttpd.service” for details.
pioreactor@pioreactor1:~ $ curl localhost/unit_api/jobs/running
curl: (7) Failed to connect to localhost port 80 after 4 ms: Couldn’t connect to server
pioreactor@pioreactor1:~ $

CamDavidsonPilon · October 29, 2024, 3:08pm

For that last one thats erroring, check that the /etc/lighttpd/conf-available/50-pioreactorui.conf looks like my linked example above. But also check:

sudo systemctl status lighttpd.service

for more info