Calibration graphs on UI

HillsLaura · January 28, 2025, 6:46pm

Hello!

I recently updated the Pioreactors to 2025.01.21 and have been able to view my calibration graphs on the UI only for my leader and not my other set ups. I recently was doing new pump calibrations on one of my workers, and while in the command line it says the calibration is active, a graph is not appearing in my UI for that specific Pioreactor and instead says no data found.

Any help with this? Thanks!

*This is the message received in the command line after activating the calibration.

Calibration is best for volumes between 0.03mL to 0.11mL, but will be okay for outside this range too.
Finished media_pump calibration media_pump-2025-01-28.
Do you want to set this calibration as the Active Calibration for media_pump? [Y/n]: Y
Setmedia_pump-2025-01-28 as the active calibration for media_pump.
Calibration ‘media_pump-2025-01-28’ of device ‘media_pump’ saved to /home/pioreactor/.pioreactor/storage/calibrations/media_pump/media_pump-2025-01-28.yaml

CamDavidsonPilon · January 28, 2025, 7:43pm

Hi @HillsLaura, welcome!

Hm, your process sounds correct to me.

Can you try the following? SSH back into that worker, and run:

curl localhost/unit_api/calibrations | jq

Do you see the calibration media_pump-2025-01-28 in that output? And if so, what does the is_active field say?

HillsLaura · January 30, 2025, 4:30pm

Heyo!

When I ran the command through on one of my workers, the below was generated. I additonally ran the command on another worker that had the same issue and the same response was generated. However, when running the line on my leader, I was able to get a result reading my most recent media pump calibration was active.

pioreactor@pioreactor02:~ $ curl localhost/unit_api/calibrations | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 209 100 209 0 0 16500 0 --:–:-- --:–:-- --:–:-- 17416
{
“error”: “Internal server error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application… See logs for more.”

CamDavidsonPilon · January 30, 2025, 6:14pm

Hi @HillsLaura,

hm, let’s try two things:

Can you see if there are any errors when you look at the logs with pio logs (on that worker)? ctrl-c to exit
Try sudo systemctl restart lighttpd.service, and try the curl test again? Still erroring, try looking for errors in sudo systemctl status lighttpd.service

HillsLaura · January 30, 2025, 6:53pm

Hey Cam,

I ran the logs on the worker and this is what came up - I’m not super knowledgeable in Python. I then did the reset and ran those logs and those are attached as well. The curl line did the same initial thing even after the reset.

Thanks

pioreactor@pioreactor02:~ $ pio logs
2025-01-28T18:24:17+0000 [pioreactorui-pioreactor02] DEBUG Starting pioreactorui-pioreactor02=25.1.21 on pioreactor02…
2025-01-28T18:24:17+0000 [pioreactorui-pioreactor02] DEBUG .env={‘DOT_PIOREACTOR’: ‘/home/pioreactor/.pioreactor/’, ‘WWW’: ‘/var/www/pioreactorui/’, ‘PIO_EXECUTABLE’: ‘’, ‘PIOS_EXECUTABLE’: ‘’}
2025-01-28T18:24:40+0000 [pioreactorui-pioreactor02] ERROR Exception on /unit_api/calibrations [GET]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1511, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 919, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/www/pioreactorui/pioreactorui/unit_api.py”, line 625, in get_all_calibrations
with local_persistent_storage(“active_calibrations”) as cache:
File “/usr/lib/python3.11/contextlib.py”, line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 413, in local_persistent_storage
with cache(cache_name, db_path=config.get(“storage”, “persistent_cache”)) as c:
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 295, in enter
self.cursor.executescript(
sqlite3.OperationalError: attempt to write a readonly database
2025-01-28T18:26:43+0000 [pioreactorui-pioreactor02] DEBUG Starting pioreactorui-pioreactor02=25.1.21 on pioreactor02…
2025-01-28T18:26:43+0000 [pioreactorui-pioreactor02] DEBUG .env={‘DOT_PIOREACTOR’: ‘/home/pioreactor/.pioreactor/’, ‘WWW’: ‘/var/www/pioreactorui/’, ‘PIO_EXECUTABLE’: ‘’, ‘PIOS_EXECUTABLE’: ‘’}
2025-01-28T18:27:17+0000 [pioreactorui-pioreactor02] ERROR Exception on /unit_api/calibrations [GET]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1511, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 919, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/www/pioreactorui/pioreactorui/unit_api.py”, line 625, in get_all_calibrations
with local_persistent_storage(“active_calibrations”) as cache:
File “/usr/lib/python3.11/contextlib.py”, line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 413, in local_persistent_storage
with cache(cache_name, db_path=config.get(“storage”, “persistent_cache”)) as c:
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 295, in enter
self.cursor.executescript(
sqlite3.OperationalError: attempt to write a readonly database
2025-01-28T18:27:45+0000 [pioreactorui-pioreactor02] ERROR Exception on /unit_api/calibrations [GET]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1511, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 919, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/www/pioreactorui/pioreactorui/unit_api.py”, line 625, in get_all_calibrations
with local_persistent_storage(“active_calibrations”) as cache:
File “/usr/lib/python3.11/contextlib.py”, line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 413, in local_persistent_storage
with cache(cache_name, db_path=config.get(“storage”, “persistent_cache”)) as c:
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 295, in enter
self.cursor.executescript(
sqlite3.OperationalError: attempt to write a readonly database
2025-01-28T18:28:05+0000 [pioreactorui-pioreactor02] ERROR Exception on /unit_api/calibrations [GET]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1511, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 919, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/www/pioreactorui/pioreactorui/unit_api.py”, line 625, in get_all_calibrations
with local_persistent_storage(“active_calibrations”) as cache:
File “/usr/lib/python3.11/contextlib.py”, line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 413, in local_persistent_storage
with cache(cache_name, db_path=config.get(“storage”, “persistent_cache”)) as c:
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 295, in enter
self.cursor.executescript(
sqlite3.OperationalError: attempt to write a readonly databases

pioreactor@pioreactor02:~ $ sudo systemctl status lighttpd.service
● lighttpd.service - Lighttpd Daemon
Loaded: loaded (/etc/systemd/system/lighttpd.service; enabled; preset: enabled)
Active: active (running) since Tue 2025-01-28 18:33:58 GMT; 27s ago
Process: 852 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
Main PID: 857 (lighttpd)
Tasks: 7 (limit: 494)
CPU: 3.961s
CGroup: /system.slice/lighttpd.service
├─857 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
└─862 /usr/bin/python3 /var/www/pioreactorui/main.fcgi

Jan 28 18:33:57 pioreactor02 systemd[1]: Starting lighttpd.service - Lighttpd Daemon…
Jan 28 18:33:58 pioreactor02 systemd[1]: Started lighttpd.service - Lighttpd Daemon.

CamDavidsonPilon · January 30, 2025, 7:02pm

Ah, I think this is a bug in the latest update. Try the following:

chown -R pioreactor:www-data /home/pioreactor/.pioreactor/storage
chmod -R 770 /home/pioreactor/.pioreactor/storage

and then try:

curl localhost/unit_api/calibrations | jq

If you see an error again from that curl command, try pio logs and lmk what you see there.

HillsLaura · January 30, 2025, 8:04pm

This worked! The graphs popped up in the UI, thanks so much!!