Calibration graphs on UI

HillsLaura · January 28, 2025, 6:46pm

Hello!

I recently updated the Pioreactors to 2025.01.21 and have been able to view my calibration graphs on the UI only for my leader and not my other set ups. I recently was doing new pump calibrations on one of my workers, and while in the command line it says the calibration is active, a graph is not appearing in my UI for that specific Pioreactor and instead says no data found.

Any help with this? Thanks!

*This is the message received in the command line after activating the calibration.

Calibration is best for volumes between 0.03mL to 0.11mL, but will be okay for outside this range too.
Finished media_pump calibration media_pump-2025-01-28.
Do you want to set this calibration as the Active Calibration for media_pump? [Y/n]: Y
Setmedia_pump-2025-01-28 as the active calibration for media_pump.
Calibration ‘media_pump-2025-01-28’ of device ‘media_pump’ saved to /home/pioreactor/.pioreactor/storage/calibrations/media_pump/media_pump-2025-01-28.yaml

CamDavidsonPilon · January 28, 2025, 7:43pm

Hi @HillsLaura, welcome!

Hm, your process sounds correct to me.

Can you try the following? SSH back into that worker, and run:

curl localhost/unit_api/calibrations | jq

Do you see the calibration media_pump-2025-01-28 in that output? And if so, what does the is_active field say?

HillsLaura · January 30, 2025, 4:30pm

Heyo!

When I ran the command through on one of my workers, the below was generated. I additonally ran the command on another worker that had the same issue and the same response was generated. However, when running the line on my leader, I was able to get a result reading my most recent media pump calibration was active.

pioreactor@pioreactor02:~ $ curl localhost/unit_api/calibrations | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 209 100 209 0 0 16500 0 --:–:-- --:–:-- --:–:-- 17416
{
“error”: “Internal server error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application… See logs for more.”

CamDavidsonPilon · January 30, 2025, 6:14pm

Hi @HillsLaura,

hm, let’s try two things:

Can you see if there are any errors when you look at the logs with pio logs (on that worker)? ctrl-c to exit
Try sudo systemctl restart lighttpd.service, and try the curl test again? Still erroring, try looking for errors in sudo systemctl status lighttpd.service

HillsLaura · January 30, 2025, 6:53pm

Hey Cam,

I ran the logs on the worker and this is what came up - I’m not super knowledgeable in Python. I then did the reset and ran those logs and those are attached as well. The curl line did the same initial thing even after the reset.

Thanks

pioreactor@pioreactor02:~ $ pio logs
2025-01-28T18:24:17+0000 [pioreactorui-pioreactor02] DEBUG Starting pioreactorui-pioreactor02=25.1.21 on pioreactor02…
2025-01-28T18:24:17+0000 [pioreactorui-pioreactor02] DEBUG .env={‘DOT_PIOREACTOR’: ‘/home/pioreactor/.pioreactor/’, ‘WWW’: ‘/var/www/pioreactorui/’, ‘PIO_EXECUTABLE’: ‘’, ‘PIOS_EXECUTABLE’: ‘’}
2025-01-28T18:24:40+0000 [pioreactorui-pioreactor02] ERROR Exception on /unit_api/calibrations [GET]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1511, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 919, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/www/pioreactorui/pioreactorui/unit_api.py”, line 625, in get_all_calibrations
with local_persistent_storage(“active_calibrations”) as cache:
File “/usr/lib/python3.11/contextlib.py”, line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 413, in local_persistent_storage
with cache(cache_name, db_path=config.get(“storage”, “persistent_cache”)) as c:
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 295, in enter
self.cursor.executescript(
sqlite3.OperationalError: attempt to write a readonly database
2025-01-28T18:26:43+0000 [pioreactorui-pioreactor02] DEBUG Starting pioreactorui-pioreactor02=25.1.21 on pioreactor02…
2025-01-28T18:26:43+0000 [pioreactorui-pioreactor02] DEBUG .env={‘DOT_PIOREACTOR’: ‘/home/pioreactor/.pioreactor/’, ‘WWW’: ‘/var/www/pioreactorui/’, ‘PIO_EXECUTABLE’: ‘’, ‘PIOS_EXECUTABLE’: ‘’}
2025-01-28T18:27:17+0000 [pioreactorui-pioreactor02] ERROR Exception on /unit_api/calibrations [GET]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1511, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 919, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/www/pioreactorui/pioreactorui/unit_api.py”, line 625, in get_all_calibrations
with local_persistent_storage(“active_calibrations”) as cache:
File “/usr/lib/python3.11/contextlib.py”, line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 413, in local_persistent_storage
with cache(cache_name, db_path=config.get(“storage”, “persistent_cache”)) as c:
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 295, in enter
self.cursor.executescript(
sqlite3.OperationalError: attempt to write a readonly database
2025-01-28T18:27:45+0000 [pioreactorui-pioreactor02] ERROR Exception on /unit_api/calibrations [GET]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1511, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 919, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/www/pioreactorui/pioreactorui/unit_api.py”, line 625, in get_all_calibrations
with local_persistent_storage(“active_calibrations”) as cache:
File “/usr/lib/python3.11/contextlib.py”, line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 413, in local_persistent_storage
with cache(cache_name, db_path=config.get(“storage”, “persistent_cache”)) as c:
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 295, in enter
self.cursor.executescript(
sqlite3.OperationalError: attempt to write a readonly database
2025-01-28T18:28:05+0000 [pioreactorui-pioreactor02] ERROR Exception on /unit_api/calibrations [GET]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 1511, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 919, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/flask/app.py”, line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/var/www/pioreactorui/pioreactorui/unit_api.py”, line 625, in get_all_calibrations
with local_persistent_storage(“active_calibrations”) as cache:
File “/usr/lib/python3.11/contextlib.py”, line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 413, in local_persistent_storage
with cache(cache_name, db_path=config.get(“storage”, “persistent_cache”)) as c:
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 295, in enter
self.cursor.executescript(
sqlite3.OperationalError: attempt to write a readonly databases

pioreactor@pioreactor02:~ $ sudo systemctl status lighttpd.service
● lighttpd.service - Lighttpd Daemon
Loaded: loaded (/etc/systemd/system/lighttpd.service; enabled; preset: enabled)
Active: active (running) since Tue 2025-01-28 18:33:58 GMT; 27s ago
Process: 852 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
Main PID: 857 (lighttpd)
Tasks: 7 (limit: 494)
CPU: 3.961s
CGroup: /system.slice/lighttpd.service
├─857 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
└─862 /usr/bin/python3 /var/www/pioreactorui/main.fcgi

Jan 28 18:33:57 pioreactor02 systemd[1]: Starting lighttpd.service - Lighttpd Daemon…
Jan 28 18:33:58 pioreactor02 systemd[1]: Started lighttpd.service - Lighttpd Daemon.

CamDavidsonPilon · January 30, 2025, 7:02pm

Ah, I think this is a bug in the latest update. Try the following:

chown -R pioreactor:www-data /home/pioreactor/.pioreactor/storage
chmod -R 770 /home/pioreactor/.pioreactor/storage

and then try:

curl localhost/unit_api/calibrations | jq

If you see an error again from that curl command, try pio logs and lmk what you see there.

HillsLaura · January 30, 2025, 8:04pm

This worked! The graphs popped up in the UI, thanks so much!!

belenadiego · March 27, 2025, 2:06pm

Hello!
We have kind of the same issue. After updating our two clusters to the most recent app and ui versions (25.3.5), we see the following:

cluster 1: on the UI, we can only see the graphs for the leader. When we try to download the files, they only download for the leader.
cluster 2: on the UI, we can only see the graphs for the leader again. When we try to download the files, they download for the Leader and for the 4 workers.

We checked the SD cards with FileZilla and the calibration files are in pioreactor/storage/calibration for all units.

Could you help us with this?

Thanks in advance!

CamDavidsonPilon · March 28, 2025, 1:08am

hi @belenadiego,

Did you try the solution here on a worker that’s not showing up? Calibration graphs on UI - #6 by CamDavidsonPilon

What does

curl localhost/unit_api/calibrations | jq

when SSH’d into the worker?

belenadiego · March 28, 2025, 9:22am

Hi @CamDavidsonPilon ,

When SSH’d into the worker, I get the following:

pioreactor@Worker2-1:~ $ chown -R pioreactor:www-data /home/pioreactor/.pioreactor/storage
pioreactor@Worker2-1:~ $ chown -R 770 /home/pioreactor/.pioreactor/storage
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/od_blank/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/od_blank': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/current_pump_calibration/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/current_pump_calibration': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/current_od_calibration/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/current_od_calibration': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/pump_calibrations/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/pump_calibrations': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/od_filtered/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/od_filtered': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/alt_media_fraction/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/alt_media_fraction': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/media_throughput/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/media_throughput': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/calibrations/media_pump/20241009 W2-1 mediapump.yaml': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/calibrations/media_pump': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/calibrations/alt_media_pump': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/calibrations/od/20241003 M9 W2_1.yaml': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/calibrations/od': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/calibrations/waste_pump/20241009 W2-1 wastepump.yaml': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/calibrations/waste_pump': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/calibrations': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/self_test_results/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/self_test_results': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/od_normalization_mean/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/od_normalization_mean': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/od_normalization_variance/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/od_normalization_variance': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/alt_media_throughput/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/alt_media_throughput': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/od_calibrations/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/od_calibrations': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/growth_rate/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/growth_rate': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/vial_volume/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/vial_volume': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/stirring_calibration/cache.db': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage/stirring_calibration': Operation not permitted
chown: changing ownership of '/home/pioreactor/.pioreactor/storage': Operation not permitted
pioreactor@Worker2-1:~ $ curl localhost/unit_api/calibrations | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   210  100   210    0     0  23521      0 --:--:-- --:--:-- --:--:-- 26250
{
  "error": "Internal server error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application. - see logs for more."

Thanks!

CamDavidsonPilon · March 28, 2025, 9:37am

Huh, try the following:

sudo chown -R pioreactor:www-data /home/pioreactor/.pioreactor/storage
sudo chmod -R 770 /home/pioreactor/.pioreactor/storage

belenadiego · March 28, 2025, 10:43am

Apparently it does nothing:

pioreactor@Worker2-1:~ $ sudo chown -R pioreactor:www-data /home/pioreactor/.pioreactor/storage
pioreactor@Worker2-1:~ $ sudo chown -R 770 /home/pioreactor/.pioreactor/storage
pioreactor@Worker2-1:~ $ curl localhost/unit_api/calibrations | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   210  100   210    0     0  22236      0 --:--:-- --:--:-- --:--:-- 23333
{
  "error": "Internal server error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application. - see logs for more."
}

What changed now is that when I click on “Download all calibrations” on the UI, I only get the zip file that contains the calibration files for Leader 2, Worker2-3, Worker2-4 and Worker2-5 (Worker2-1 is now missing). Before, for this cluster, I was getting the calibration files for all the Workers and the Leader (even though they did not show in the UI).

belenadiego · March 28, 2025, 10:53am

And I can see the calibrations of the worker when accessing through the command line:

pioreactor@Worker2-1:~ $ pio calibrations list
Device                   Name                                              Created At               Active?
--------------------------------------------------------------------------------------------------------------
media_pump               20241009 W2-1 mediapump                           2024-10-08 10:34:23      ✅
od                       20241003 M9 W2_1                                  2024-09-18 15:03:57      ✅
waste_pump               20241009 W2-1 wastepump                           2024-10-08 10:38:01      ✅

CamDavidsonPilon · March 28, 2025, 10:58am

On that worker, can you try: pio logs - do you see any errors there?

Next, try:

sudo systemctl restart lighttpd.service
curl localhost/unit_api/calibrations | jq

belenadiego · March 28, 2025, 11:03am

With: pio logs, I see:

pioreactor@Worker2-1:~ $ pio logs
+ pioreactorui.tasks.sync_clock
+ pioreactorui.tasks.pio_update_app
+ pioreactorui.tasks.pio_update
+ pioreactorui.tasks.pio_update_ui
+ pioreactorui.tasks.rm
+ pioreactorui.tasks.shutdown
+ pioreactorui.tasks.reboot
+ pioreactorui.tasks.pios
+ pioreactorui.tasks.save_file
+ pioreactorui.tasks.write_config_and_sync
+ pioreactorui.tasks.post_to_worker
+ pioreactorui.tasks.multicast_post_across_cluster
+ pioreactorui.tasks.get_from_worker
+ pioreactorui.tasks.multicast_get_across_cluster
+ pioreactorui.tasks.patch_to_worker
+ pioreactorui.tasks.multicast_patch_across_cluster
+ pioreactorui.tasks.delete_from_worker
+ pioreactorui.tasks.multicast_delete_across_cluster
2025-03-28T11:44:29+0100 [huey.consumer] INFO Starting Huey consumer...
2025-03-28T11:44:29+0100 [huey.consumer] INFO Starting Huey consumer...
2025-03-28T11:44:29+0100 [huey.consumer] INFO Starting Huey consumer...
2025-03-28T11:44:29+0100 [huey.consumer] INFO Cache directory = /tmp/pioreactor_cache
2025-03-28T11:44:29+0100 [huey.consumer] INFO Starting Huey consumer...
2025-03-28T11:44:29+0100 [huey.consumer] INFO Cache directory = /tmp/pioreactor_cache
2025-03-28T11:44:29+0100 [huey.consumer] INFO Starting Huey consumer...
2025-03-28T11:44:29+0100 [huey.consumer] INFO Starting Huey consumer...
2025-03-28T11:44:29+0100 [huey.consumer] INFO Starting Huey consumer...
2025-03-28T11:44:29+0100 [huey.consumer] INFO Cache directory = /tmp/pioreactor_cache
2025-03-28T11:44:29+0100 [huey.consumer] INFO Cache directory = /tmp/pioreactor_cache
2025-03-28T11:44:29+0100 [huey.consumer] INFO Starting Huey consumer...
2025-03-28T11:44:29+0100 [huey.consumer] INFO Cache directory = /tmp/pioreactor_cache
2025-03-28T11:44:29+0100 [huey.consumer] INFO Starting Huey consumer...
2025-03-28T11:44:29+0100 [huey.consumer] INFO Cache directory = /tmp/pioreactor_cache
2025-03-28T11:44:29+0100 [huey.consumer] INFO Cache directory = /tmp/pioreactor_cache
2025-03-28T11:44:29+0100 [huey.consumer] INFO Starting Huey consumer...
2025-03-28T11:44:29+0100 [huey.consumer] INFO Cache directory = /tmp/pioreactor_cache
2025-03-28T11:44:29+0100 [huey.consumer] INFO Cache directory = /tmp/pioreactor_cache
2025-03-28T11:44:29+0100 [huey.consumer] INFO Cache directory = /tmp/pioreactor_cache
2025-03-28T11:44:42+0100 [pioreactorui-Worker2-1] DEBUG Starting pioreactorui-Worker2-1=25.3.5 on Worker2-1...
2025-03-28T11:44:42+0100 [pioreactorui-Worker2-1] DEBUG .env={'DOT_PIOREACTOR': '/home/pioreactor/.pioreactor/', 'WWW': '/var/www/pioreactorui/', 'PIO_EXECUTABLE': '', 'PIOS_EXECUTABLE': ''}
2025-03-28T11:44:43+0100 [monitor] NOTICE Worker2-1 is online and ready.
2025-03-28T11:44:43+0100 [monitor] INFO Ready.
2025-03-28T11:44:43+0100 [monitor] DEBUG monitor is blocking until disconnected.
2025-03-28T11:44:44+0100 [monitor] DEBUG IPv4 address: 10.42.0.240
2025-03-28T11:44:44+0100 [monitor] DEBUG WLAN MAC address: b8:27:eb:c2:4a:71
2025-03-28T11:44:44+0100 [monitor] DEBUG Ethernet MAC address: b8:27:eb:97:1f:24
2025-03-28T11:44:44+0100 [monitor] DEBUG Disk space at 12%.
2025-03-28T11:44:44+0100 [monitor] DEBUG CPU temperature at 52 ℃.
2025-03-28T11:44:44+0100 [monitor] DEBUG Pioreactor UI version: 25.3.5
2025-03-28T11:44:44+0100 [monitor] DEBUG PWM power supply at ~5.05V.
2025-03-28T11:44:44+0100 [monitor] DEBUG Power status okay.
2025-03-28T11:44:44+0100 [monitor] DEBUG Heating PCB temperature at 24 ℃.
2025-03-28T11:55:47+0100 [pioreactorui-Worker2-1] ERROR Exception on /unit_api/calibrations [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/www/pioreactorui/pioreactorui/unit_api.py", line 654, in get_all_calibrations
    with local_persistent_storage("active_calibrations") as cache:
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/__init__.py", line 459, in local_persistent_storage
    with cache(cache_name, db_path=config.get("storage", "persistent_cache")) as c:
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/__init__.py", line 344, in __enter__
    self.cursor.executescript(
sqlite3.OperationalError: attempt to write a readonly database
2025-03-28T11:55:57+0100 [pioreactorui-Worker2-1] ERROR Exception on /unit_api/calibrations [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/www/pioreactorui/pioreactorui/unit_api.py", line 654, in get_all_calibrations
    with local_persistent_storage("active_calibrations") as cache:
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/__init__.py", line 459, in local_persistent_storage
    with cache(cache_name, db_path=config.get("storage", "persistent_cache")) as c:
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/__init__.py", line 344, in __enter__
    self.cursor.executescript(
sqlite3.OperationalError: attempt to write a readonly database

And it does not finish the process.

belenadiego · March 28, 2025, 11:07am

And after:

pioreactor@Worker2-1:~ $ sudo systemctl restart lighttpd.service
pioreactor@Worker2-1:~ $ curl localhost/unit_api/calibrations | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   210  100   210    0     0   9409      0 --:--:-- --:--:-- --:--:--  9545
{
  "error": "Internal server error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application. - see logs for more."
}

Now I tried again to download the files, and the calibration file can be downloaded (still not shown in the graphs though).

CamDavidsonPilon · March 28, 2025, 1:14pm

So the problem is a permissions issue on a cache database. However, a prior command should have fixed it. Can you show me:

ls -al .pioreactor/storage/

belenadiego · March 28, 2025, 4:16pm

It returns:

pioreactor@Worker2-1:~ $ ls -al .pioreactor/storage/
total 84
drwxr-xr-x 18 pioreactor www-data    4096 Mar 28 12:17 .
drwxr-xr-x  5 pioreactor pioreactor  4096 Aug 22  2024 ..
drwxr-xr-x  2 pioreactor www-data    4096 Feb  4 08:24 alt_media_fraction
drwxr-xr-x  2 pioreactor www-data    4096 Feb  4 08:24 alt_media_throughput
drwxr-xr-x  6 pioreactor www-data    4096 Mar 26 16:09 calibrations
drwxr-xr-x  2 pioreactor www-data    4096 Mar 26 16:09 current_od_calibration
drwxr-xr-x  2 pioreactor www-data    4096 Mar 26 16:09 current_pump_calibration
drwxr-xr-x  2 pioreactor www-data    4096 Feb  4 08:38 growth_rate
-rwxrwx---  1 pioreactor www-data   12288 Mar 26 16:09 local_persistent_pioreactor_metadata.sqlite
drwxr-xr-x  2 pioreactor www-data    4096 Feb  4 08:24 media_throughput
drwxr-xr-x  2 pioreactor www-data    4096 Feb  1 09:50 od_blank
drwxr-xr-x  2 pioreactor www-data    4096 Mar 26 16:09 od_calibrations
drwxr-xr-x  2 pioreactor www-data    4096 Feb  4 08:38 od_filtered
drwxr-xr-x  2 pioreactor www-data    4096 Feb  1 09:50 od_normalization_mean
drwxr-xr-x  2 pioreactor www-data    4096 Feb  1 09:50 od_normalization_variance
drwxr-xr-x  2 pioreactor www-data    4096 Mar 26 16:09 pump_calibrations
drwxr-xr-x  2 pioreactor www-data    4096 Aug 22  2024 self_test_results
drwxr-xr-x  2 pioreactor www-data    4096 Feb  1 09:46 stirring_calibration
drwxr-xr-x  2 pioreactor www-data    4096 Feb  4 08:24 vial_volume

CamDavidsonPilon · March 28, 2025, 11:55pm

Hi @belenadiego, try the following

sudo chown pioreactor:www-data /home/pioreactor/.pioreactor/storage
sudo chmod g+srwx /home/pioreactor/.pioreactor/storage

And access the UI’s calibration page again. Does that work?

belenadiego · March 31, 2025, 9:28am

Hi @CamDavidsonPilon ,

Thanks for the help! I think I figured out what went wrong. I did not try your last suggestion, but I tried to update each worker again with pio update when the leader was connected to the internet. We started having problems after updating all the workers and leaders. We updated initially using the UI from the .zip files and when this started to give us problems, we used the command line (option 2 and 3 in your website). Initially, updating from the .zip files using the UI resulted in the leader updating and the workers not doing so. Also, the UI was not updating accordingly with the app. Using the last option you recommend in the instructions, we managed to have all workers and leaders (app and UI) with the same version. However, problems with visualization of calibration files and connection problems started.

Updating all the workers and leaders by connecting them to the internet and running the code you suggested here: sudo chown pioreactor:www-data /home/pioreactor/.pioreactor/storage sudo chmod g+srwx /home/pioreactor/.pioreactor/storage, seems to have solved our issues! I don’t understand what is different since the installed version is the same we downloaded as a .zip, but it did the trick!

Thanks for all the help!