Hi there,
I’ve been trying to update my cluster to 25.1.21 (started at 24.10.29 and updated successfully to 24.12.10 first) over the past few days and have run into some issues. On only two of my workers (out of 10 + the leader), the update failed (tried updating through the UI from the downloaded .zip and by pushing it to each Pioreactor and individually updating manually through ssh-ing in) and when checking the logs I get the following warning:
WARNING [read config] Not found in configuration: 'storage.persistent_cache'. Are you missing the following in your config?
[storage]
persistent_cache=some value
In the shared config.ini file, the filepath for persistent_cache is set to persistent_cache = /home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite and it isn’t overwritten in the config for either of the Pios that aren’t updating.
When I try to open that filepath for either of the two that are failing the update the document is empty in nano. However, in any of the others that successfully updated, the file is populated (with what looks like data that isn’t meant to be viewed in nano).
Thinking it might be related to the changed in this update fixing the typo I also checked /home/pioreactor/.pioreactor/storage/local_persistant_pioreactor_metadata.sqlite, but that file location is also blank in nano
Is there something I can manually add to this file to make these Pios update or do I just need to reflash them? I had done stirring, pump, and OD calibrations with each of these, so it seems wrong that the files are empty.
Here’s the full cache from one of the two in case there’s something else in it that helps diagnose the problem:
Requirement already satisfied: pyusb==1.2.1 in /usr/local/lib/python3.11/dist-packages (from pioreactor==25.1.21->pioreactor==25.1.21) (1.2.1)
Requirement already satisfied: rpi_hardware_pwm==0.2.1 in /usr/local/lib/python3.11/dist-packages (from pioreactor==25.1.21->pioreactor==25.1.21) (0.2.1)
Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.11/dist-packages (from adafruit-circuitpython-ads1x15==2.2.23->pioreactor==25.1.21->pioreactor==25.1.21) (4.12.2)
pioreactor is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.
2025-02-06T14:24:39-0500 [update_app] DEBUG sudo bash /tmp/release_25.1.21/update.sh
2025-02-06T14:24:45-0500 [read config] WARNING Not found in configuration: 'storage.persistent_cache'. Are you missing the following in your config?
[storage]
persistent_cache=some value
2025-02-06T14:24:46-0500 [update_app] DEBUG + export LC_ALL=C
+ LC_ALL=C
+++ dirname /tmp/release_25.1.21/update.sh
++ cd /tmp/release_25.1.21
++ pwd
+ SCRIPT_DIR=/tmp/release_25.1.21
++ crudini --get /home/pioreactor/.pioreactor/config.ini cluster.topology leader_hostname
+ LEADER_HOSTNAME=leader
+ STORAGE_DIR=/home/pioreactor/.pioreactor/storage
+ DB=/home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite
+ '[' '!' -f /home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite ']'
+ chown -R pioreactor:www-data /home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite /home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite-shm /home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite-wal
+ chmod -R 770 /home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite /home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite-shm /home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite-wal
+ sudo -u pioreactor mkdir -p /home/pioreactor/.pioreactor/storage/calibrations/od /home/pioreactor/.pioreactor/storage/calibrations/media_pump /home/pioreactor/.pioreactor/storage/calibrations/waste_pump /home/pioreactor/.pioreactor/storage/calibrations/alt_media_pump
+ sudo pip3 install /tmp/release_25.1.21/PyYAML-6.0.2-cp311-cp311-linux_armv7l.whl
+ cp /tmp/release_25.1.21/create_diskcache.sh /usr/local/bin/create_diskcache.sh
+ sudo bash /usr/local/bin/create_diskcache.sh
+ set -e
+ export LC_ALL=C
+ LC_ALL=C
+ DIR=/tmp/pioreactor_cache
+ mkdir -p /tmp/pioreactor_cache
+ chmod -R 770 /tmp/pioreactor_cache/
+ chown -R pioreactor:www-data /tmp/pioreactor_cache/
+ chmod g+s /tmp/pioreactor_cache
+ touch /tmp/pioreactor_cache/huey.db
+ touch /tmp/pioreactor_cache/huey.db-shm
+ touch /tmp/pioreactor_cache/huey.db-wal
+ touch /tmp/pioreactor_cache/local_intermittent_pioreactor_metadata.sqlite
+ touch /tmp/pioreactor_cache/local_intermittent_pioreactor_metadata.sqlite-shm
+ touch /tmp/pioreactor_cache/local_intermittent_pioreactor_metadata.sqlite-wal
+ chmod -R 770 /tmp/pioreactor_cache/
+ sudo -u pioreactor python /tmp/release_25.1.21/cal_convert.py /home/pioreactor/.pioreactor/storage/od_calibrations/cache.db
+ sudo -u pioreactor python /tmp/release_25.1.21/cal_convert.py /home/pioreactor/.pioreactor/storage/pump_calibrations/cache.db
+ chown -R pioreactor:www-data /home/pioreactor/.pioreactor/storage/calibrations/
+ sudo -u pioreactor python /tmp/release_25.1.21/cal_active.py /home/pioreactor/.pioreactor/storage/current_pump_calibration/cache.db
2025-02-06T14:24:45-0500 WARNING [read config] Not found in configuration: 'storage.persistent_cache'. Are you missing the following in your config?
[storage]
persistent_cache=some value
Traceback (most recent call last):
File "/usr/lib/python3.11/configparser.py", line 805, in get
value = d[option]
~^^^^^^^^
File "/usr/lib/python3.11/collections/__init__.py", line 1004, in __getitem__
return self.__missing__(key) # support subclasses that define __missing__
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/collections/__init__.py", line 996, in __missing__
raise KeyError(key)
KeyError: 'persistent_cache'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/release_25.1.21/cal_active.py", line 43, in <module>
main()
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/release_25.1.21/cal_active.py", line 28, in main
with local_persistent_storage("active_calibrations") as c:
File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/__init__.py", line 413, in local_persistent_storage
with cache(cache_name, db_path=config.get("storage", "persistent_cache")) as c:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pioreactor/config.py", line 92, in get
raise e
File "/usr/local/lib/python3.11/dist-packages/pioreactor/config.py", line 77, in get
return super().get(section, option, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/configparser.py", line 808, in get
raise NoOptionError(option, section)
configparser.NoOptionError: No option 'persistent_cache' in section: 'storage'
2025-02-06T14:24:46-0500 [update_app] ERROR Update failed. See logs.