Export to .netcdf?

Software
1

After doing lots of thinking/hacking around with mirroring sql etc. to the cloud I concluded that the best way to do things might be to export each experiment to a .netcdf file - these files can be self-describing and are individual, where the SQL database gets (obviously) bigger and bigger. My plan would be to write a plugin to export experiments to .netcdf and then (for our purposes) send to a cloud storage bucket. A SQL/bigquery database then has the metadata of the runs and the path to the .netcdf files and to view data from experiments would be SQL to find the file and then standard python/xarray to visualise/manipulate etc. Once backed up the experiment can then be wiped from the SQLlite db on the pi to save space.

While experiments are live one would still have to use the sqldb on the pi to see data, but a cron script to export every x time to .netcdf and upload would also be useful for backing up in case of errors (although I know the db is mirrored across pioreactors in a cluster).

Wondering if anybody else has had similar thoughts?

2

Hi @noahsprent , thanks for sharing how you’re thinking about this. We mounted an SSD to get more storage, and then have a Github Action to backup the database to s3 every few days. Ours will run out of capacity at some point in the future (which is where your solution is neater!) but sharing in case it’s a helpful approach for anyone else:

Hardware: Raspberry Pi SSD Kit for Raspberry Pi 5 - The Pi Hut

I followed these instructions: https://www.raspberrypi.com/documentation/accessories/m2-hat-plus.html#installation plus these to mount the storage rather than use it for booting: https://www.raspberrypi.com/documentation/computers/configuration.html#automatically-mount-a-storage-device

3

Thanks Vicky! I’m assuming that that HAT only works if your leader doesn’t have the pioreactor HAT attached as well?

4

Hiya, I added the right link to the hardware now! Yes, our cluster has 24 workers with some extra sensors - we decided to keep our leader as a leader only, responsible for control of the workers.

1 Like