Automating Backup and Restart Cycle

Feedback and suggestions
1

Hello, we are currently using the Pios as a shared resource in our group for long-term experiments. This works nicely, however this usually leads to there always being something running on the clusters, making it very difficult to regularly restart the cluster and to allow for the leader database backup to happen. As the experiments are by different people it is also not easily possible for one person to manually stop and start the whole cluster.

Would it be possible to implement a routine that can either back-up the database during running experiments, or to automate a cluster-wide restart with subsequent continuation of all the experiments? Maybe also not fully automated but in a semi-automated / supervised fashion so that it does not go unnoticed if something goes wrong during reboot.

Thanks already!

Kai

1 Like
2

Hi @Jackd4w,

You can force the backup to happen. Try

pio run backup_database --force

on the leader.

I think the database attempts a backup weekly. I can show you how to change this to include the use of --force as well.

3

@Jackd4w, you beat me to it!

We were discussing the same issue in my group, so thank you for the thread. I was wondering if:

  1. forced backup can potentially interfere an ongoing experiment in case of crash due to large data backup?
  2. pio docs contain steps on how to reach the backup_database for export?