Remove/add new pioreactors

Software
1

Hi everyone,

Since from the last 2-3 software updates, when I remove the Pioreactors form the cluster and try to add them again after flashing the SD cards. I am constantly getting denial because the ssh key of the removed Pioreactor remains saved in known_hosts file of the leader.

If I go the command line interface and remove the ssh key of the respective Pioreactor, then I can add the pioreactor after that.

Hence, I was wondering:

  1. Did anyone experience similar issue?
  2. Could this be a bug in the recent updates or more of a personal issue?

The respective error log that gets generated in this case should be similar to this:

´´´
e[32m2024-05-26T16:08:55+0200 INFO [add_pioreactor] Adding new pioreactor P02 to cluster.e[0m e[31m2024-05-26T16:11:41+0200 ERROR [add_pioreactor] + set -e + export LC_ALL=C + LC_ALL=C + SSHPASS=raspberry + PIO_VERSION=1.0 + PIO_MODEL=pioreactor_20ml + HOSTNAME=P02 + HOSTNAME_local=P02.local + USERNAME=pioreactor ++ hostname + LEADER_HOSTNAME=L01 + ssh-keygen -R P02.local + ssh-keygen -R P02 ++ getent hosts P02.local ++ cut '-d ’ -f1 + ssh-keygen -R 10.42.0.217 + N=120 + counter=0 + sshpass -p raspberry ssh pioreactor@P02.local ‘test -d /home/pioreactor/.pioreactor && echo ‘'‘exists’'’’
´´´´

EDIT:
I am using leader_only image on the leader and 24.5.22 software version in all the Pioreactors

2

Hi @sharknaro,

We haven’t seen this issue before. In the script to add a new worker, we execute:

# remove from known_hosts if already present
ssh-keygen -R "$HOSTNAME_local"          >/dev/null 2>&1
ssh-keygen -R "$HOSTNAME"                >/dev/null 2>&1
ssh-keygen -R "$(getent hosts "$HOSTNAME_local" | cut -d' ' -f1)"                 >/dev/null 2>&1

which should clear the key from known_hosts for the hostname, hostname.local, and ip. What command do you have execute before it works? Maybe we can add that command to our script.

3

Hi @CamDavidsonPilon,

The script you provided should work fine as it is what I have executed as well, so the steps I followed was:

  1. Inside the leader, remove pio_worker.local in the known_hosts file
    ssh-keygen -f "/home/pioreactor/.ssh/known_hosts" -R "P081.local"

  2. Clear the cache
    sudo systemctl restart ssh

Perhaps the way I removed the workers were not sufficient. I just clicked remove to the respective worker in the_inventory_. I do not know if I also had to run additional steps in the command line or config.ini

4

Perhaps the way I removed the workers were not sufficient. I just clicked remove to the respective worker in the_inventory_. I do not know if I also had to run additional steps in the command line or config.ini

Should be fine! That’s the way we expect you to do it.

Hm, I wonder if we should look into clearing the cache too, or specify the file…

We can certainly harden this up for you.

1 Like