Unable to save config changes to worker2

Using the Pioreactor Troubleshooting
1

Hi!

I have a pioreactor cluster with one of them working as leader-worker and two “normal” workers.

I have experiencing with no problems during the self-test and other previous steps, but when I try to save the changes in the shared config,ini or the “confing_worker2.ini”, the system returns the following message (see attached files):

“syncs_configs encountered a warning in leader
Encountered error syncing config to worker2”

shared
shared1920×1080 137 KB

I have tried to reboot the worker 2, reboot all the pioreactors and even reboot the router without success.

I was able to change the timestamp of all the pio’s trough SSH, so I don’t think it is a problem related with this protocol. However, the log file point to that:

Column 1 Column 2 Column 3 Column 4 E
2024-07-16 18:40:37 $experiment leader 2024-07-16T16:40:37.672766Z Syncing configs on leader…
2024-07-16 18:40:37 $experiment leader 2024-07-16T16:40:37.674861Z Syncing configs on worker1…
2024-07-16 18:40:37 $experiment leader 2024-07-16T16:40:37.929436Z Syncing configs on worker2…
2024-07-16 18:40:48 $experiment leader 2024-07-16T16:40:48.435474Z Encountered error syncing configs to worker2.
2024-07-16 18:40:48 $experiment leader 2024-07-16T16:40:48.446021Z RAN: /usr/bin/rsync -z --timeout 30 --inplace -e ssh /home/pioreactor/.pioreactor/config.ini worker2.local:/home/pioreactor/.pioreactor/config.ini STDOUT: STDERR: Permission denied, please try again. Permission denied, please try again. pioreactor@worker2.local: Permission denied (publickey,password). rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(231) [sender=3.2.7] Traceback (most recent call last): File /usr/local/lib/python3.11/dist-packages/pioreactor/cli/pios.py, line 430, in _thread_function sync_config_files(unit, shared, specific) File /usr/local/lib/python3.11/dist-packages/pioreactor/cli/pios.py, line 113, in sync_config_files cp_file_across_cluster(unit, localpath, remotepath, timeout=30) File /usr/local/lib/python3.11/dist-packages/pioreactor/utils/networking.py, line 14, in cp_file_across_cluster rsync( File /usr/local/lib/python3.11/dist-packages/sh.py, line 1508, in call rc = self.class.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3.11/dist-packages/sh.py, line 737, in init self.wait() File /usr/local/lib/python3.11/dist-packages/sh.py, line 799, in wait self.handle_command_exit_code(exit_code) File /usr/local/lib/python3.11/dist-packages/sh.py, line 826, in handle_command_exit_code raise exc sh.ErrorReturnCode_255: RAN: /usr/bin/rsync -z --timeout 30 --inplace -e ssh /home/pioreactor/.pioreactor/config.ini worker2.local:/home/pioreactor/.pioreactor/config.ini STDOUT: STDERR: Permission denied, please try again. Permission denied, please try again. pioreactor@worker2.local: Permission denied (publickey,password). rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(231) [sender=3.2.7]

How can I solve this problem?

2

Hi @Pedre91,

Hm, looks like there’s a permission issue in the logs you posted:

pioreactor@worker2.local: Permission denied (publickey,password).

Hm, did you use the default password raspberry when setting it up (I’m guessing yes), or did you change a password after?

Can you try SSHing into your leader machine and running the following:

ssh pioreactor@worker2.local

Does that fail / succeed? Does it ask for a password, and if so, does the default password work?

3

Hi Cameron!

Yes, I use the default pass (raspberry) and I didn’t change it.

Yes, I can SSHing to worker2 trough leader machine. The system ask for password, and yes, it is the default one.

Should I change the pass of each pioreactor?

Can I do it via SSH without reinstall the whole image in the SD card?

Thanks!

4

Hm, I’ve seen this problem before (I think).

Normally, when you “add” a worker to your cluster, there’s a song-and-dance between the leader and worker to establish password-less logins.

While on your command line, can you try to re-add the worker2 (it’s a safe operation to re-add a worker over again)? So try:

pio workers add worker2

Any errors / warnings? After it completes, you should be able to SSH into it without the password.

(I believe that the password being prompted is the source of the original problem above)

5

Hi Cameron.

Sorry for the later reply, I take some days off.

The problem seems solved following your steps. Now, the system don’t return an error message when saving changes in config.ini

This is the result on the console after following your instructions:

laptop@laptop-HP:~$ ssh pioreactor@leader.local
pioreactor@leader.local's password: 
Linux leader 6.6.20+rpt-rpi-v7 #1 SMP Raspbian 1:6.6.20-1+rpt1 (2024-03-07) armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jul 16 23:46:08 2024 from 192.168.1.221
pioreactor@leader:~ $ pio workers add worker2
2024-07-16T23:49:51+0200 INFO   [add_pioreactor] Adding new pioreactor worker2 to cluster.
2024-07-16T23:49:59+0200 NOTICE [add_pioreactor] New pioreactor worker2 successfully added to cluster.
pioreactor@leader:~ $ ssh pioreactor@worker2.local
Linux worker2 6.6.20+rpt-rpi-v7 #1 SMP Raspbian 1:6.6.20-1+rpt1 (2024-03-07) armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jul 16 23:45:06 2024 from 192.168.1.221
6

Okay, this looks good to me then!

I’m not sure the root cause yet, but I’ll keep this as a data point for future issues.