Blue light does not flash but ssh still connects?

xykb86 · February 19, 2025, 12:15pm

Hi Developers,
My pioreacrors have been running smoothly in the past 2 months. But today, I terminated a week-long experiment and tried to start a new one. All of a sudden, the system of one leader and two workers stopped working:
–The blue flash on each pioreactor did not flash at all even after rebooting. I pressed the buttons, no flash.
– I was able to open the UI but no previous data showed up. I could not create a new experiment. I could notr retrieve any log either.
– I could connect to the leader by ssh but when I tried to install air bubbler by cmd, the installation failed.

The version of pioreactor os was from last October.

Any suggetion?

Xiao

CamDavidsonPilon · February 19, 2025, 12:46pm

Hi @xykb86, rest assured your data is safe

It sounds like some start-up processes failed to start. Can you SSH into your leader Pioreactor and report back the output of

sudo systemctl status pioreactor_startup_run@monitor.service

xykb86 · February 19, 2025, 12:51pm

xykb86 · February 19, 2025, 1:00pm

“Pio run monitor” returns the follow:

pioreactor@leader:~ $ pio run monitor
Traceback (most recent call last):
File “/usr/lib/python3.11/configparser.py”, line 805, in get
value = d[option]
~^^^^^^^^
File “/usr/lib/python3.11/collections/init.py”, line 1004, in getitem
return self.missing(key) # support subclasses that define missing
^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/collections/init.py”, line 996, in missing
raise KeyError(key)
KeyError: ‘temporary_cache’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/pioreactor/config.py”, line 77, in get
return super().get(section, option, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/configparser.py”, line 808, in get
raise NoOptionError(option, section)
configparser.NoOptionError: No option ‘temporary_cache’ in section: ‘storage’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/bin/pio”, line 8, in
sys.exit(pio())
^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/monitor.py”, line 621, in click_monitor
job = Monitor(unit=whoami.get_unit_name(), experiment=whoami.UNIVERSAL_EXPERIMENT)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py”, line 105, in call
obj = type.call(cls, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/monitor.py”, line 102, in init
super().init(unit=unit, experiment=experiment)
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py”, line 941, in init
super().init(unit, experiment, source=“app”)
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py”, line 282, in init
self._check_for_duplicate_activity()
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py”, line 917, in _check_for_duplicate_activity
if is_pio_job_running(self.job_name) and not is_testing_env():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 446, in is_pio_job_running
with JobManager() as jm:
^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/utils/init.py”, line 613, in init
db_path = config.get(“storage”, “temporary_cache”)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/config.py”, line 84, in get
create_logger(“read config”).warning(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/logging.py”, line 178, in create_logger
experiment = get_assigned_experiment_name(unit)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/whoami.py”, line 35, in get_assigned_experiment_name
return _get_assigned_experiment_name(unit_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/whoami.py”, line 73, in _get_assigned_experiment_name
raise ConnectionError(
ConnectionError: Not able to access experiments in UI. Check http://127.0.0.1/api/experiments.

CamDavidsonPilon · February 19, 2025, 1:05pm

Did you update recently? (and if so, how?) It looks like you need to add to your configuration.

Using the UI, add the following to the [storage] section in your configuration:

[storage]
database=/home/pioreactor/.pioreactor/storage/pioreactor.sqlite
temporary_cache=/tmp/pioreactor_cache/local_intermittent_pioreactor_metadata.sqlite
persistent_cache=/home/pioreactor/.pioreactor/storage/local_persistent_pioreactor_metadata.sqlite

and then reboot

xykb86 · February 19, 2025, 1:08pm

Yes, today, after terminating the experiment, I downloaded 25.2.11 and tried to install on the UI. But an error message said I need to install 25.1.21 first.

Then the system stopped as descriped above.

xykb86 · February 19, 2025, 1:13pm

Now, it seems that I cannot access the full config.ini. I can only see shared config.ini:

xykb86 · February 19, 2025, 1:20pm

Is it possible to add the [storage] info to config.ini through ssd/cmd or write to microSD directly?

CamDavidsonPilon · February 19, 2025, 1:40pm

You should have been prompted to actually go back another version, and install version 24.12.10 first.

That’s okay. We can fix this. Let’s first install 24.12.10. Your webserver isn’t working, so we’ll do it manually.

Download release_24.12.10.zip.
We’ll have to scp it to your leader. From your computer, run
```
scp /path/to/release_24.12.10.zip pioreactor@leader.local:/tmp
```
(but fill in the path part)

SSH into your leader, and try:

pio update --source /tmp/release_24.12.10.zip

Does this succeed? You may need to reboot. If it succeeds, try the same thing for the workers.
Now, you can update to 25.1.21, then 25.2.11 (probably via the UI).

xykb86 · February 19, 2025, 2:30pm

Thanks! The instruction worked for the leader, not the workers.
For the workers, I tried

starting from step 2, changing @leader.local to @wk1.local. It returns:“ssh: Could not resolve hostname wk1.local: \262\273\326\252\265\300\325\342\321\371\265\304\326\367\273\372\241\243
lost connection”
ssh to leader, then to wk1, put in the commands from step 3. It returns what the picture shows.

image969×170 85.8 KB

It seems that the workers as nested inside the leader have hard time finding the “tmp” folder.

BTW, the workers are hooked to the leader through local access point.

CamDavidsonPilon · February 19, 2025, 2:53pm

Okay so try this:

From the leader, this will copy the file from the leader to the workers

pios cp /tmp/release_24.12.10.zip -y

Then try pio update... on the workers.

Note, you may have to scp the release_24.12.10.zip onto the leader again if you rebooted.

xykb86 · February 19, 2025, 3:38pm

Great! It worked. My problem is solved! Thanks a lot!

xykb86 · February 19, 2025, 4:14pm

Sorry to bother you again. But I cannot install air bubbler any more.
I tried both the command of “pio plugins install pioreactor_air_bubbler” and install across the cluster at the UI. Neither worked.

Any suggestion? Should I download the file locally and install from there?

CamDavidsonPilon · February 19, 2025, 4:28pm

What does the error say when you run pio plugins install pioreactor_air_bubbler?

Does your cluster have access to the internet?

If not:

You can download the .whl file locally,
do the scp to get the file onto the leader,
do a pios cp /path/to/file.whl -y to distribute the file to the workers,
then pios plugins install pioreactor_air_bubbler --source /path/to/file.whl to install on the leader + workers.

xykb86 · February 19, 2025, 4:37pm

pioreactor@leader:~ $ pio plugins install pioreactor_air_bubbler
2025-02-20T00:34:24+0800 DEBUG [install_plugin] Installing plugin pioreactor_air_bubbler.
2025-02-20T00:34:24+0800 DEBUG [install_plugin] bash /usr/local/bin/install_pioreactor_plugin.sh pioreactor_air_bubbler
2025-02-20T00:35:00+0800 ERROR [install_plugin] Failed to install plugin pioreactor_air_bubbler. See logs.
2025-02-20T00:35:00+0800 DEBUG [install_plugin] b’Looking in indexes: Simple index, https://www.piwheels.org/simple\n’
2025-02-20T00:35:00+0800 DEBUG [install_plugin] b’+ export LC_ALL=C\n+ LC_ALL=C\n+ plugin_name=pioreactor_air_bubbler\n+ source=\n+ clean_plugin_name=pioreactor_air_bubbler\n+ clean_plugin_name_with_dashes=pioreactor-air-bubbler\n+ clean_plugin_name_with_underscores=pioreactor_air_bubbler\n++ python3 -c 'import site; print(site.getsitepackages()[0])'\n+ install_folder=/usr/local/lib/python3.11/dist-packages/pioreactor_air_bubbler\n++ crudini --get /home/pioreactor/.pioreactor/config.ini cluster.topology leader_hostname\n+ leader_hostname=leader\n++ hostname\n+ '[' leader = leader ']'\n+ am_i_leader=true\n+ '[' -n '' ']'\n+ download_and_check_if_leader_only pioreactor-air-bubbler\n+ local PACKAGE_NAME=pioreactor-air-bubbler\n+ local CLEAN_PACKAGE_NAME=pioreactor_air_bubbler\n+ pip download -qq --no-deps --dest /tmp pioreactor-air-bubbler\nWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x75786290>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x75786690>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x75786b50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x75786f70>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7578a3d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7578ad30>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7578f0f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7578f510>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7578f930>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7578fd50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nERROR: Could not find a version that satisfies the requirement pioreactor-air-bubbler (from versions: none)\nERROR: No matching distribution found for pioreactor-air-bubbler\n+ local WHL_FILE\n++ ls '/tmp/pioreactor_air_bubbler*.whl'\nls: cannot access '/tmp/pioreactor_air_bubbler*.whl': No such file or directory\n+ WHL_FILE=\n+ local TEMPDIR\n++ mktemp -d\n+ TEMPDIR=/tmp/tmp.rHzaTmsaB6\n+ unzip -d /tmp/tmp.rHzaTmsaB6\nUnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.\n\nUsage: unzip [-Z] [-opts[modifiers]] file[.zip] [list] [-x xlist] [-d exdir]\n Default action is to extract files in list, except those in xlist, to exdir;\n file[.zip] may be a wildcard. -Z => ZipInfo mode (“unzip -Z” for usage).\n\n -p extract files to pipe, no messages -l list files (short format)\n -f freshen existing files, create none -t test compressed archive data\n -u update files, create if necessary -z display archive comment only\n -v list verbosely/show version info -T timestamp archive to latest\n -x exclude files that follow (in xlist) -d extract files into exdir\nmodifiers:\n -n never overwrite existing files -q quiet mode (-qq => quieter)\n -o overwrite files WITHOUT prompting -a auto-convert any text files\n -j junk paths (do not make directories) -aa treat ALL files as text\n -U use escapes for all non-ASCII Unicode -UU ignore any Unicode fields\n -C match filenames case-insensitively -L make (some) names lowercase\n -X restore UID/GID info -V retain VMS version numbers\n -K keep setuid/setgid/tacky permissions -M pipe through “more” pager\nSee “unzip -hh” or unzip.txt for more help. Examples:\n unzip data1 -x joe => extract all files except joe from zipfile data1.zip\n unzip -p foo | more => send contents of foo.zip via pipe into program more\n unzip -fo foo ReadMe => quietly replace existing ReadMe if archive file newer\n+ '[' -f /tmp/tmp.rHzaTmsaB6/LEADER_ONLY ']'\n+ rm -rf /tmp/tmp.rHzaTmsaB6\n+ rm '/tmp/pioreactor_air_bubbler*.whl'\nrm: cannot remove '/tmp/pioreactor_air_bubbler*.whl': No such file or directory\n+ return 1\n+ sudo pip3 install --upgrade --force-reinstall --ignore-installed pioreactor-air-bubbler\nWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x757afef0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x757b23b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x757b2850>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x757b2c50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x757b80b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x757b8a10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x757b8db0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x757bf210>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x757bf590>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nWARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x757bf9d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pioreactor-air-bubbler/\nERROR: Could not find a version that satisfies the requirement pioreactor-air-bubbler (from versions: none)\nERROR: No matching distribution found for pioreactor-air-bubbler\n’
Traceback (most recent call last):
File “/usr/local/bin/pio”, line 8, in
sys.exit(pio())
^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/plugin_management/install_plugin.py”, line 45, in click_install_plugin
install_plugin(name_of_plugin, source)
File “/usr/local/lib/python3.11/dist-packages/pioreactor/plugin_management/install_plugin.py”, line 34, in install_plugin
raise BashScriptError(f"Failed to install plugin {name_of_plugin}. See logs.")
pioreactor.exc.BashScriptError: Failed to install plugin pioreactor_air_bubbler. See logs.

xykb86 · February 19, 2025, 4:46pm

event 1 resulted from install across the cluster but I did not see the setting options from the Pioreactor panel.

event 2 resulted from running the command line instruction.

CamDavidsonPilon · February 19, 2025, 5:50pm

So you don’t have internet access (that’s okay!) - did you try my scp instructions?

xykb86 · February 19, 2025, 6:35pm

I believe it is a firewall issue (I am from China). I solved it by using VPN. Thanks a lot for your help!