New Pioreactor release: 24.4.11

Software
1

:wave: It’s recommend to be on 24.4.3 (the previous version) before updating to this version!

24.4.11

Enhancements

  • Faster app start-up performance, which should translate to faster response times.
  • Log when workers change experiment assignments.
  • Log when workers change active status.
  • Adding [pioreactor] section to config.inis
  • improvements to calibration charts

Breaking changes

  • pio install-plugin is now pio plugins install. Likewise for uninstall.
  • pio list-plugins is now pio plugins list.
  • pios install-plugin is now pios plugins install. Likewise for uninstall.

Bug fixes

  • fixed Log table in the UI not showing all entries.
  • fixed HAT button response in the UI.

How to update?

1 Like
2

heyhey! i just updated to the next release and something is not working out with the UI, when i want to click on my pioreactors after creating an experiment, this is happening:

Something went wrong with the PioreactorUI!

Don’t worry. It’s our fault. Here’s what you can do:

Looks like there’s a bug in the UI. See the below for error information. We would appreciate it if you create an issue in Github for us, with the information below.

URL: http://pioreactor2.piogate.izi.fraunhofer.de:8080/pioreactors
i.map is not a function

if i want to SSH into the pioreactor and run for example stirring, it’s saying this:

pioreactor@pioreactor2:~ $ pio run stirring
Traceback (most recent call last):
File “/usr/local/lib/python3.11/dist-packages/pioreactor/whoami.py”, line 57, in _get_assigned_experiment_name
result.raise_for_status()
File “/usr/local/lib/python3.11/dist-packages/pioreactor/mureq.py”, line 232, in raise_for_status
raise HTTPErrorStatus(self.status_code)
pioreactor.mureq.HTTPErrorStatus: HTTP response returned error code 404

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/bin/pio”, line 8, in
sys.exit(pio())
^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/stirring.py”, line 509, in click_stirring
st = start_stirring(target_rpm=target_rpm, use_rpm=use_rpm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/stirring.py”, line 477, in start_stirring
experiment = experiment or get_assigned_experiment_name(unit)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/whoami.py”, line 37, in get_assigned_experiment_name
return _get_assigned_experiment_name(unit_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/whoami.py”, line 66, in _get_assigned_experiment_name
raise NotAssignedAnExperimentError(f"Worker {unit_name} is not assigned to an experiment")
pioreactor.exc.NotAssignedAnExperimentError: Worker pioreactor2 is not assigned to an experiment

pio version = 24.4.11

it seems like pio 2, which is my leader, is now not my leader anymore but i also don’t know how to change that. If i want to add any worker via the new “inventory” button to my cluster it doesn’t work, i guess it is because i have no leader assigned, the following error message occurs:

Unable to complete installation. The following error occurred: e[32m2024-04-12T14:22:06+0200 INFO [add_pioreactor] Adding new pioreactor pioreactor1 to cluster.e[0m e[31m2024-04-12T14:22:08+0200 ERROR [add_pioreactor] + set -e + export LC_ALL=C + LC_ALL=C + SSHPASS=raspberry + HOSTNAME=pioreactor1 + HOSTNAME_local=pioreactor1.local + USERNAME=pioreactor + ssh-keygen -R pioreactor1.local + ssh-keygen -R pioreactor1 ++ getent hosts pioreactor1.local ++ cut '-d ’ -f1 + ssh-keygen -R 10.131.125.66 + N=120 + counter=0 + sshpass -p raspberry ssh pioreactor@pioreactor1.local ‘test -d /home/pioreactor/.pioreactor && echo ‘'‘exists’'’’ + pio discover-workers -t + grep -q pioreactor1 Usage: pio [OPTIONS] COMMAND [ARGS]… Try ‘pio --help’ for help. Error: No such command ‘discover-workers’. + echo ‘Unable to confirm if pioreactor1 is a Pioreactor worker. Not found in ‘'‘pio discover-workers -t’'’. Did you install the worker image?’ + exit 1 e[0m Traceback (most recent call last): File “/usr/local/bin/pio”, line 8, in sys.exit(pio()) ^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1157, in call return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/pioreactor/cluster_management/init.py”, line 91, in add_worker raise BashScriptError(res.stderr) pioreactor.exc.BashScriptError: + set -e + export LC_ALL=C + LC_ALL=C + SSHPASS=raspberry + HOSTNAME=pioreactor1 + HOSTNAME_local=pioreactor1.local + USERNAME=pioreactor + ssh-keygen -R pioreactor1.local + ssh-keygen -R pioreactor1 ++ getent hosts pioreactor1.local ++ cut '-d ’ -f1 + ssh-keygen -R 10.131.125.66 + N=120 + counter=0 + sshpass -p raspberry ssh pioreactor@pioreactor1.local ‘test -d /home/pioreactor/.pioreactor && echo ‘'‘exists’'’’ + pio discover-workers -t + grep -q pioreactor1 Usage: pio [OPTIONS] COMMAND [ARGS]… Try ‘pio --help’ for help. Error: No such command ‘discover-workers’. + echo ‘Unable to confirm if pioreactor1 is a Pioreactor worker. Not found in ‘'‘pio discover-workers -t’'’. Did you install the worker image?’ + exit 1

any ideas? thanks!!!

3

Hi @LeoKleym,

Were you on version 24.4.3 prior to updating to 24.4.11? If not, you’ll need to manually run some update scripts. Copy&paste the these three files to your leader Pioreactor. And execute them like so:

sudo bash update.sh
sudo sqlite3 /home/pioreactor/.pioreactor/storage/pioreactor.sqlite < update.sql
sudo bash post_update.sh

I think that should be enough…

4

hmm this is what happens:

pioreactor@pioreactor2:~ $ sudo bash update.sh
sudo sqlite3 /home/pioreactor/.pioreactor/storage/pioreactor.sqlite < update.sql
sudo bash post_update.sh
bash: update.sh: No such file or directory
-bash: update.sql: No such file or directory
bash: post_update.sh: No such file or directory

and i just updated them now and updated to the release before first, then to the release now

5

Sorry I should have been more clear. Create new files on your leader, example:

nano update.sh

and copy&paste the update.sh contents from my link above into that file (ctrl-x to save and exit). Do that for update.sql and post_update.sh. Then try the commands I posted.

and i just updated them now and updated to the release before first, then to the release now

hm, so you were on 23.4.3 previously? Were there any errors during that update, or problems with the UI?

6

i can’t say if there were any problems, i just clicked the button in the UI saying “update to next release” when that update was done, i updated to the next one

7

is this what i am supposed to see ?

sudo sqlite3 /home/pioreactor/.pioreactor/storage/pioreactor.sqlite < update.sql
sudo bash post_update.sh

  • set -e
  • export LC_ALL=C
  • LC_ALL=C
    ++ cat
  • NEW_SCRIPT='#!/bin/bash

set -x
set -e

export LC_ALL=C

Get the first IPv4 address

IP=$(hostname -I | awk ‘'’{print $1}‘'’)

Check if the network interfaces exist and get their MAC addresses

if [ -d /sys/class/net/wlan0 ]; then
WLAN_MAC=$(cat /sys/class/net/wlan0/address)
else
WLAN_MAC=“Not available”
fi

if [ -d /sys/class/net/eth0 ]; then
ETH_MAC=$(cat /sys/class/net/eth0/address)
else
ETH_MAC=“Not available”
fi

Write the information to a file in key-value format

echo “HOSTNAME=$(hostname)” >> /boot/firmware/network_info.txt
echo “IP=$IP” > /boot/firmware/network_info.txt
echo “WLAN_MAC=$WLAN_MAC” >> /boot/firmware/network_info.txt
echo “ETH_MAC=$ETH_MAC” >> /boot/firmware/network_info.txt’

  • sudo echo '#!/bin/bash

set -x
set -e

export LC_ALL=C

Get the first IPv4 address

IP=$(hostname -I | awk ‘'’{print $1}‘'’)

Check if the network interfaces exist and get their MAC addresses

if [ -d /sys/class/net/wlan0 ]; then
WLAN_MAC=$(cat /sys/class/net/wlan0/address)
else
WLAN_MAC=“Not available”
fi

if [ -d /sys/class/net/eth0 ]; then
ETH_MAC=$(cat /sys/class/net/eth0/address)
else
ETH_MAC=“Not available”
fi

Write the information to a file in key-value format

echo “HOSTNAME=$(hostname)” >> /boot/firmware/network_info.txt
echo “IP=$IP” > /boot/firmware/network_info.txt
echo “WLAN_MAC=$WLAN_MAC” >> /boot/firmware/network_info.txt
echo “ETH_MAC=$ETH_MAC” >> /boot/firmware/network_info.txt’

  • sudo chmod +x /usr/local/bin/write_ip.sh
  • PIO_DIR=/home/pioreactor/.pioreactor
    ++ hostname
  • HOSTNAME=pioreactor2
    ++ crudini --get /home/pioreactor/.pioreactor/config.ini cluster.topology leader_address
  • LEADER_ADDRESS=pioreactor2.izi.fraunhofer.de
    update.sh: line 87: unexpected EOF while looking for matching `"’
    Parse error near line 21: duplicate column name: experiment
  • set -e
  • export LC_ALL=C
  • LC_ALL=C
  • PIO_DIR=/home/pioreactor/.pioreactor
    ++ hostname
  • HOSTNAME=pioreactor2
    ++ crudini --get /home/pioreactor/.pioreactor/config.ini cluster.topology leader_address
  • LEADER_ADDRESS=pioreactor2.izi.fraunhofer.de
    ++ crudini --get /home/pioreactor/.pioreactor/config.ini storage database
  • DB_FILE=/home/pioreactor/.pioreactor/storage/pioreactor.sqlite
  • ‘[’ pioreactor2.local = pioreactor2.izi.fraunhofer.de ‘]’
8

It didn’t change the problem that when i want to access the Pios via the UI this comes up:

Something went wrong with the PioreactorUI!

Don’t worry. It’s our fault. Here’s what you can do:

Looks like there’s a bug in the UI. See the below for error information. We would appreciate it if you create an issue in Github for us, with the information below.

URL: http://pioreactor2.piogate.izi.fraunhofer.de:8080/pioreactors
i.map is not a function

9

Try the following to refresh the webserver:

sudo systemctl restart lighttpd.service
sudo systemctl restart huey.service

And try the UI again.

EDIT: Actually, I see the error in your output. It’s a small issue with the leader_address (and we’ve fixed it in the future). Use this as your post_update.sh instead:

First delete the old one:

rm post_update.sh
nano post_update.sh

copy and paste the contents here: gist:6f8de25c8c141504531f9c9a2e5bc31c · GitHub

sudo bash post_update.sh

The error was because we were checking your leader_address, and we assumed it was of the form “leader.local”, which is wrong on our end. In your case, leader_address is pioreactor2.izi.fraunhofer.de

10

And now that I see the problem, you’ll also want to rerun this newer update.sh: update.sh · GitHub

11

this is the new output:

pioreactor@pioreactor2:~ $ sudo bash update.sh
sudo sqlite3 /home/pioreactor/.pioreactor/storage/pioreactor.sqlite < update.sql
sudo bash post_update.sh

  • set -e
  • export LC_ALL=C
  • LC_ALL=C
  • PIO_DIR=/home/pioreactor/.pioreactor
    ++ hostname
  • HOSTNAME=pioreactor2
    ++ crudini --get /home/pioreactor/.pioreactor/config.ini cluster.topology leader_hostname
  • LEADER_HOSTNAME=pioreactor2
  • ‘[’ pioreactor2 = pioreactor2 ‘]’
  • systemd_files=(“/lib/systemd/system/avahi_aliases.service” “/lib/systemd/system/load_rp2040.service”)
  • for file in “${systemd_files[@]}”
  • sudo crudini --ini-options=nospace --set /lib/systemd/system/avahi_aliases.service Service User pioreactor --set /lib/systemd/system/avahi_aliases.service Service EnvironmentFile /etc/environment
  • for file in “${systemd_files[@]}”
  • sudo crudini --ini-options=nospace --set /lib/systemd/system/load_rp2040.service Service User pioreactor --set /lib/systemd/system/load_rp2040.service Service EnvironmentFile /etc/environment
  • sudo systemctl daemon-reload
  • sudo systemctl restart avahi_aliases.service
  • sudo systemctl restart load_rp2040.service
  • FILE_PATH=/usr/local/bin/add_new_pioreactor_worker_from_leader.sh
  • sudo sed -i ‘s/pio discover-workers/pio workers discover/g’ /usr/local/bin/add_new_pioreactor_worker_from_leader.sh
    Parse error near line 21: duplicate column name: experiment
  • set -e
  • export LC_ALL=C
  • LC_ALL=C
  • PIO_DIR=/home/pioreactor/.pioreactor
    ++ hostname
  • HOSTNAME=pioreactor2
    ++ crudini --get /home/pioreactor/.pioreactor/config.ini cluster.topology leader_hostname
  • LEADER_HOSTNAME=pioreactor2
    ++ crudini --get /home/pioreactor/.pioreactor/config.ini storage database
  • DB_FILE=/home/pioreactor/.pioreactor/storage/pioreactor.sqlite
  • ‘[’ pioreactor2 = pioreactor2 ‘]’
  • crudini --get /home/pioreactor/.pioreactor/config.ini cluster.inventory
    ++ crudini --get /home/pioreactor/.pioreactor/config.ini cluster.inventory
  • units=‘pioreactor2
    pioreactor1
    pioreactor3
    pioreactor4
    pioreactor5’
  • IFS=
  • read -r unit
  • sqlite3 /home/pioreactor/.pioreactor/storage/pioreactor.sqlite ‘INSERT OR IGNORE INTO workers (pioreactor_unit, added_at, is_active) VALUES (’'‘pioreactor2’'‘, STRFTIME(’'‘%Y-%m-%dT%H:%M:%f000Z’'‘, ‘'‘NOW’'’), 1);’
  • IFS=
  • read -r unit
  • sqlite3 /home/pioreactor/.pioreactor/storage/pioreactor.sqlite ‘INSERT OR IGNORE INTO workers (pioreactor_unit, added_at, is_active) VALUES (’'‘pioreactor1’'‘, STRFTIME(’'‘%Y-%m-%dT%H:%M:%f000Z’'‘, ‘'‘NOW’'’), 1);’
  • IFS=
  • read -r unit
  • sqlite3 /home/pioreactor/.pioreactor/storage/pioreactor.sqlite ‘INSERT OR IGNORE INTO workers (pioreactor_unit, added_at, is_active) VALUES (’'‘pioreactor3’'‘, STRFTIME(’'‘%Y-%m-%dT%H:%M:%f000Z’'‘, ‘'‘NOW’'’), 1);’
  • IFS=
  • read -r unit
  • sqlite3 /home/pioreactor/.pioreactor/storage/pioreactor.sqlite ‘INSERT OR IGNORE INTO workers (pioreactor_unit, added_at, is_active) VALUES (’'‘pioreactor4’'‘, STRFTIME(’'‘%Y-%m-%dT%H:%M:%f000Z’'‘, ‘'‘NOW’'’), 1);’
  • IFS=
  • read -r unit
  • sqlite3 /home/pioreactor/.pioreactor/storage/pioreactor.sqlite ‘INSERT OR IGNORE INTO workers (pioreactor_unit, added_at, is_active) VALUES (’'‘pioreactor5’'‘, STRFTIME(’'‘%Y-%m-%dT%H:%M:%f000Z’'‘, ‘'‘NOW’'’), 1);’
  • IFS=
  • read -r unit
  • sqlite3 /home/pioreactor/.pioreactor/storage/pioreactor.sqlite ‘INSERT INTO experiment_worker_assignments (pioreactor_unit, experiment, assigned_at) SELECT pioreactor_unit, experiment, added_at FROM workers JOIN latest_experiment;’
    Error: stepping, UNIQUE constraint failed: experiment_worker_assignments.pioreactor_unit (19)
12

it worked :grimacing: :innocent: thanks a lot

1 Like
13

hehe yea I think you can ignore that last Error: stepping, UNIQUE constraint failed

Anyways, I’ve also gone back and fixed the previous update scripts for 24.4.3, which is where the problem started. Thanks for letting us know!