New Pioreactor releases: 24.7.3 & 24.7.7

Software
1

24.7.7

Is an important bug fix release for 24.7.3. The bug is only present if you installed the 24.7.3 image, or upgrade to 24.7.3 from a release-archive.

24.7.3

Enhancements

  • A new live preview in the UI’s experiment profile editor. This preview tool is useful for getting immediate feedback when writing a profile. We’ll keep on adding to this to improve the edit-profile workflow - please send us feedback!
  • new when action type in experiment profiles that will execute an action (or list of actions) when some expression is true. For example, start a chemostat when a threshold OD is first achieved, log a message when event is triggered, or monitor a bioreactor parameter and execute an action if it goes out of bounds.
  • New config turbidostat.config that can be used to modify some internal turbidostat settings:
    [turbidostat.config]
signal_channel=2
od_smoothing_ema=0.5
  • Better user interaction on the Pioreactors page when the assigned experiment and “viewing” experiment are different.
  • Select / Deselect all Pioreactors to assign to an experiment faster.
  • Added unit() function to experiment profiles expressions that returns the unit name the expression is evaluated for. Ex: if: ${{ unit() == worker01 }}.
  • Added job_name() function to experiment profiles expressions that returns the job_name the expression is evaluated for. Ex: if: ${{ job_name() == stirring }}.
  • Added experiment() function to experiment profiles expressions that returns the experiment the expression is evaluated for. Ex: if: ${{ experiment() == exp001 }}.

Breaking changes

  • significant web backend API changes! See list of rules in docs.

Bug fixes

  • Fix UI code editor from being unresponsive when all the text was removed.
  • Experiment profiles won’t be overwritten if providing the same filename as an existing profile.
2

Hi there,
I have just installed software for my new set of pioreactors with this latest software version. Seemed there is a problem relating to the software as the seft test of leader pioreactor keeps failing. All configuration is correct and hardware is fine. Self test results of workers are fine.
The failed tests are all about photodiodes. The warning shows that “Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python interpreter from there”.

Could you please let me know how to solve this issue?

Best regards,
Sansanee

3

wowow that sounds not good. Let me check this right now. Sorry @Sansanee

Edit 1:

Ithink this only effected new installs, and updates via release-archives (zips), and not “updates over the internet”

Edit 2:

This is now fixed, version 24.7.7 doesn’t have this issue anymore. The getting-started docs link to the 24.7.7 image too.

4

For now, I’m going to pull the latest release, and fix-forward.

1 Like
5

@Sansanee, if you want, you can “downgrade” to a working version across your cluster

  1. SSH into your Pioreactors and running sudo pip uninstall numpy -y on each one.
  2. Downloading the release_24.6.10.zip from this release page, and following the instructions here: Updating the Pioreactor software | Pioreactor Docs

Later today we’ll produce a proper release that resolves this issue, and you can upgrade to that.

6

Thank you @CamDavidsonPilon
That sorted out the problem in leader.
However, there is another problem after I uninstall numpy in my workers.
I cannot do od calibration the error shows as below
Reading to start stirring? [Y/n]: y
Starting stirring and blocking until near target RPM.
Warming up OD…
2024-07-04T00:42:06+0100 ERROR [od_reading] module ‘numpy’ has no attribute ‘asarray’
Traceback (most recent call last):
File “/usr/local/bin/pio”, line 8, in
sys.exit(pio())
^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1666, in invoke
rv = super().invoke(ctx)
^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/click/decorators.py”, line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/actions/od_calibration.py”, line 734, in click_od_calibration
od_calibration(json_file)
File “/usr/local/lib/python3.11/dist-packages/pioreactor/actions/od_calibration.py”, line 534, in od_calibration
inferred_od600s, voltages = start_recording_and_diluting(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/actions/od_calibration.py”, line 235, in start_recording_and_diluting
with start_od_reading(
^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py”, line 1266, in start_od_reading
return ODReader(
^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py”, line 105, in call
obj = type.call(cls, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py”, line 917, in init
self.adc_reader.take_reading(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py”, line 523, in take_reading
raise e
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py”, line 466, in take_reading
self.most_appropriate_AC_hz = self.determine_most_appropriate_AC_hz(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py”, line 544, in determine_most_appropriate_AC_hz
argmin_freq1 = _compute_best_freq(timestamps[channel], aggregated_signals[channel])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py”, line 536, in _compute_best_freq
_, AIC = self._sin_regression_with_known_freq(timestamps, aggregated_signals, freq=freq)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py”, line 344, in sin_regression_with_known_freq
x = np.asarray(x)
^^^^^^^^^^
AttributeError: module ‘numpy’ has no attribute ‘asarray’

Could you please let me know how to sort this out?

Thank you, Sansanee

7

Ah, sorry @Sansanee, gimme few more hours to fix. Apologies for wasting your time here.

8

@Sansanee are your machines connected to the internet or not (i.e they might be on a local-access-point, and not connected to the internet)?

9

Nope, they were connected to local access point (leader).

10

Hm hm hm, okay, I suggest the following solutions:

  • reflash the workers with the new worker image (24.7.7), available now. You can “remove” the workers from your cluster, and reuse the same names.

or

  • Download the required .deb file from here on a computer, and we’ll move it to your leader. You can use scp to do this, or you can use filezilla. Here are some docs on moving files with scp or filezilla (replace the release-archive with this downloaded .deb file).

    Then, once it’s on your leader, distribute it to your cluster with:

    pios cp python3-numpy_1.24.2-1_armhf.deb -y

    Finally, SSH into each worker and run:

    sudo dpkg -i ./python3-numpy_1.24.2-1_armhf.deb

    I think that should fix it!

11

Thank you so much.
I’ll try this on Monday and let you know.

12

Rreflashing the workers did not help as I cannot add them into the cluster. There was warning as below;
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:0bP7aHEDMvmS9G5eB79Xb7Wf660doHoiVtaIQwr3TdI.
Please contact your system administrator.
Add correct host key in /home/pioreactor/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/pioreactor/.ssh/known_hosts:9
remove with:
ssh-keygen -f “/home/pioreactor/.ssh/known_hosts” -R “pio04.local”
Host key for pio04.local has changed and you have requested strict checking.
Host key verification failed.
++ date

  • echo ‘Connection to Pio04.local missed - Thu Jul 4 01:56:13 BST 2024’
  • sshpass -v -p raspberry ssh pioreactor@Pio04.local
  • grep ‘Wrong password’
  • counter=119
  • ‘[’ 119 -eq 120 ‘]’
  • sleep 1
  • sshpass -p raspberry ssh pioreactor@Pio04.local ‘test -d /home/pioreactor/.pioreactor && echo ‘'‘exists’'’’
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that a host key has just been changed.
    The fingerprint for the ED25519 key sent by the remote host is
    SHA256:0bP7aHEDMvmS9G5eB79Xb7Wf660doHoiVtaIQwr3TdI.
    Please contact your system administrator.
    Add correct host key in /home/pioreactor/.ssh/known_hosts to get rid of this message.
    Offending ED25519 key in /home/pioreactor/.ssh/known_hosts:9
    remove with:
    ssh-keygen -f “/home/pioreactor/.ssh/known_hosts” -R “pio04.local”
    Host key for pio04.local has changed and you have requested strict checking.
    Host key verification failed.
    ++ date
  • echo ‘Connection to Pio04.local missed - Thu Jul 4 01:56:14 BST 2024’
  • sshpass -v -p raspberry ssh pioreactor@Pio04.local
  • grep ‘Wrong password’
  • counter=120
  • ‘[’ 120 -eq 120 ‘]’
  • echo ‘Attempted to connect 120 times, but failed. Exiting.’
  • exit 1
13

hm, we should be resolving that before trying to add a worker. Can you try SSHing into your leader and running:

ssh-keygen -f "/home/pioreactor/.ssh/known_hosts" -R "pio04.local"

(does that return an error at all)

and trying to add again? (tip: you can use pio workers add pio04 on the leader’s command line to add a new worker named pio4).

14

By using both commands, I can add only 2 workers (pio02 and pio04) to the cluster, but I have 3, actually. There was nothing happenned when I used the command to add the third worker.
Also, the event log shows that pio02 and pio04 are succesfully added and ready. But there are not active bacause I cannot start self test or manage both via UI.

15

I’m thinking to reflash SD card for leader as well and will SSH into leader and running sudo pip uninstall numpy -y if the I got the same warning of " Error impporting numpy". (It’s kind of start over from the begining again)
How do you think?

16

sudo pip uninstall numpy -y

this command shouldn’t be necessary anymore. Be sure to use the latest leader-worker image provided in our docs site.

It’s always safe to start over! Just costs time :frowning:

But there are not active bacause I cannot start self test or manage both via UI.

I’m unsure why, yes, maybe a restart is best. Sorry for the trouble!

1 Like