New Pioreactor release: 26.11.0 and 26.11.1

Software
1

:wave: this is an important release for an upcoming 2.0 launch. It starts to give back some “authority” to workers (part of a longer trend to decouple leaders and workers). It also has some useful performance fixes, bug fixes, and UX improvements.

:down_arrow: Download release_26.11.0.zip

26.11.0

Breaking changes

  • Replaced pio workers update-model WORKER -m NAME -v VERSION with pio workers model set WORKER NAME VERSION. Added pio workers model get WORKER to print the worker’s model assignment as JSON.
  • Adding a worker now requires it to run the same software version as the leader; an older worker is rejected with a message to update it first.
  • Changing a Pioreactor’s model now requires the unit to be reachable and its activities to be stopped. Update workers before changing their models through the leader.
  • Bioreactor MQTT topics, pioreactor/<unit>/<experiment>/bioreactor/<variable>, now publish {"timestamp": ..., "value": ...} instead of a bare number. Experiment profile expressions that read these topics need .value, for example ${{ ::bioreactor:current_volume_ml.value }}.
  • Replaced the PWM MQTT topic pioreactor/<unit>/<experiment>/pwms/dc with pioreactor/<unit>/<experiment>/pwms/channel_dc, keyed by PWM channel instead of GPIO pin. Update any plugins or scripts that consume this topic. Upgrade the leader and all workers together, then reload browser tabs.

Enhancements

  • Added pio cluster leave to remove a worker from its cluster from the worker itself, for example before moving it to another lab. It keeps the worker’s model, calibrations, estimators and unit_config.ini.

  • Adding a worker is more reliable: a failed step can be retried, a wrong SSH password fails right away, and the UI no longer gives up after 90 seconds while setup is still running.

  • Added automatic imports from the SD card’s boot partition. Place files in these folders on the boot partition, then reboot:

    • pioreactor/plugins/: plugin .whl files (and any dependency wheels) are installed.
    • pioreactor/experiment_profiles/ (leaders only), pioreactor/models/, pioreactor/hardware/: .yaml files are imported.

    Successful imports are removed from the boot partition; failed ones are kept with a .log file explaining why.

  • Each Pioreactor now stores its own model, so it survives reboots and is available when the leader is unreachable. Custom models defined only on a worker can now be assigned to it. New CLI commands to inspect or change the local model:

    pio model get
pio model set pioreactor_20ml 1.5

  • Jobs now buffer up to 10,000 MQTT messages (was 100) while the broker is unreachable, so a leader reboot or short Wi-Fi drop loses far fewer readings.

  • Workers now log a warning at startup if the system clock isn’t synchronised, since measurement timestamps may be wrong.

  • Faster data exports and chart loading, and lower browser memory use for live charts and logs during long experiments.

  • Added Mount and Eject buttons for USB drives on the Plugins page, with clearer guidance when a drive is missing, unmounted, or has no plugins.

Bug fixes

  • Fixed pio workers add -p <password> writing the worker’s SSH password to the logs when setup failed.
  • Fixed commands from the leader possibly running on the wrong Pioreactor when an address is stale or reused.
  • Fixed dosing events being discarded when an addition would overfill the vial, which could stop dosing automations from pausing correctly.
  • Fixed jobs failing immediately when the leader’s web server is briefly unavailable, for example while restarting.
  • Automation events, bioreactor values and PWM duty cycles are now stored with the time they were produced on the worker, not the time the leader received them.
  • Fixed a re-added worker still pointing at its previous leader’s MQTT broker.
  • Fixed configuration editing being blocked when configuration history is slow or unavailable.
  • Fixed the Plugins page hiding plugins on read-only USB drives.
  • Fixed PWM duty-cycle history failing to save on a leader-only unit with no assigned model.
  • Fixed the monitor’s self-checks erroring when the leader is unreachable.
  • Fixed Akima calibration fitting with only two data points.
2

Known issues:

  • UI dialog boxes confirming too aggressively! hotfix incoming
3

Fixed a bug where cancelling or dismissing some confirmation dialogs could still execute the action, including stopping activities, rebooting or shutting down Pioreactors, and deleting items.

We recommend updating to 26.11.1. After updating, reload any open Pioreactor UI tabs.

:down_arrow: Download release_26.11.1.zip

26.11.1

Bug fixes

  • Fixed cancelling or dismissing confirmation dialogs still stopping activities, rebooting or shutting down Pioreactors, removing or unassigning workers, or deleting profiles, calibrations, estimators and camera snapshots. These actions now require explicit confirmation. Cancelling the leader system archive import also leaves the file picker closed.
4

26.11.0 update fails on armhf/Trixie workers: bundled Bookworm mosquitto packages need libssl3

I was a bit behind on updating but encountered an issue when trying to update to 26.11.0. Found a workaround with some troubleshooting and vibecoding. Maybe just a fault somewhere on my side but wanted to share either way :slight_smile:

Below is the (Claude) summary of what I did:

Environment

  • Cluster of ~30 workers plus 1 leader. All units were on 26.10.0 before updating.
  • Workers: armhf, Raspberry Pi OS Trixie (libc-bin 2.41-12+rpt1), 512 MB RAM. [model, e.g. Pi Zero 2 W / 3A+]
  • Leader: [model / arch]

Problem 1: mosquitto install fails (update step 7, 50_install_mosquitto.sh)

  • Mosquitto was not installed on the workers before this update. The dpkg log shows install mosquitto <none> → 2.0.11-1.2+deb12u2.
  • The script installs the bundled *_2.0.11-1.2+deb12u2_armhf.deb packages. These depend on libssl3 (>= 3.0.0), which doesn’t exist on Trixie (it’s libssl3t64 there).
  • Result: libmosquitto1, mosquitto and mosquitto-clients are left unpacked but not configured, apt is left in a broken state, and the update fails on every worker.
  • Side effect: the Python package was already upgraded (pioreactor-26.11.0.dist-info is timestamped at the failed run), so pio version reports 26.11.0 even though steps 7 onward never ran.

dpkg error:

dpkg: dependency problems prevent configuration of libmosquitto1:armhf:
 libmosquitto1:armhf depends on libssl3 (>= 3.0.0); however:
  Package libssl3 is not installed.

Workaround (tested on one worker, then rolled out)

bash

sudo dpkg --purge mosquitto-clients mosquitto libmosquitto1
sudo apt-get update && sudo apt-get install -y libmosquitto1 mosquitto mosquitto-clients   # Trixie 2.0.21-1
pio update app --version 26.11.0

50_install_mosquitto.sh then skips the bundled packages because the installed version is ≥ 2.0. All 13 steps complete, pio repair reports all services active, and chrony correctly uses the leader as its time source.

Suggested fix: detect the Debian codename and install from apt on Trixie, or bundle deb13 packages.

Can you confirm this is the intended workaround, and whether the mosquitto broker service should be enabled on workers? The Trixie package enables it by default.

Problem 2: update archives are never removed from /tmp on workers

  • Each update attempt leaves a ~25–27 MB /tmp/pioreactor_update_archive_*.zip behind. After several releases and retries, about 20 had built up on the leader.
  • On Trixie, /tmp is a RAM disk (237 MB on 512 MB units). It filled up completely, and later updates failed at the copy step with rsync: write failed ... No space left on device (28).
  • This also holds about half the RAM on these units.

Minor issues (possibly not caused by 26.11.0)

  • sudo: unable to resolve host PioXX appears on every sudo call: the workers’ /etc/hosts has no entry for their own hostname.
5

This is very helpful, thanks @Stkral !

I’ve updated the Release 26.11.0 · Pioreactor/pioreactor · GitHub release file with some suggested fixes.