Problems with specific pioreactor getting lost after 4 hours

Software
1

Hello,

I’ve been facing this issue with a specific pioreactor in my cluster (pio11) always getting lost after 3 hours and 58 minutes. This has happened twice, and its been in the exact same timepoint for some reason.

I’m unsure if this is a software or a hardware problem to be honest. I SSH’d in the pio and didn’t find anything interesting in the logs, only that they were updated again aftrer I turned the pioreactor off and on again. I’ve attached them below

2026-04-22T13:52:57+0100 [temperature_automation] DEBUG PID output = 8.654000000000037
2026-04-22T13:56:17+0100 [temperature_automation] DEBUG features={‘previous_heater_dc’: 100.0, ‘room_temp’: 22.0, ‘is_rpi_zero’: False, ‘volume’: 25.5, ‘time_series_of_temp’: [66.0625, 62.166666666666664, 59.34375, 56.96875, 54.979166666666664, 53.229166666666664, 51.697916666666664, 50.322916666666664, 49.125, 48.03125, 47.0625, 46.15625, 45.34375, 44.625, 43.9375, 43.322916666666664, 42.75, 42.25, 41.75, 41.3125, 40.875]}
2026-04-22T13:56:17+0100 [temperature_automation] DEBUG PID output = 10.339999999999968
2026-04-22T13:59:37+0100 [temperature_automation] DEBUG features={‘previous_heater_dc’: 100.0, ‘room_temp’: 22.0, ‘is_rpi_zero’: False, ‘volume’: 25.5, ‘time_series_of_temp’: [66.0625, 62.1875, 59.354166666666664, 56.989583333333336, 54.979166666666664, 53.229166666666664, 51.666666666666664, 50.302083333333336, 49.09375, 48.0, 47.020833333333336, 46.135416666666664, 45.333333333333336, 44.604166666666664, 43.927083333333336, 43.3125, 42.729166666666664, 42.21875, 41.739583333333336, 41.25, 40.864583333333336]}
2026-04-22T13:59:37+0100 [temperature_automation] DEBUG PID output = 10.960000000000035
2025-10-01T01:16:26+0100 [systemd] DEBUG huey.service successful
2025-10-01T01:16:27+0100 [pioreactor-pio11-api] DEBUG Starting pioreactor-pio11-api=26.4.2 on pio11…

Any ideas on what’s causing this? Thank you!

2

This is indeed strange. 3 hours and 58 minutes after an experiment started, or after the unit was powered on?

Anything different about that unit’s PSU?

When you say “Lost”, do you mean lost in the UI (a red dot) but still responsive, or totally lost + unresponsive + you are unable to SSH in until a power cycle?

3

Its after an experiment has started - my pios are always on, and I never noticed pio11 getting Lost when an experiment is not running

No differences than the other units - well, three of them (pio10, pio11, pio12) are on Raspberry Pi 5 and the rest are on Raspberry Pi 4, but that is the only hardware difference I can think of at the top of my head. I will check again and fully update you on this, but no other differences from what I recall

And yes, the red dot appears in the UI of pio11 informing me thats its lost, and that I need to manually power-cycle the unit

4

It sounds like a power issue tbh. During an experiment, the power load is more demanding, so it might unable to get enough power. A few things to look at:

Is the PSU different? Is the power cable length different? Is pio11 under additional power load (heavier stirring or temperature demands?)

5

Hi Cam! I checked yesterday and the power cable length are all the same for the pios. The pio11 conditions I am testing do have additional demands, but its replicates (pio03 and pio07) experienced no issues.

The only thing I can think of is that there could be insufficient power delivered by the power strip I am using; I changed the pio11 plug to another, standalone socket and will be re-testing today.

I will update you once the run is complete!

6

Update:

Seems that the problem was the cable. The pio died in the first three minutes this time, and again at 1h, so I changed the raspberry pi cable and I haven’t encountered any issues since!

1 Like