Stirring malfunction at high temperatures

Hey,

I am using the pioreactors for growing thermophilic organisms (60 °C). It seems like I get problems with stirring at these temperatures. First I get this message:

Stirring RPM is 0 - attempting to restart it automatically. It may be a temporary stall, target RPM may be too low, insufficient power applied to fan, or not reading sensor correctly.

Then it stops being responsive altogether and I can’t start and stop it anymore in the online interface. It works fine when I start the experiment but (my theory is) this starts happening once everything starts heating up. Could this be the case or am I missing something? OD readings etc. still work normally.

Hm, it’s possible there’s some weird temperature dependent effect on either the magnets or the Hall sensor (which measures magnetic fields)…

Are you putting the Pioreactors in an incubator / warm chamber, using the onboard heating to achieve 60C, or an both?


Then it stops being responsive altogether and I can’t start and stop it anymore in the online interface.

That’s weird! This seems unrelated though. What version of the software are you using?


To skip over this issue completely (but not my favourite solution), you can turn off the RPM feedback and just apply constant power. You can turn off the feedback loop by setting use_rpm=0 in the [stirring] section of your configuration. Set the initial_duty_cycle to some appropriate value (20 to 40), as well.

The Pioreactors are placed in an incubator. All workers are updated to the current 24.6.10 version. All other commands work as they should but if I press start on the stirring it just loads for a while and then nothing happens.

This is what I see:

Update:
I realised this is only happening to the three pioreactors close to the fan (heat source). The ones that are a bit further away are behaving as they should. The temperature gradient is something for me to solve but it supports the theory that the heating does have something to do with it

hm, a 60C ambient might be pushing the limits of the RPi. The RPi chip gets warm due to operations, and if the internal chip temp exceeds 80C (which is likely if the ambient is 60C), it’ll throttle itself to limit operations.

What Pis are you using? Have you tried setting 50C ambient, and then using the onboarding heating to achieve the last 10C?

I am using the 1.0 Pioreactors with high temp plastic. Our reasoning for using external heating only was to minimize variation as we don’t know how consistent the heating pads are. But if the high temperature causes more problems we will switch to use lower ambient + PCB heating. In that case I would consider upgrading with the improved temperature sensor

It looks like it will be a fine balance. Lowering the incubator temperature and turning on PCB heating looked like it was working well for a bit. But when I just checked the setup I got this warning and all Pio’s were shut down:

Temperature of heating surface has exceeded 66.0℃ - currently 66.375℃. This is beyond our recommendations. Shutting down Raspberry Pi to prevent further problems. Take caution when touching the heating surface and wetware.

Temperature of the incubator is currently 48-50 °C.

To maybe further help the troubleshoot. Pio2 is currently the only one not stirring. When I reboot it and it kills all tasks, stirring stays on in the UI.

After restarting it gives a warning that the CPU is 77 °C

Let me get back to you soon. I have some ideas.


The error

Temperature of heating surface has exceeded 66.0℃ - currently 66.375℃. This is beyond our recommendations. Shutting down Raspberry Pi to prevent further problems.

can be fixed easily:

  1. SSH into your leader, and we’ll create a simple plugin:
nano ~/.pioreactor/plugins/high_temp_plugin.py

And paste in this Python code.

Ctrl-x to exit.

  1. Next, you can “copy” that plugin to all the workers with one command:
pios cp ~/.pioreactor/plugins/high_temp_plugin.py -y
  1. All your workers can now safely get above the temperature threshold. Note that this is only allowable because you have the high-temperature plastics for the 1.0 version.

  2. I recommend rebooting your workers, too:

pios reboot -y

That “on” stirring label is annoying, and I think it’s a silly cache issue. Can you SSH into the worker PioBCF002 and run stirring manually:

pio run stirring

and then cancel it with ctrl-C? That should fix the problem I think.

These fix the issues. Thank you for the quick help once again! It is very much appreciated :slight_smile: :grinning:

1 Like

Hey Cameron,
to come back to this discussion. The heating via PCB and setting incubator at 50 °C makes the whole setup a lot more stable. However, I have one or two pioreactors of which the stirring still crashes occasionally. With the same unresponsiveness in the UI. I can now easily fix it manually but it is annoying when it happens when I am not around. Do you have any idea whether there is any underlying software problem that could cause this?

My current guess is it could also just be a hardware problem because the magnets/fan are too close or too far from the stirrer bar. (We are using the crossbars you previously recommended at 800 rpm). But I don’t want to replace parts if it turns out there is something else causing the problem.

Thanks again!
Stijn

Hm, so are both the following things happening simultaneously?

  1. A warning is repeatedly occurring about the RPM not being read correctly (i.e. reporting 0 RPM)
  2. The [stop] button is unresponsive (even after a browser refresh)?

The former seems like a hardware, but the latter seems like a software. Of course, they could both be caused by hardware or software, too.

I am still not totally comfortable with 50C ambient temps on the Pis. On one of the workers that is crashing: what is the output of the following

cat /var/log/pioreactor.log | grep "CPU temperature"

2024-06-23T02:09:40+0200 [monitor] WARNING CPU temperature at 75 ℃.
2024-06-23T06:09:40+0200 [monitor] WARNING CPU temperature at 75 ℃.
2024-06-23T10:09:40+0200 [monitor] WARNING CPU temperature at 75 ℃.
2024-06-23T14:09:40+0200 [monitor] WARNING CPU temperature at 75 ℃.
2024-06-23T18:09:40+0200 [monitor] WARNING CPU temperature at 74 ℃.
2024-06-23T22:09:40+0200 [monitor] WARNING CPU temperature at 74 ℃.
2024-06-24T02:09:40+0200 [monitor] WARNING CPU temperature at 74 ℃.
2024-06-24T06:09:40+0200 [monitor] WARNING CPU temperature at 74 ℃.
2024-06-24T10:09:40+0200 [monitor] WARNING CPU temperature at 74 ℃.
2024-06-24T14:09:40+0200 [monitor] WARNING CPU temperature at 77 ℃.
2024-06-24T18:09:40+0200 [monitor] DEBUG CPU temperature at 45 ℃.
2024-06-24T22:09:40+0200 [monitor] DEBUG CPU temperature at 43 ℃.
2024-06-25T02:09:40+0200 [monitor] DEBUG CPU temperature at 42 ℃.
2024-06-25T06:09:40+0200 [monitor] DEBUG CPU temperature at 42 ℃.
2024-06-25T10:09:40+0200 [monitor] DEBUG CPU temperature at 43 ℃.
2024-06-25T14:09:40+0200 [monitor] DEBUG CPU temperature at 43 ℃.
2024-06-25T16:17:22+0200 [monitor] DEBUG CPU temperature at 47 ℃.
2024-06-25T21:02:37+0200 [monitor] DEBUG CPU temperature at 44 ℃.
2024-06-26T01:02:37+0200 [monitor] DEBUG CPU temperature at 43 ℃.
2024-06-26T05:02:37+0200 [monitor] DEBUG CPU temperature at 42 ℃.
2024-06-26T09:02:37+0200 [monitor] DEBUG CPU temperature at 43 ℃.

Also,

I just wanted to let you know that upon our hardware inspection we realised that the plastic plate above the cpu is warped. Looks like it melted a bit and deformed. I dont think it will cause much problems but it might be good to know that theres limits to the ‘thermostability’

So this is what we ideally want: temps less than 70C. Above that, the Pi may start to misbehave unpredictably. If possible, I would reduce the ambient temp further to see if this resolves the problem.


There’s two types of plastic with use: PLA, which is soft and will deform around 60C (its glass transition temp) if force is applied, and PCCF, which has a much much higher glass transition temp.

It looks like your top body is PCCF, but the bottom might not be? Either we sent you the wrong part, or it wasn’t replaced with the PCCF version. On the bottom of that piece of plastic should be the lettering pccf if is made from PCCF (and no letter == PLA). You can check.

Will it cause problems? Probably not, so long as the ambient temp is away from 60C. I imagine this deformation occurred when the chamber was at 60C.