Stirring malfunction at high temperatures

Using the Pioreactor
1

Hey,

I am using the pioreactors for growing thermophilic organisms (60 °C). It seems like I get problems with stirring at these temperatures. First I get this message:

Stirring RPM is 0 - attempting to restart it automatically. It may be a temporary stall, target RPM may be too low, insufficient power applied to fan, or not reading sensor correctly.

Then it stops being responsive altogether and I can’t start and stop it anymore in the online interface. It works fine when I start the experiment but (my theory is) this starts happening once everything starts heating up. Could this be the case or am I missing something? OD readings etc. still work normally.

2

Hm, it’s possible there’s some weird temperature dependent effect on either the magnets or the Hall sensor (which measures magnetic fields)…

Are you putting the Pioreactors in an incubator / warm chamber, using the onboard heating to achieve 60C, or an both?

Then it stops being responsive altogether and I can’t start and stop it anymore in the online interface.

That’s weird! This seems unrelated though. What version of the software are you using?

To skip over this issue completely (but not my favourite solution), you can turn off the RPM feedback and just apply constant power. You can turn off the feedback loop by setting use_rpm=0 in the [stirring] section of your configuration. Set the initial_duty_cycle to some appropriate value (20 to 40), as well.

3

The Pioreactors are placed in an incubator. All workers are updated to the current 24.6.10 version. All other commands work as they should but if I press start on the stirring it just loads for a while and then nothing happens.

This is what I see:

image
image881×200 10.2 KB

4

Update:
I realised this is only happening to the three pioreactors close to the fan (heat source). The ones that are a bit further away are behaving as they should. The temperature gradient is something for me to solve but it supports the theory that the heating does have something to do with it

5

hm, a 60C ambient might be pushing the limits of the RPi. The RPi chip gets warm due to operations, and if the internal chip temp exceeds 80C (which is likely if the ambient is 60C), it’ll throttle itself to limit operations.

What Pis are you using? Have you tried setting 50C ambient, and then using the onboarding heating to achieve the last 10C?

6

I am using the 1.0 Pioreactors with high temp plastic. Our reasoning for using external heating only was to minimize variation as we don’t know how consistent the heating pads are. But if the high temperature causes more problems we will switch to use lower ambient + PCB heating. In that case I would consider upgrading with the improved temperature sensor

7

It looks like it will be a fine balance. Lowering the incubator temperature and turning on PCB heating looked like it was working well for a bit. But when I just checked the setup I got this warning and all Pio’s were shut down:

Temperature of heating surface has exceeded 66.0℃ - currently 66.375℃. This is beyond our recommendations. Shutting down Raspberry Pi to prevent further problems. Take caution when touching the heating surface and wetware.

Temperature of the incubator is currently 48-50 °C.

8

To maybe further help the troubleshoot. Pio2 is currently the only one not stirring. When I reboot it and it kills all tasks, stirring stays on in the UI.

image
image1141×265 14.5 KB

After restarting it gives a warning that the CPU is 77 °C

9

Let me get back to you soon. I have some ideas.

The error

Temperature of heating surface has exceeded 66.0℃ - currently 66.375℃. This is beyond our recommendations. Shutting down Raspberry Pi to prevent further problems.

can be fixed easily:

  1. SSH into your leader, and we’ll create a simple plugin:
nano ~/.pioreactor/plugins/high_temp_plugin.py

And paste in this Python code.

Ctrl-x to exit.

  1. Next, you can “copy” that plugin to all the workers with one command:
pios cp ~/.pioreactor/plugins/high_temp_plugin.py -y

  1. All your workers can now safely get above the temperature threshold. Note that this is only allowable because you have the high-temperature plastics for the 1.0 version.

  2. I recommend rebooting your workers, too:

pios reboot -y

That “on” stirring label is annoying, and I think it’s a silly cache issue. Can you SSH into the worker PioBCF002 and run stirring manually:

pio run stirring

and then cancel it with ctrl-C? That should fix the problem I think.

10

These fix the issues. Thank you for the quick help once again! It is very much appreciated :slight_smile: :grinning:

1 Like
11

Hey Cameron,
to come back to this discussion. The heating via PCB and setting incubator at 50 °C makes the whole setup a lot more stable. However, I have one or two pioreactors of which the stirring still crashes occasionally. With the same unresponsiveness in the UI. I can now easily fix it manually but it is annoying when it happens when I am not around. Do you have any idea whether there is any underlying software problem that could cause this?

My current guess is it could also just be a hardware problem because the magnets/fan are too close or too far from the stirrer bar. (We are using the crossbars you previously recommended at 800 rpm). But I don’t want to replace parts if it turns out there is something else causing the problem.

Thanks again!
Stijn

12

Hm, so are both the following things happening simultaneously?

  1. A warning is repeatedly occurring about the RPM not being read correctly (i.e. reporting 0 RPM)
  2. The [stop] button is unresponsive (even after a browser refresh)?

The former seems like a hardware, but the latter seems like a software. Of course, they could both be caused by hardware or software, too.

I am still not totally comfortable with 50C ambient temps on the Pis. On one of the workers that is crashing: what is the output of the following

cat /var/log/pioreactor.log | grep "CPU temperature"
13

2024-06-23T02:09:40+0200 [monitor] WARNING CPU temperature at 75 ℃.
2024-06-23T06:09:40+0200 [monitor] WARNING CPU temperature at 75 ℃.
2024-06-23T10:09:40+0200 [monitor] WARNING CPU temperature at 75 ℃.
2024-06-23T14:09:40+0200 [monitor] WARNING CPU temperature at 75 ℃.
2024-06-23T18:09:40+0200 [monitor] WARNING CPU temperature at 74 ℃.
2024-06-23T22:09:40+0200 [monitor] WARNING CPU temperature at 74 ℃.
2024-06-24T02:09:40+0200 [monitor] WARNING CPU temperature at 74 ℃.
2024-06-24T06:09:40+0200 [monitor] WARNING CPU temperature at 74 ℃.
2024-06-24T10:09:40+0200 [monitor] WARNING CPU temperature at 74 ℃.
2024-06-24T14:09:40+0200 [monitor] WARNING CPU temperature at 77 ℃.
2024-06-24T18:09:40+0200 [monitor] DEBUG CPU temperature at 45 ℃.
2024-06-24T22:09:40+0200 [monitor] DEBUG CPU temperature at 43 ℃.
2024-06-25T02:09:40+0200 [monitor] DEBUG CPU temperature at 42 ℃.
2024-06-25T06:09:40+0200 [monitor] DEBUG CPU temperature at 42 ℃.
2024-06-25T10:09:40+0200 [monitor] DEBUG CPU temperature at 43 ℃.
2024-06-25T14:09:40+0200 [monitor] DEBUG CPU temperature at 43 ℃.
2024-06-25T16:17:22+0200 [monitor] DEBUG CPU temperature at 47 ℃.
2024-06-25T21:02:37+0200 [monitor] DEBUG CPU temperature at 44 ℃.
2024-06-26T01:02:37+0200 [monitor] DEBUG CPU temperature at 43 ℃.
2024-06-26T05:02:37+0200 [monitor] DEBUG CPU temperature at 42 ℃.
2024-06-26T09:02:37+0200 [monitor] DEBUG CPU temperature at 43 ℃.

14

Also,

I just wanted to let you know that upon our hardware inspection we realised that the plastic plate above the cpu is warped. Looks like it melted a bit and deformed. I dont think it will cause much problems but it might be good to know that theres limits to the ‘thermostability’

20240626_112621
20240626_1126211920×2560 449 KB

15

So this is what we ideally want: temps less than 70C. Above that, the Pi may start to misbehave unpredictably. If possible, I would reduce the ambient temp further to see if this resolves the problem.

There’s two types of plastic with use: PLA, which is soft and will deform around 60C (its glass transition temp) if force is applied, and PCCF, which has a much much higher glass transition temp.

It looks like your top body is PCCF, but the bottom might not be? Either we sent you the wrong part, or it wasn’t replaced with the PCCF version. On the bottom of that piece of plastic should be the lettering pccf if is made from PCCF (and no letter == PLA). You can check.

Will it cause problems? Probably not, so long as the ambient temp is away from 60C. I imagine this deformation occurred when the chamber was at 60C.

16

Hey Cameron,

I am back from my holiday. After starting an experiment I noticed that the Pioreactors did not manage to heat above somewhere around 52 °C, despite setting them to 60 °C (Ambient is again 45). The only thing I changed is performing all the updates. When I tried increasing the ambient temperature, all Pios turned themselves off because of a PCB heating warning. I thought we fixed this by installing the high temp upgrade as you described here. I tried re-installing the high temp plugin but that did not have an effect. I am not sure where to start troubleshooting with this… Do you have an idea?

17

Ah, we did change some internal code in the recent update! It was a big change to automations, and we require users to upgrade plugins as a result.

You’ll have to delete the old plugin and add a new one. It’s easy:

  1. SSH into your leader, and we’ll remove the existing plugin with:

    rm ~/.pioreactor/plugins/high_temp_plugin.py

  2. Then enter:

    nano ~/.pioreactor/plugins/high_temp_plugin.py

    and copy this Python code into the file.

  3. Send this plugin to your workers:

    pios cp ~/.pioreactor/plugins/high_temp_plugin.py -y

    This will overwrite the existing plugin on the workers.

18

Hey Cameron,

that does seem to resolve the issue of the heating!

Also, I noticed that after turning on the heating automation, I start getting the same problem with the stirring in some of the reactors, seemingly quite random which ones:
‘‘Stirring RPM is 0 - attempting to restart it automatically. It may be a temporary stall, target RPM may be too low, insufficient power applied to fan, or not reading sensor correctly.’’

Thought I had, could it maybe be a power issue? The heating and stirring ask too much of the reactor or outlet that I am using perhaps.

The not properly stirring pioreactors all have a PWM intensity of 100%, probably to try and catch the stirrer?

Edit: I did a hardware check to see if there was any similarity between the 3 non-stirring reactors thats different to the rest. I Couldn’t find anything like difference in distance between magnet stirrer and reactor, or any touching parts. For your information, I am trying to use crossbar stirrers and am stopping the stirring for every OD measurement using the code described in this forum

19

The not properly stirring pioreactors all have a PWM intensity of 100%, probably to try and catch the stirrer?

hm, one theory: that could be spinning too fast for the sensor to read it.

stopping the stirring for every OD measurement using the code described in this forum

Can you share this? I suspect there are some updates we can make to this code to make it work better.

20

This is the link to the code:

hm, one theory: that could be spinning too fast for the sensor to read it.

I can visually confirm that the stirrer is not working