Hey @CamDavidsonPilon,
I am having this strange issue where the pio crashes when I run the air_bubbler and growth_rate_calculating jobs at the same time ONLY when enable_dodging_od is on under [air_bubbler.config]. I have tested extensively and this specific condition is the only time it occurs.
This does not occur when I am dodging LEDs during od_reading and growth_rate_calculating so it should be specific to the air_bubbler plugin. When the pio crashes I am not getting any error messages when running manually via ssh so I have nothing to go off of here.
Do you have any idea where the air_bubbler OD dodging and growth_rate_calculating coincide in the code? If you point me in the right direction, I can investigate it further.
Hey @Eleni_M,
From previous posts, I know you also use the air_bubbler plugin. Have you experienced this same issue described above?
After further investigation, I have found a workaround that may be pointing to the culprit.
If I leave the growth_rate_calculating job to finish calculating the OD statistics, and then turn on air_bubbler then it doesn’t crash. I am running long-term experiments meaning my OD sampling rate is low. So it takes a while for it to finish gathering data points to calculate the OD_statistics. Could the issue be related to MQTT?
Like the RPi freezes up or shuts down? Or that some parts of the software become unresponsive?
The two jobs don’t really intersect. Both listen to od_readings (via MQTT), but that’s a one way information transfer.
Now, the initial period in growth rate job will turn on stirring (if not currently turned on). This could cause problems if the air-bubbler and stirring are somehow connected (ex: via PWMs or hardware or power).
Otherwise, I’m not sure what might be going on. I’ll try to repro locally too.
I’m not sure. SSH freezes up and can’t reconnect until I power cycle. Also, the UI becomes unresponsive and does not show any data. The Pi does not turn off fully and starts actuating the external LED and air bubbler sporadically.
I have this disabled on my system as I do not use stirring. I did this by removing the relevant stirring code from od_blank.py
I don’t have any issues with the air_bubbler otherwise (using the air_bubbler default plugin). The air bubbler is connected via PWM with the following settings:
Ended up finding my issue. I was running out of memory. This was not happening prior but I have since started using the recommended VPN (tailscale) for remote access. And I am using the Pi Zero 2W which doest have too much memory. Tailscale was using 13%. Pio run and mqtt streaming were also using a similar amount each. But issue arrises during memory spikes occuring in scenario mentioned above as well as when I use the stop all feature.
Solution was to allocate some swap memory for the memory spikes that occur at different scenarious. You may want to include this in the docs for others using pi 02w
Interesting! I haven’t seen this happen before, but I’ll add monitoring for memory spikes and investigate memory footprints of the mentioned jobs / actions.
