Hello! I am facing the issue that one specific pioreactor that ran perfectly for almost 4 weeks suddenly has started losing connection very often. It happens as far as i can tell at completely random time-points and not in relation to another event happening (such as dilutions, OD measurements etc.). I have downloaded the logs for both the leader and the worker that loses connection and can send them too.
Might be useful to know:
The pioreactor worker that loses connection is part of an experiment with 3 other workers and in a cluster with 16 pioreactors running of a separate modem and wifi so not the local access point.
Furthermore, the message displayed is not that the worker is fully “disconnected” it just prints “lost” in front of all automations that were running before it lost connection. Also it is running an experiment profile on the side of the automations and this one keeps working even whilst the automations are “lost”.
Hi @Ms2013DT,
Sorry about the troubles! Are you able to SSH into that worker in that “lost” state? If so, do the logs in pio logs say anything interesting? Furthermore, if able to SSH in, can you run ps x - do you see your jobs there? Ex:
I am indeed able to ssh into when it is “lost”, what i think happens is that something goes wrong which forces the pioreactor to disconnect and then all the automations that were running now just sit in the “lost” state.
Below are some pictures of the things you asked for:
And here are some logs that i got just right now when it crashed again 5 minutes ago:
Yea, I think this is partially fixed in the latest 25.3.5 release (yesterday). What might be happening is your Pioreactor is browning-out (losing power for a moment), and not cleaning up jobs correctly.
Is this happening early in the experiment, or mid-way through? Perhaps you lose connection after dosing, or some temperature change?
It is something that has started happening after the experiment has been running perfectly smooth for more than 20 days.
I do believe it might have something to do with the dosing automations, since it seems to only happen when I have the turbidostat automation running.
Possibly there is a loose connection in the pump for that Pioreactor? Can you try manually running the pump, shaking the wires a bit, and seeing if that causes a reboot?
Yes it seems like you are right on this one! Every time I manually run the waste pump the worker gets “lost”.
I have now managed to get around the issue by swapping out the pump for another one, but that might still mean that this pump suddenly out of nowhere stopped being functional.
It could be bad wiring (our fault), or something mechanical. Can you email us at info@pioreactor.com?