Hi guys, I have updated the leader to 25.4.11 to solve the “PCB not responding” issue, but now i am getting a new warning:
Do you think the issue is that my leader is at 25.4.11 and the UI is at 25.4.9 (this is the most recent right)?
I tried to check if the mqtt_to_db_streaming.service is running:
Do you have any idea what is happening? I am not too worried because mqtt_to_db_streaming.service seems to be running, but it is bugging me that the warning keeps popping up every 1 minute.
Hi @Eleni_M,
Are all your other Pioreactors on the same software version (more generally, 25.4.3 or later)? The field calibrated
was introduced in 25.4.3, and we now expect od-readings to have that boolean field.
The ones that are online yes…there are 4-5 that are in a box not online. Does that matter?
That doesn’t matter. Just some ideas:
Hm, did the od_reading job start before you updated the workers?
Do you have any plugins that might modify od_reading?
On that sudo systemctl status pioreactor_startup_run@mqtt_to_db_streaming.service
output, you can press the right-arrow key to view more to the right (it’s cutoff). If you do, you will probably see the message that is causing the problem, which would include a pioreactor unit name - does that provide any clues?
The missing part of the message is this: Error in parse_od. Payload that caused error: {"timestamp":"2025-04-15T19:06:13.841774Z","angle":"90","od":0.0021015880433365676,"channel":"2"}
I don’t think the OD readings were on while i updated. The temperature automation was the first thing we tried to start and then realized there is a problem with the software version we had. then we updated to the latest version. It seems that the problem comes from the missing “calibrated” field. But we are not using any calibration for the OD.
The calibrated
field is just populated True
or False
, so all od readings should have it, regardless of it they are calibrated or not.
Hm, I’m not sure what is happening. It looks like one of your workers (or more) is running an older software, but your inventory page suggests otherwise. Here is something you can do to narrow down the problem:
- SSH into leader, and run:
pio mqtt -t "pioreactor/+/+/od_reading/od2"
- This streams real-time OD readings from your workers to this console. Look for messages that don’t contain a
calibrated
field. On that same line, there will be a pioreactor worker unit name, Ex: 2025-04-15T20:31:15 | pioreactor/worker01/expABC/od_reading/od2 | {"timestamp":"2025-04-15T19:31:15.334000Z","angle":"90","od":4.001058114862541,"channel":"2"}
is a message associated with worker01
, and lacks the calibration
field.
- ctrl-c to exit.
If you can find the worker(s), try rebooting them and seeing if the problem persists. You can do this later, too. But note that none of that worker’s OD reading data is being saved to the database until you resolve this problem!
Ok this helps a lot! I see that worker9, 10 and 11 don’t have the calibration field. These are not assign to the experiment that is running now, so thats ok, i will power cycle them tomorrow because rebooting them remotely didn’t fix it. But good to know that the workers that are actually running an experiment now are logging the ODs. 
Today i power cycled them and tried again here is what i get:
I notice that for worker10 and 11 first it doesnt say “calibrated”:0 and then on the next line it does. while for worker02 and worker03 it says “calibrated”:0 immediately.
But this looks right:
And the warnings have stopped. So maybe its solved?