Lately, all the Pioreactors were measuring higher OD for really low values (e.g., OD Pio = 0.3-1.2 for OD = 0.05 - measured by spectrophotometer). Thinking that it was me messing up with the pioreactor OD calibration caused this issue:
- I re-flashed the leader with leader_only image and workers with worker_image
- Replaced the HAT, Raspberry Pi, LED, and photodioes on some of the Pioreactors
After this, I ran self-test on all 8 Pioreactors. However, to my suprise, regardless of replacing hardware or not, I constantly got the error log:
in test PD is near 0 volts for blank: Blank signal too high: mean_signal=1.1106748282254029 > 0.035
I tried to update the firmware on the workers, however, this did not solve the problem.
Did anyone get similar problem or have any idea what can be the cause? Perhaps
Cluster status:
- Cluster connection = Local Access Point on Raspberry Pi
- Software version = 24.5.22
- Firmware version = 0.2
Hi @sharknaro,
Hm mean_signal=1.1106748282254029 > 0.035 does seem really high.
Hm, I wonder if the REF signal is very low, since that would cause a very high PD signal. Do any other the other self-test tests fail as well?
And did this test fail for all 8 workers?
Hi Cameron,
I do not clearly remember but some workers also failed reference photodiode is correct magnitude, but for for most photodiode measured near zero signal for clear water is the only one that is failing.
Yes, near zero for clear water failed for all 8 pioreactors.
A quick way to check what each signal is dosing (REF and non-REF), try the following:
- Make a new experiment so as to not clutter an existing experiment. Assign Pioreactors.
- Under
[od_config.photodiode_channel] in your config.ini, change both values to 90:[od_config.photodiode_channel]
1=90
2=90
and save.
- Run OD reading for all Pioreactors. In the OD reading chart, you should see 2 lines per Pioreactor: one for each PD sensor.
I suspect that the [REF] signal is very very small. In my screen shot below, I’m using water (no stirring), and my [REF] signal is 0.06, which is normal (anything between 0.01 and 0.200 is normal), and my [90deg] signal is near 0, as expected for water.
Does your data look something like this?
Note that we are plotting the “raw” signals in the chart above. Normally in this chart, we plot the ratio:
[90deg] / [REF]
and call that the “OD Reading”.
Hence, if your [REF] is very low, it makes the entire ratio very large. By using led_intensity=auto, we set the IR led intensity s.t. [REF] is equal to 0.1.
Yes, this makes sense. Thanks for the quick suggestion Cameron. Unfortunately I am out of the lab and did not manage to leave a vial in one of the Pioreactors to test this remotely, so I will run this as first thing tomorrow.
Quick note, in my cluster config.ini the led_intensity=50 as when I set it to led_intensity=auto I cannot save the OD calibration data due to requiring a float entry instead of string. At least this was the case last time I calibrated with led_intensity=auto. However, I tried to play with the intensity level and set it both led_intensity=auto and led_intensity=20. Though, the results were still the same, so that, the same test was still failing.
I will keep you updated tomorrow.
Okay so I realized that the config.ini file of some Pioreactors were in version = 1.1, so I fixed that issue assuming I added them wrong at the beginning. However, then I saw that other Pioreactors which were version = 1.0 in the config.ini, stated to be version = 1.1 in the inventory.
In advance, I apologize for the poor quality picture below, I realized this issue after I updated the Pioreactors, and so did not manage to save a clear picture!
I changed the config.ini for the version = 1.1 stated Pioreactors, and rebooted all of them. After that, I ran the tests as you suggested and for some Pioreactors (P06 - P07) I got similar to your results:
Whereas for others, I got really high (P04 - P05) and zero (P08):
Hence, I tried to unplug/replug the PD and LED on P07 and P06 to test if they will work, as the results looked similar to yours, and today I got 3-4 failed tests instead of 1:
In the mean time, I will try to troubleshoot this, but do not have a clear reason why all this happening, so if anyone in the team has an idea, I will be happy to put it to test.
Yea, a restart is required to update the Inventory page with the new model. It’s a bit annoying now sorry.
Many self-tests will fail while you have the config:
[od_config.photodiode_channel]
1=90
2=90
The above config is just for testing the signals. Revert back to
[od_config.photodiode_channel]
1=REF
2=90
to do self tests
Okay so why are some signals very high? Try pushing the IR LED in more - I’ve seen if it’s too far back, too much light hits the REF PD. Generally a high REF can be okay, too.
Why are some signals very low, like in P08? This is a more serious problem. Can you check the connections?
- make sure the PD leads aren’t touching
- try swapping the PDs to see if one of them is not working as expected.
Also, using IR intensity = 50 in the config is fine. Prior to auto, we hardcoded 50, so it should be a good value.
Okay so first of all, I apologize for my questions:
My memory went totally blind and I forgot to revert the config setups on the Pioreactors.
I will try to replace some of the “sealing caps” for the IR, some of them got loose over time because we were pressing it to seal well. Hence, replacing new ones should help for the stated issue.
I replaced the places of the PDs and it seemed to recover:
Perhaps, it was a by-chance missed conduction between the REF PD and HAT channel 2?
Though, the PD near zero test still fails:
in test PD is near 0 volts for blank: Blank signal too high: mean_signal=0.16571198797839162 > 0.035
Perhaps! Maybe just an imperfect connection somewhere.
in test PD is near 0 volts for blank: Blank signal too high: mean_signal=0.16571198797839162 > 0.035
This might be okay anyways? I’ve made some simple improvements to this test in the next release that will be easier to diagnose issues. Simply: we don’t use the REF, which as we know, if very small, can blow up the signal even if the [90 deg] signal is within bounds.
1 Like
