V1.1 Self test failure - photodiodes are responsive to IR LED

[DonalC]

Hi all,

I recently upgraded to v1.1 on all my Pioreactors (some already calibrated as v1.0, some brand new), and they are all failing one portion of the self test “Photodiodes are responsive to IR LED”. The log I get for these is always of the form (the correlation number is always less than 1 and the majority of the time greater than 0, but it was less than 0 once):

test all positive correlations between pds and leds: 
missing A -> 2, correlation 0.70

I’ve tried rerunning the self test, power cycling, using a diluted culture instead of water, with & without the window cover, and unplugging & replugging the IR and PD cables, but none of these have solved it.

I’m currently on release 24.9.19 (separate issue of when I tried updating via zip file upload or through ssh the leader would update but none of my workers, but none of the changes in more recent versions seem relevant to this issue). Any help or advice diagnosing the issue would be greatly appreciated!

[CamDavidsonPilon]

all of them are failing this just test? Hm, sounds like a software problem maybe, but I haven’t heard of others experiencing this. Can you show me the output after running the following on a failing Pioreactor:

pio run self_test -k test_all_positive_correlations_between_pds_and_leds

[DonalC]

Interestingly, that fixed things for all of them except one, here’s the output of that worker that is still failing:

pioreactor@worker4:~ $ pio run self_test -k test_all_positive_correlations_between_pds_and_leds
2024-10-23T12:57:53-0400 INFO   [self_test] Starting self-test. Running 1 tests.
2024-10-23T12:57:53-0400 DEBUG  [self_test] Using ADC class Pico_ADC.
2024-10-23T12:57:53-0400 DEBUG  [self_test] ADC ready to read from PD channels 1, 2, with gain 1.
2024-10-23T12:57:53-0400 DEBUG  [stirring] Init.
2024-10-23T12:57:53-0400 DEBUG  [stirring] Operating with RPM feedback loop.
2024-10-23T12:57:53-0400 DEBUG  [PWM@GPIO-17] Initialized GPIO-17 using software-timing, initial frequency = 200.0 hz.
2024-10-23T12:57:53-0400 DEBUG  [stirring] Found stirring calibration `linear_v1`.
2024-10-23T12:57:53-0400 INFO   [stirring] Ready.
2024-10-23T12:57:53-0400 DEBUG  [stirring] Starting stirring with 1250.0 RPM.
2024-10-23T12:57:54-0400 DEBUG  [stirring] stirring is blocking until RPM is near 1250.0.
2024-10-23T12:57:59-0400 DEBUG  [self_test] AC hz estimate: 60.0
2024-10-23T12:58:32-0400 DEBUG  [self_test] Corr(A, 1) = 1.0
2024-10-23T12:58:32-0400 DEBUG  [self_test] [(32, 0.11319729826261538), (35, 0.11645316573464659), (53, 0.13969259764568326), (44, 0.1272335102770111), (38, 0.11929265348357343), (47, 0.13106927376252428), (50, 0.13468386939072852), (41, 0.1229954111798649), (56, 0.1416669714107971), (59, 0.14419956902096062), (62, 0.1474579013098966), (65, 0.1505958044823293)]
2024-10-23T12:58:32-0400 DEBUG  [self_test] Corr(A, 2) = 0.84
2024-10-23T12:58:32-0400 DEBUG  [self_test] [(32, 0.07586260960511847), (35, 0.07746413427911912), (53, 0.08440953003490194), (44, 0.08614302616002119), (38, 0.08369686502114804), (47, 0.09914271718540253), (50, 0.09538439622104831), (41, 0.09091027411192128), (56, 0.11801751361277583), (59, 0.10273599170880308), (62, 0.10976676038887971), (65, 0.10599587343183271)]
2024-10-23T12:58:32-0400 DEBUG  [PWM@GPIO-17] Cleaned up GPIO-17.
2024-10-23T12:58:33-0400 INFO   [stirring] Disconnected.
2024-10-23T12:58:34-0400 DEBUG  [stirring] Disconnected successfully from MQTT.
2024-10-23T12:58:35-0400 DEBUG  [self_test] Correlations between LEDs and PD:
{('A', '1'): 1.0, ('A', '2'): 0.84}
2024-10-23T12:58:35-0400 DEBUG  [self_test] missing A ⇝ 2, correlation: 0.84
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 456, in _run
    test(client, logger, unit, testing_experiment)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 236, in test_all_positive_correlations_between_pds_and_leds
    results[(ir_led_channel, ir_pd_channel)] >= 0.90
AssertionError: missing A ⇝ 2, correlation: 0.84
2024-10-23T12:58:35-0400 WARNING [self_test] test all positive correlations between pds and leds: missing A ⇝ 2, correlation: 0.84
2024-10-23T12:58:35-0400 DEBUG  [self_test] test_all_positive_correlations_between_pds_and_leds: ❌
2024-10-23T12:58:35-0400 INFO   [self_test] 1 failed test ❌

And for comparison, here is the output of a worker where it is successful

pioreactor@worker6:~ $ pio run self_test -k test_all_positive_correlations_between_pds_and_leds
2024-10-23T13:02:04-0400 INFO   [self_test] Starting self-test. Running 1 tests.
2024-10-23T13:02:04-0400 DEBUG  [self_test] Using ADC class Pico_ADC.
2024-10-23T13:02:04-0400 DEBUG  [self_test] ADC ready to read from PD channels 1, 2, with gain 1.
2024-10-23T13:02:04-0400 DEBUG  [stirring] Init.
2024-10-23T13:02:04-0400 DEBUG  [stirring] Operating with RPM feedback loop.
2024-10-23T13:02:04-0400 DEBUG  [PWM@GPIO-17] Initialized GPIO-17 using software-timing, initial frequency = 200.0 hz.
2024-10-23T13:02:05-0400 INFO   [stirring] Ready.
2024-10-23T13:02:05-0400 DEBUG  [stirring] Starting stirring with 1250.0 RPM.
2024-10-23T13:02:05-0400 DEBUG  [stirring] stirring is blocking until RPM is near 1250.0.
2024-10-23T13:02:16-0400 DEBUG  [stirring] Waited 10.7 seconds for RPM to match, breaking out early.
2024-10-23T13:02:17-0400 DEBUG  [self_test] AC hz estimate: 60.0
2024-10-23T13:02:50-0400 DEBUG  [self_test] Corr(A, 1) = 1.0
2024-10-23T13:02:50-0400 DEBUG  [self_test] [(32, 0.09630914656726725), (35, 0.0989580555365065), (53, 0.11874379859027333), (44, 0.108384323695022), (38, 0.10164121052769894), (47, 0.11171294900050915), (50, 0.11458887079339199), (41, 0.10479893438168313), (56, 0.12034435087196012), (59, 0.12287950550699296), (62, 0.1254446291930579), (65, 0.12809717441555016)]
2024-10-23T13:02:50-0400 DEBUG  [self_test] Corr(A, 2) = 0.9
2024-10-23T13:02:50-0400 DEBUG  [self_test] [(32, 0.0549328065469796), (35, 0.054977338379630945), (53, 0.055057973956550134), (44, 0.055034835345700917), (38, 0.05506736581325818), (47, 0.05506356778092433), (50, 0.05511317738678565), (41, 0.055135391863610266), (56, 0.05519240787513422), (59, 0.055256302394984524), (62, 0.055360886322157854), (65, 0.055378338136902176)]
2024-10-23T13:02:50-0400 DEBUG  [PWM@GPIO-17] Cleaned up GPIO-17.
2024-10-23T13:02:51-0400 INFO   [stirring] Disconnected.
2024-10-23T13:02:52-0400 DEBUG  [stirring] Disconnected successfully from MQTT.
2024-10-23T13:02:53-0400 DEBUG  [self_test] Correlations between LEDs and PD:
{('A', '1'): 1.0, ('A', '2'): 0.9}
2024-10-23T13:02:53-0400 DEBUG  [self_test] test_all_positive_correlations_between_pds_and_leds: ✅
2024-10-23T13:02:53-0400 INFO   [self_test] All tests passed ✅

[DonalC]

I reflashed all my pios (for an independent issue similar to Cycle media in dosing does not work) and had the same issue again, but rerunning the calibration a few times got it to work, so it is a reoccurring error for whatever reason.

[Grace_h]

Hey @CamDavidsonPilon, I’m having a similar issue too.

I’ve just upgraded my 2 reactors from 1.0 to 1.1. The worker passed the test just fine, but the leader’s results are less consistent.

At first, it failed the PD magnitude and PD Responsiveness tests. I followed the troubleshooting steps for responsiveness and added a bit of squash to my vial as I had done with the worker when it failed the PD responsiveness test. After re-running it, the PD magnitude test passed but not the PD responsiveness. I unplugged and reinserted the PDs and LEDs at both ends and the next test failed and had the following warning:

This warning sometimes appeared when the test passed too. Each warning had significantly different values all below 0.

I re-ran the test, and it failed with a warning. I came to the forum to see if others had the same issue and found this. When running pio run self_test -k test_all_positive_correlations_between_pds_and_leds it passed.

This immediately updated on the UI. I retested one more time to check, as so far it had flipped back and forth between passing and failing quite a bit.
This test failed too.

I’m not quite sure why it would flip between passing and not, and I’m not sure what to do next.

Any advice would be really appreciated! Thank you

[CamDavidsonPilon]

Hi @Grace_h, we realized this test was a bit flakey previously, and made some improvements to it in the latest software release, 24.12.10.

In lieu of upgrading, have you tried adding a turbid solution into the vial? Even if this test fails, your Pioreactor probably still works fine.