V1.1 Self test failure - photodiodes are responsive to IR LED

Hi all,

I recently upgraded to v1.1 on all my Pioreactors (some already calibrated as v1.0, some brand new), and they are all failing one portion of the self test “Photodiodes are responsive to IR LED”. The log I get for these is always of the form (the correlation number is always less than 1 and the majority of the time greater than 0, but it was less than 0 once):

test all positive correlations between pds and leds: 
missing A -> 2, correlation 0.70

I’ve tried rerunning the self test, power cycling, using a diluted culture instead of water, with & without the window cover, and unplugging & replugging the IR and PD cables, but none of these have solved it.

I’m currently on release 24.9.19 (separate issue of when I tried updating via zip file upload or through ssh the leader would update but none of my workers, but none of the changes in more recent versions seem relevant to this issue). Any help or advice diagnosing the issue would be greatly appreciated!

all of them are failing this just test? Hm, sounds like a software problem maybe, but I haven’t heard of others experiencing this. Can you show me the output after running the following on a failing Pioreactor:

pio run self_test -k test_all_positive_correlations_between_pds_and_leds

Interestingly, that fixed things for all of them except one, here’s the output of that worker that is still failing:

pioreactor@worker4:~ $ pio run self_test -k test_all_positive_correlations_between_pds_and_leds
2024-10-23T12:57:53-0400 INFO   [self_test] Starting self-test. Running 1 tests.
2024-10-23T12:57:53-0400 DEBUG  [self_test] Using ADC class Pico_ADC.
2024-10-23T12:57:53-0400 DEBUG  [self_test] ADC ready to read from PD channels 1, 2, with gain 1.
2024-10-23T12:57:53-0400 DEBUG  [stirring] Init.
2024-10-23T12:57:53-0400 DEBUG  [stirring] Operating with RPM feedback loop.
2024-10-23T12:57:53-0400 DEBUG  [PWM@GPIO-17] Initialized GPIO-17 using software-timing, initial frequency = 200.0 hz.
2024-10-23T12:57:53-0400 DEBUG  [stirring] Found stirring calibration `linear_v1`.
2024-10-23T12:57:53-0400 INFO   [stirring] Ready.
2024-10-23T12:57:53-0400 DEBUG  [stirring] Starting stirring with 1250.0 RPM.
2024-10-23T12:57:54-0400 DEBUG  [stirring] stirring is blocking until RPM is near 1250.0.
2024-10-23T12:57:59-0400 DEBUG  [self_test] AC hz estimate: 60.0
2024-10-23T12:58:32-0400 DEBUG  [self_test] Corr(A, 1) = 1.0
2024-10-23T12:58:32-0400 DEBUG  [self_test] [(32, 0.11319729826261538), (35, 0.11645316573464659), (53, 0.13969259764568326), (44, 0.1272335102770111), (38, 0.11929265348357343), (47, 0.13106927376252428), (50, 0.13468386939072852), (41, 0.1229954111798649), (56, 0.1416669714107971), (59, 0.14419956902096062), (62, 0.1474579013098966), (65, 0.1505958044823293)]
2024-10-23T12:58:32-0400 DEBUG  [self_test] Corr(A, 2) = 0.84
2024-10-23T12:58:32-0400 DEBUG  [self_test] [(32, 0.07586260960511847), (35, 0.07746413427911912), (53, 0.08440953003490194), (44, 0.08614302616002119), (38, 0.08369686502114804), (47, 0.09914271718540253), (50, 0.09538439622104831), (41, 0.09091027411192128), (56, 0.11801751361277583), (59, 0.10273599170880308), (62, 0.10976676038887971), (65, 0.10599587343183271)]
2024-10-23T12:58:32-0400 DEBUG  [PWM@GPIO-17] Cleaned up GPIO-17.
2024-10-23T12:58:33-0400 INFO   [stirring] Disconnected.
2024-10-23T12:58:34-0400 DEBUG  [stirring] Disconnected successfully from MQTT.
2024-10-23T12:58:35-0400 DEBUG  [self_test] Correlations between LEDs and PD:
{('A', '1'): 1.0, ('A', '2'): 0.84}
2024-10-23T12:58:35-0400 DEBUG  [self_test] missing A ⇝ 2, correlation: 0.84
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 456, in _run
    test(client, logger, unit, testing_experiment)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 236, in test_all_positive_correlations_between_pds_and_leds
    results[(ir_led_channel, ir_pd_channel)] >= 0.90
AssertionError: missing A ⇝ 2, correlation: 0.84
2024-10-23T12:58:35-0400 WARNING [self_test] test all positive correlations between pds and leds: missing A ⇝ 2, correlation: 0.84
2024-10-23T12:58:35-0400 DEBUG  [self_test] test_all_positive_correlations_between_pds_and_leds: ❌
2024-10-23T12:58:35-0400 INFO   [self_test] 1 failed test ❌

And for comparison, here is the output of a worker where it is successful

pioreactor@worker6:~ $ pio run self_test -k test_all_positive_correlations_between_pds_and_leds
2024-10-23T13:02:04-0400 INFO   [self_test] Starting self-test. Running 1 tests.
2024-10-23T13:02:04-0400 DEBUG  [self_test] Using ADC class Pico_ADC.
2024-10-23T13:02:04-0400 DEBUG  [self_test] ADC ready to read from PD channels 1, 2, with gain 1.
2024-10-23T13:02:04-0400 DEBUG  [stirring] Init.
2024-10-23T13:02:04-0400 DEBUG  [stirring] Operating with RPM feedback loop.
2024-10-23T13:02:04-0400 DEBUG  [PWM@GPIO-17] Initialized GPIO-17 using software-timing, initial frequency = 200.0 hz.
2024-10-23T13:02:05-0400 INFO   [stirring] Ready.
2024-10-23T13:02:05-0400 DEBUG  [stirring] Starting stirring with 1250.0 RPM.
2024-10-23T13:02:05-0400 DEBUG  [stirring] stirring is blocking until RPM is near 1250.0.
2024-10-23T13:02:16-0400 DEBUG  [stirring] Waited 10.7 seconds for RPM to match, breaking out early.
2024-10-23T13:02:17-0400 DEBUG  [self_test] AC hz estimate: 60.0
2024-10-23T13:02:50-0400 DEBUG  [self_test] Corr(A, 1) = 1.0
2024-10-23T13:02:50-0400 DEBUG  [self_test] [(32, 0.09630914656726725), (35, 0.0989580555365065), (53, 0.11874379859027333), (44, 0.108384323695022), (38, 0.10164121052769894), (47, 0.11171294900050915), (50, 0.11458887079339199), (41, 0.10479893438168313), (56, 0.12034435087196012), (59, 0.12287950550699296), (62, 0.1254446291930579), (65, 0.12809717441555016)]
2024-10-23T13:02:50-0400 DEBUG  [self_test] Corr(A, 2) = 0.9
2024-10-23T13:02:50-0400 DEBUG  [self_test] [(32, 0.0549328065469796), (35, 0.054977338379630945), (53, 0.055057973956550134), (44, 0.055034835345700917), (38, 0.05506736581325818), (47, 0.05506356778092433), (50, 0.05511317738678565), (41, 0.055135391863610266), (56, 0.05519240787513422), (59, 0.055256302394984524), (62, 0.055360886322157854), (65, 0.055378338136902176)]
2024-10-23T13:02:50-0400 DEBUG  [PWM@GPIO-17] Cleaned up GPIO-17.
2024-10-23T13:02:51-0400 INFO   [stirring] Disconnected.
2024-10-23T13:02:52-0400 DEBUG  [stirring] Disconnected successfully from MQTT.
2024-10-23T13:02:53-0400 DEBUG  [self_test] Correlations between LEDs and PD:
{('A', '1'): 1.0, ('A', '2'): 0.9}
2024-10-23T13:02:53-0400 DEBUG  [self_test] test_all_positive_correlations_between_pds_and_leds: ✅
2024-10-23T13:02:53-0400 INFO   [self_test] All tests passed ✅

I reflashed all my pios (for an independent issue similar to Cycle media in dosing does not work) and had the same issue again, but rerunning the calibration a few times got it to work, so it is a reoccurring error for whatever reason.