Four of six workers not reporting OD

Hi @CamDavidsonPilon

After our previous conversation about 2 of my workers misbehaving, I reflashed the two of them straight to 25.5.1 and re-added my previous calibrations from a download. When I tried starting a simple batch experiment, I found that only the two I had reflashed were properly sending OD data (OD, normalized OD, implied growth rate) to the leader (despite mqtt_to_db_streaming and monitor both displaying as on). The other four workers were sending temperature data and were fully functional in all other ways except they weren’t taking OD data. I checked their jobs in the UI and they were properly running and also tried pio run od_reading, but that errored out because it said the job was already running (similar results for growth_rate_calculating too). I also checked the export file and there was only OD data for the two that I had reflashed earlier.

Today I reflashed all my remaining workers (including ones that weren’t part of the experiment) in case it was a similar situation to what had happened with the other two. However, even after this, still only those two workers I reflashed yesterday would display OD data (did the same checks as I described above). As a final try, I reflashed my leader and reconnected all my workers, so now everything is on fresh installs on 25.5.1. This has not fixed my problem still, as I am only getting OD data from the two workers.

I don’t understand what could possibly be causing this, there are no messages in the logs, no errors/warnings, and everything appears to be functioning properly but I’m not getting any OD data specifically. I don’t even understand what else I could test as there aren’t any errors to follow up on. It also doesn’t appear to be a mqtt_to_db_streaming issue, as I am getting temperature data from all the workers on my current test experiment. The workers that aren’t publishing data also pass self-tests so clearly it isn’t a hardware issue either (though I would be quite surprised for 4 Pioreactors to simultaneous have the same hardware issue as they are powered by a surge protected power supply). These non-publishing workers are also both 20 mL v1.1 and 40 mL 1.0, so it does not appear to be version specific either. Unless it’s somehow something specific to my calibrations, I don’t understand what could possibly be causing this. Attached below is there error for trying to run while the od_reading job is theoretically active:

pioreactor@worker02:~ $ pio run od_reading
2025-05-15T18:57:32-0400 INFO   [calibration_transformer] Using OD calibration `od_cal_02_15_25_worker02` for channel 2.
2025-05-15T18:57:32-0400 DEBUG  [calibration_transformer] Using OD calibration `od_cal_02_15_25_worker02` of type `od` for PD channel 2, calibration_data.curve_type='poly', calibration_data.curve_data_=[0.0]
2025-05-15T18:57:32-0400 WARNING [od_reading] od_reading is already running. Skipping.
Traceback (most recent call last):
  File "/usr/local/bin/pio", line 8, in <module>
    sys.exit(pio())
             ^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 1330, in click_od_reading
    od = start_od_reading(
         ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 1282, in start_od_reading
    return ODReader(
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 105, in __call__
    obj = type.__call__(cls, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 797, in __init__
    super(ODReader, self).__init__(unit=unit, experiment=experiment)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 970, in __init__
    super().__init__(unit, experiment, source="app")
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 282, in __init__
    self._check_for_duplicate_activity()
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 924, in _check_for_duplicate_activity
    raise RuntimeError(f"{self.job_name} is already running. Skipping.")
RuntimeError: od_reading is already running. Skipping.

and the following is what happens if I stop the od_reading automation in the UI and try to run it from the terminal:

pioreactor@worker03:~ $ pio run od_reading
2025-05-15T20:12:10-0400 INFO   [calibration_transformer] Using OD calibration `od_cal_02_15_25_worker03` for channel 2.
2025-05-15T20:12:10-0400 DEBUG  [calibration_transformer] Using OD calibration `od_cal_02_15_25_worker03` of type `od` for PD channel 2, calibration_data.curve_type='poly', calibration_data.curve_data_=[0.0]
2025-05-15T20:12:10-0400 DEBUG  [od_reading] Init.
2025-05-15T20:12:11-0400 DEBUG  [od_reading] Using ADC class Pico_ADC.
2025-05-15T20:12:11-0400 DEBUG  [od_reading] ADC ready to read from PD channels 2, 1, with gain 1.
2025-05-15T20:12:12-0400 DEBUG  [od_reading] AC hz estimate: 60.0
2025-05-15T20:12:12-0400 DEBUG  [od_reading] ADC offsets: {'2': 2740.4580433720907, '1': 2815.725450700035}, and in voltage: {'2': 0.06901336647686127, '1': 0.0709088368994972}
2025-05-15T20:12:12-0400 DEBUG  [od_reading] Starting od_reading with PD channels {'2': '90'}, with IR LED intensity 57.87% from channel A, every 5.0 seconds
2025-05-15T20:12:12-0400 INFO   [od_reading] Ready.
2025-05-15T20:12:12-0400 DEBUG  [od_reading] od_reading is blocking until disconnected.
^C2025-05-15T20:12:56-0400 DEBUG  [od_reading] Exiting caused by signal Interrupt.
2025-05-15T20:12:56-0400 INFO   [od_reading] Disconnected.
2025-05-15T20:12:57-0400 DEBUG  [od_reading] Disconnected successfully from MQTT.

Aborted!

Any help would be greatly appreciated.

Okay I figured it out and it was super subtle:

One thing I noticed looking at this as I posted it was that the calibration for od was reading as empty, so I looked into that a bit more. It turns out the OD calibration I was using for the media formulation I was working with was older (before how calibrations were done was overhauled), and so even though I copied it directly from the .yaml file, it was showing without the polynomial curve (just the individual points - the “curve” was just a horizontal line at the origin). I changed the calibrations to a different one for a slightly different media formulation and that worked for the two I’d been testing with, but not the last two. In order to get the last two to work, trying pio run od_reading in the terminal wasn’t enough, I had to turn off the OD and growth rate automations, then run the command to start reading, abort it, and restart the automation.

I think this was mostly my fault for not checking my calibrations more closely, but maybe some sort of warning that any calibrations from the old calibration system are deprecated would be nice.

!! yea we can easily add a valid calibration check. Sorry about this!

In fact, the logs you posted have a example of an invalid calibration:

calibration_data.curve_data_=[0.0]

Right so what’s weird is that the calibration is “valid” in that it has the proper data format under the old system, it just for some reason didn’t autogenerate the coefficients for the polynomial fit (despite me running with it multiple times back in February) in the newer version of the .yaml file when I downloaded it as a backup from the UI. For example, the first block of code is the older one that didn’t work, and the second is a newer one that does work. A future improvement to prevent this might be something for OD calibrations in particular that if they don’t have a fitted curve (i.e. analyzed) OD related jobs can’t be started with that calibration set active.

calibration_type: od
calibration_name: od_cal_02_15_25_worker02
calibrated_on_pioreactor_unit: worker02
created_at: 2025-02-15 22:20:27.728208+00:00
curve_data_:
- 0
curve_type: poly
x: OD600
y: Voltage
recorded_data:
  x:
  - 1.728
  - 1.3292307692307692
  - 1.08
  - 1.093
  - 0.8407692307692307
  - 0.683125
  - 0.715
  - 0.5499999999999999
  - 0.44687499999999997
  - 0.403
  - 0.31
  - 0.251875
  - 0.288
  - 0.22153846153846152
  - 0.18
  - 0.178
  - 0.1369230769230769
  - 0.11124999999999997
  - 0.112
  - 0.08615384615384616
  - 0.07
  - 0.072
  - 0.05538461538461538
  - 0.045
  - 0.041
  y:
  - 0.3402798604890755
  - 0.3656732976998638
  - 0.0034348253147851032
  - 0.3193280909434557
  - 0.2628434938412554
  - 0.2228773012004922
  - 0.2163942544564001
  - 0.16543878618764363
  - 0.13512413778638138
  - 0.13652874520143743
  - 0.10470049218494812
  - 0.08736146194582833
  - 0.08534586077267065
  - 0.06873895935845514
  - 0.05669983195979364
  - 0.05714887423255021
  - 0.04537940956345676
  - 0.03955609360106227
  - 0.040214811801494055
  - 0.03340793298143788
  - 0.025679675711402852
  - 0.026515951993872664
  - 0.020929899981524506
  - 0.018089319582161298
  - 0.0018912683628450038
ir_led_intensity: 70.0
angle: '90'
pd_channel: '2'
calibration_type: od
calibration_name: od-cal-2025-03-01
calibrated_on_pioreactor_unit: worker02
created_at: 2025-03-01 01:15:41.545904+00:00
curve_data_:
- -0.9822869881109039
- 2.4900619033046962
- -1.9395983232439824
- 0.7042398710285068
- -0.0254230601148485
curve_type: poly
x: OD600
y: Voltage
recorded_data:
  x:
  - 1.196
  - 1.244
  - 0.956
  - 0.724
  - 0.356
  - 0.042
  - 0.042
  y:
  - 0.30792763248120164
  - 0.2797192627850483
  - 0.22106082808539057
  - 0.14853549028189628
  - 0.07446716408117778
  - 0.0009235280290714283
  - 0.00110971888096573
ir_led_intensity: 70.0
angle: '90'
pd_channel: '2'