After our previous conversation about 2 of my workers misbehaving, I reflashed the two of them straight to 25.5.1 and re-added my previous calibrations from a download. When I tried starting a simple batch experiment, I found that only the two I had reflashed were properly sending OD data (OD, normalized OD, implied growth rate) to the leader (despite mqtt_to_db_streaming and monitor both displaying as on). The other four workers were sending temperature data and were fully functional in all other ways except they weren’t taking OD data. I checked their jobs in the UI and they were properly running and also tried pio run od_reading, but that errored out because it said the job was already running (similar results for growth_rate_calculating too). I also checked the export file and there was only OD data for the two that I had reflashed earlier.
Today I reflashed all my remaining workers (including ones that weren’t part of the experiment) in case it was a similar situation to what had happened with the other two. However, even after this, still only those two workers I reflashed yesterday would display OD data (did the same checks as I described above). As a final try, I reflashed my leader and reconnected all my workers, so now everything is on fresh installs on 25.5.1. This has not fixed my problem still, as I am only getting OD data from the two workers.
I don’t understand what could possibly be causing this, there are no messages in the logs, no errors/warnings, and everything appears to be functioning properly but I’m not getting any OD data specifically. I don’t even understand what else I could test as there aren’t any errors to follow up on. It also doesn’t appear to be a mqtt_to_db_streaming issue, as I am getting temperature data from all the workers on my current test experiment. The workers that aren’t publishing data also pass self-tests so clearly it isn’t a hardware issue either (though I would be quite surprised for 4 Pioreactors to simultaneous have the same hardware issue as they are powered by a surge protected power supply). These non-publishing workers are also both 20 mL v1.1 and 40 mL 1.0, so it does not appear to be version specific either. Unless it’s somehow something specific to my calibrations, I don’t understand what could possibly be causing this. Attached below is there error for trying to run while the od_reading job is theoretically active:
pioreactor@worker02:~ $ pio run od_reading
2025-05-15T18:57:32-0400 INFO [calibration_transformer] Using OD calibration `od_cal_02_15_25_worker02` for channel 2.
2025-05-15T18:57:32-0400 DEBUG [calibration_transformer] Using OD calibration `od_cal_02_15_25_worker02` of type `od` for PD channel 2, calibration_data.curve_type='poly', calibration_data.curve_data_=[0.0]
2025-05-15T18:57:32-0400 WARNING [od_reading] od_reading is already running. Skipping.
Traceback (most recent call last):
File "/usr/local/bin/pio", line 8, in <module>
sys.exit(pio())
^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 1330, in click_od_reading
od = start_od_reading(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 1282, in start_od_reading
return ODReader(
^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 105, in __call__
obj = type.__call__(cls, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 797, in __init__
super(ODReader, self).__init__(unit=unit, experiment=experiment)
File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 970, in __init__
super().__init__(unit, experiment, source="app")
File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 282, in __init__
self._check_for_duplicate_activity()
File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 924, in _check_for_duplicate_activity
raise RuntimeError(f"{self.job_name} is already running. Skipping.")
RuntimeError: od_reading is already running. Skipping.
and the following is what happens if I stop the od_reading automation in the UI and try to run it from the terminal:
pioreactor@worker03:~ $ pio run od_reading
2025-05-15T20:12:10-0400 INFO [calibration_transformer] Using OD calibration `od_cal_02_15_25_worker03` for channel 2.
2025-05-15T20:12:10-0400 DEBUG [calibration_transformer] Using OD calibration `od_cal_02_15_25_worker03` of type `od` for PD channel 2, calibration_data.curve_type='poly', calibration_data.curve_data_=[0.0]
2025-05-15T20:12:10-0400 DEBUG [od_reading] Init.
2025-05-15T20:12:11-0400 DEBUG [od_reading] Using ADC class Pico_ADC.
2025-05-15T20:12:11-0400 DEBUG [od_reading] ADC ready to read from PD channels 2, 1, with gain 1.
2025-05-15T20:12:12-0400 DEBUG [od_reading] AC hz estimate: 60.0
2025-05-15T20:12:12-0400 DEBUG [od_reading] ADC offsets: {'2': 2740.4580433720907, '1': 2815.725450700035}, and in voltage: {'2': 0.06901336647686127, '1': 0.0709088368994972}
2025-05-15T20:12:12-0400 DEBUG [od_reading] Starting od_reading with PD channels {'2': '90'}, with IR LED intensity 57.87% from channel A, every 5.0 seconds
2025-05-15T20:12:12-0400 INFO [od_reading] Ready.
2025-05-15T20:12:12-0400 DEBUG [od_reading] od_reading is blocking until disconnected.
^C2025-05-15T20:12:56-0400 DEBUG [od_reading] Exiting caused by signal Interrupt.
2025-05-15T20:12:56-0400 INFO [od_reading] Disconnected.
2025-05-15T20:12:57-0400 DEBUG [od_reading] Disconnected successfully from MQTT.
Aborted!
Any help would be greatly appreciated.