Hi @CamDavidsonPilon
I started an experiment on Saturday afternoon with the new 40 mL Pioreactors for the first time (on version 25.5.1), and have been working to export it with similar issues to this topic (and Export file is empty). All my workers (except one, separate issue I’m going to make another post about later once this is resolved) and the leader are on 25.5.1 and data was showing up in the plots on the overview screen (but seemed delayed/“laggy” in how it updated). Because of this delay/inaccuracy I thought I’d export to manually view my growth curves, but the exported zip file is empty (22 bytes) unless I add every possible exportable item (then it is 837 bytes, for the experiment profiles csv).
I checked the status with sudo systemctl status pioreactor_startup_run@mqtt_to_db_streaming.service
and restarted with sudo systemctl restart pioreactor_startup_run@mqtt_to_db_streaming.service
but each time I got the following error where it pretty much immediately failed (see error log):
sudo systemctl status pioreactor_startup_run@mqtt_to_db_streaming.service
× pioreactor_startup_run@mqtt_to_db_streaming.service - Start up mqtt_to_db_streaming on boot.
Loaded: loaded (/etc/systemd/system/pioreactor_startup_run@.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Mon 2025-05-12 11:06:23 EDT; 1min 41s ago
Duration: 1.500s
Process: 6136 ExecStart=pio run mqtt_to_db_streaming (code=exited, status=1/FAILURE)
Main PID: 6136 (code=exited, status=1/FAILURE)
CPU: 505ms
May 12 11:06:22 leader pio[6136]: File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 946, in __>
May 12 11:06:22 leader pio[6136]: super().__init__(unit, experiment, source="app")
May 12 11:06:22 leader pio[6136]: File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 282, in __>
May 12 11:06:22 leader pio[6136]: self._check_for_duplicate_activity()
May 12 11:06:22 leader pio[6136]: File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 924, in _c>
May 12 11:06:22 leader pio[6136]: raise RuntimeError(f"{self.job_name} is already running. Skipping.")
May 12 11:06:22 leader pio[6136]: RuntimeError: mqtt_to_db_streaming is already running. Skipping.
May 12 11:06:23 leader systemd[1]: pioreactor_startup_run@mqtt_to_db_streaming.service: Main process exited, code=exited, status=1/FAI>
May 12 11:06:23 leader systemd[1]: pioreactor_startup_run@mqtt_to_db_streaming.service: Failed with result 'exit-code'.
May 12 11:06:23 leader systemd[1]: pioreactor_startup_run@mqtt_to_db_streaming.service: Triggering OnFailure= dependencies.
pioreactor@leader:~ $ sudo systemctl restart pioreactor_startup_run@mqtt_to_db_streaming.service
pioreactor@leader:~ $ sudo systemctl status pioreactor_startup_run@mqtt_to_db_streaming.service
× pioreactor_startup_run@mqtt_to_db_streaming.service - Start up mqtt_to_db_streaming on boot.
Loaded: loaded (/etc/systemd/system/pioreactor_startup_run@.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Mon 2025-05-12 11:08:27 EDT; 1s ago
Duration: 1.502s
Process: 6175 ExecStart=pio run mqtt_to_db_streaming (code=exited, status=1/FAILURE)
Main PID: 6175 (code=exited, status=1/FAILURE)
CPU: 506ms
May 12 11:08:26 leader pio[6175]: File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 946, in __>
May 12 11:08:26 leader pio[6175]: super().__init__(unit, experiment, source="app")
May 12 11:08:26 leader pio[6175]: File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 282, in __>
May 12 11:08:26 leader pio[6175]: self._check_for_duplicate_activity()
May 12 11:08:26 leader pio[6175]: File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 924, in _c>
May 12 11:08:26 leader pio[6175]: raise RuntimeError(f"{self.job_name} is already running. Skipping.")
May 12 11:08:26 leader pio[6175]: RuntimeError: mqtt_to_db_streaming is already running. Skipping.
May 12 11:08:27 leader systemd[1]: pioreactor_startup_run@mqtt_to_db_streaming.service: Main process exited, code=exited, status=1/FAI>
May 12 11:08:27 leader systemd[1]: pioreactor_startup_run@mqtt_to_db_streaming.service: Failed with result 'exit-code'.
May 12 11:08:27 leader systemd[1]: pioreactor_startup_run@mqtt_to_db_streaming.service: Triggering OnFailure= dependencies.
As a result, I tried running pio kill --job-name monitor as you suggested, but this had the same result:
pioreactor@leader:~ $ sudo systemctl status pioreactor_startup_run@mqtt_to_db_streaming.service
× pioreactor_startup_run@mqtt_to_db_streaming.service - Start up mqtt_to_db_streaming on boot.
Loaded: loaded (/etc/systemd/system/pioreactor_startup_run@.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Mon 2025-05-12 11:08:27 EDT; 3min 21s ago
Duration: 1.502s
Process: 6175 ExecStart=pio run mqtt_to_db_streaming (code=exited, status=1/FAILURE)
Main PID: 6175 (code=exited, status=1/FAILURE)
CPU: 506ms
May 12 11:08:26 leader pio[6175]: File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 946, in __init__
May 12 11:08:26 leader pio[6175]: super().__init__(unit, experiment, source="app")
May 12 11:08:26 leader pio[6175]: File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 282, in __init__
May 12 11:08:26 leader pio[6175]: self._check_for_duplicate_activity()
May 12 11:08:26 leader pio[6175]: File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 924, in _check_for_duplicate_activity
May 12 11:08:26 leader pio[6175]: raise RuntimeError(f"{self.job_name} is already running. Skipping.")
May 12 11:08:26 leader pio[6175]: RuntimeError: mqtt_to_db_streaming is already running. Skipping.
May 12 11:08:27 leader systemd[1]: pioreactor_startup_run@mqtt_to_db_streaming.service: Main process exited, code=exited, status=1/FAILURE
May 12 11:08:27 leader systemd[1]: pioreactor_startup_run@mqtt_to_db_streaming.service: Failed with result 'exit-code'.
May 12 11:08:27 leader systemd[1]: pioreactor_startup_run@mqtt_to_db_streaming.service: Triggering OnFailure= dependencies.
Following this, my leader also appeared “greyed out” in the inventory screen (with mqtt_to_db_screening and montior listed as on). Following a power cycle, both these jobs displayed as on, but I now consistently get a mqtt to db streaming error “Encountered error in saving to DB: Object missing field ‘calibrated’. See logs.”, which logs has the following:
2025-05-12T10:29:50-0400 [mqtt_to_db_streaming] WARNING Encountered error in saving to DB: Object missing required field `calibrated`. See logs.
2025-05-12T10:29:50-0400 [mqtt_to_db_streaming] DEBUG Error in parse_od. Payload that caused error: `{"timestamp":"2025-05-12T15:25:49.608237Z","angle":"90","od":0.30547514686295285,"channel":"2"}`. Topic: `pioreactor/worker06/Pioreactor Aeration (20 mL reactor)/od_reading/od2`
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/leader/mqtt_to_db_streaming.py", line 119, in callback
new_rows = parser(message.topic, message.payload)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/leader/mqtt_to_db_streaming.py", line 168, in parse_od
od_reading = msgspec_loads(payload, type=structs.ODReading)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
msgspec.ValidationError: Object missing required field `calibrated`
However, all my workers have calibrations established and set to active, so I don’t understand why it would be listing that as an issue.
Any help would be greatly appreciated. I recognize data from this particular experiment is probably lost, but I’d like to solve this as soon as possible for future experiments, as I’m trying to figure out best practices with the 40 mL compared to the 20 mL versions
