Air bubbler dodging OD issues

Software
1

In our system, OD readings are taken every 10 seconds and the air bubbler runs for 3 seconds during each interval (see config below). We’ve had a few instances where the air bubbler failed to dodge the OD readings and just switched on constantly during an experiment. This results in excessive evaporation and spills from the air outflow tube.

From reviewing the logs, the issue appears to occur when an experiment profile attempts to restart activities that are already active.

In one example, OD readings were already running, so the restart was skipped. However, the air bubbler seems to have received a message that OD readings had been turned off—even though they were still active. Immediately after this, the air bubbler stopped dodging OD readings.

timestamp message level task
2025-11-13T09:41:42.041Z od_reading is already running. Skipping. WARNING od_reading
2025-11-13T09:41:42.140Z OD reading turned off. Stop dodging. DEBUG air_bubbler

In another example, both the OD readings and the air bubbler were already running when the profile attempted to activated them again. In this case, there were no logs from the air bubbler about dodging OD readings, but the air bubbler stopped dodging immediately after this restart attempt:

timestamp message level task
2025-09-11T18:12:32.168Z od_reading is already running. Skipping. WARNING od_reading
2025-09-11T18:12:33.276Z air_bubbler is already running. Skipping. WARNING air_bubbler

We will try not to restart activities that are already started, but it may be a more general issue.

[od_reading.config]
samples_per_second=0.1
ir_led_intensity=50
pd_reference_ema=0.9
smoothing_penalizer=6.0
duration_between_led_off_and_od_reading = 0.125
turn_off_leds_during_reading=0

[air_bubbler.config]
duty_cycle=10
hertz=200
pre_delay_duration=6
post_delay_duration=1
enable_dodging_od=1

2

!!! yea that’s convincing! Let me try to reproduce this locally

3

I dont see how it failing dodging will lead to higher evaporation if its only dodging for a few a short duration. How long is air bubbler shut off for?

Also strange you are reporting high evaporation and over flowing as they are effectively opposite.

Overflowing could indicate a high void fraction, as water level gets higher when air is on. Also makes waste removal less effective as the waste pump is removing a mix of air and water (effectively lower than calibration flowrate when aerating). I would try increasing the waste pump multiplier to prevent the overflow problem. If aeration is turned off during dilution then disregard this.

I similarly had a bunch of problems with od dodging turning off bubbler, ended up rewriting it.

4

So I was able to repro this error, and it’s like you said: when od_readings tries (and fails) to run, it can break dodging jobs. This will be fixed in the next release. Thanks for the bug report!

5

Also, you may be able to avoid this issue now by using conditionals in the profile step that starts od_reading:

    od_reading:
      actions:
        - type: start
          hours_elapsed: 6.0
          if: ${{::od_reading:$state != ready }}
1 Like
6

Thanks Cameron! We’ll use the conditional for now, thanks for the suggestion.

7

Thanks for you response. The air bubbler is off for 7 seconds every 10 seconds, so if it is on constantly, the air flow is effectively 3x higher than it is supposed to :slight_smile: