Abnormal behavior of optics system

Hi there, I recently completed a batch growth experiment that had some bizarre behavior of the optical system in multiple Pioreactors that I’d like some help understanding.

My experiment had varied media composition across conditions (36, 3.6, and 0.36 g/L sucrose with all other nutrients with constant concentration). I calibrated the optical systems of my Pioreactors with a relatively dense culture grown in medium with 36 g/L sucrose (as I didn’t have dense cultures of the other conditions at that point).

I used the “older” OD calibration method via serial dilutions (not the newer one with set standards) and used the same vial of culture for all 9 of my workers so that they had consistent measurements across individual sets of photodiodes. To do this, between each step I moved the vial with culture to another worker (after the measurement was complete and I was prompted to add more media), took that measurement, and repeated for all my workers (and then looped back to the first one). I started at a measured OD600 on a plate reader of 1.73 and ended with my media blank at 0.041. I diluted in 3 mL additions (18 total steps per worker) and took measurements on a plate reader for cross reference at each opportunity.

The spread between the voltage range of individual workers’ seemed relatively wide to me, so I don’t know why that was the case. I also would’ve thought over such a range the calibration would fit a 4th order polynomial, but it reported 1st order for all of my calibrations (and seemed to ignore my densest point in all cases, maybe because of how much of a change there was in the reading from the first dilution). See graph below:

When I went to prep for my experiment by taking a blank measurement with each Pioreactor, when I clicked the button in the UI on the first worker, the wheel icon spun for a while with the button greyed out and then it stayed that way with no further progress. There was nothing in the logs that showed that this action had occurred or that it was making progress and all of my other workers had the blank button similarly greyed out as if the operation was still occurring.

I was under a slight time crunch, so I moved forward with starting the automations for my experiment with my inoculated cultures. For each condition, I prepared one inoculum with extra volume for each condition (30 mL) and then dispensed 10 mL of media + cells to 2 vials per condition (along with a no cell of plain media). Despite my earlier calibration using very similar media that should not have changed the OD reading significantly (based on measurements performed on multiple different plate readers), when I started my automations, I got a warning in all of my cultures that the signal was outside the calibration range. Signal outside suggested calibration range. Trimming signal. Calibrated for OD=[0.041, 1.73], V=[0.00189, 0.366]. Observed 0.041V.

The reported value was lower than what blank media at the end of the calibration had reported as, despite being in some cases identical or quite similar (sucrose concentration in this range appeared to have a very minor impact on scattering in my experience). Additionally, all the inoculated cultures read notably lower than what was recorded on the plate reader - obviously there would be some variation, but this was beyond what I would expect, especially after having been calibrated.

When I checked the plots the next day, there was bizarre behavior in two of the workers (I’m only showing one here representatively). Despite being inoculated with cells, the initial density held perfectly constant for about 4 hours (likely due to being below the calibration threshold, which was somewhat expected given the warnings). Following this, the reported OD dropped, and then slowly climbed over the length of the experiment (but nowhere near the actual expected growth based on the replicate that did not have this issue). See graphs below of the no cell, impacted worker, and unaffected worker respectively.



This happened with a worker in two of my conditions and in the third condition while this didn’t happen, the growth curve of one of the replicates was significantly depressed compared to each other. Keep in mind in all of these cases, the inoculum between the two replicates for each condition was identical. I’ve attached a graph I plotted of each worker for each condition grouped for easier visibility that shows this phenomenon:

When I ended the experiment, I checked the final cell density of each culture on two different plate readers, and while there was some variation between replicates, they were relatively close to each other in final density (except for the last case where there was reduced growth in one replicate, so that likely was a biological issue rather than a hardware/software issue).

I recognize something clearly was not ideal about my calibration that led to some of these issues, but I’m having trouble wrapping my head around what would’ve caused the weird phenomenon multiple hours into the experiment in two of my workers. From checking the logs (which is more difficult given how frequent the signal warning was posted), I don’t see anything different that caused the signal drop and it to start performing properly. I know this is a very long post, but I wanted to provide as much context as I could to try to figure out what caused this error, I’ve never encountered anything like it before. Thank you!

Hi @DonalC,

I’m sorry for the difficulties! I’ve identified a few places where we can improve things, and ways to help your next experiment.

  1. For the OD calibrations, the default was set to 1 degree, but in the calibration routine, you are also given the choice of degree. Either way, you can change the degree now:

    • on any worker, try:
      pio calibrations analyze --device od --name <name of your od calibration>
      
      This will open up a new chart + way to change the degree. (You can get the name with pio calibrations list --device od).
  2. There’s a UI lag where the “Run blanks” button appears as clickable, but it actually should be disabled. If you’ve done an OD calibration, you don’t need to run a blank.

  3. I suspect a lot of the problems around “Signal outside suggested calibration range.” were caused by the linear calibration curve (btw, we’ve updated the default to three for the next release) - the raw signal seen was mapped to a linear curve, which may have produced a negative number (for example), and raising a warning, and returning some default. Likewise, I suspect the “drop” you saw is a similar problem, where we return the max OD in some cases: pioreactor/pioreactor/background_jobs/od_reading.py at master · Pioreactor/pioreactor · GitHub - I’m not 100% sure, but I suspect the error is something like that. Can you export your calibrations and send them to me cam@pioreactor.com?

  4. The calibration data you posted looks strange - there is that


Nvm, we found a pretty significant bug in new OD calibrations created after 25.1.22. We’re preparing a hot fix.

Here’s how you can fix this right now. This only applies to od calibrations:

  1. For each worker, SSH in. We’re going to edit the calibrations files:
  2. nano ~/.pioreactor/storage/calibrations/od/<the calibration name>.yaml
  3. Once editor is open, make the following changes:
      1. Change the lines:
        • x: voltagesx: OD600
        • y: od600sy: Voltage
      1. Swap the x and y in recorded_data. Example:
      recorded_data:
        x:
        - 0.08217615673010362
        - 0.07865894848996766
        - 0.07763404630172312
        - 0.07704347951556274
        - 0.07564041787627875
        - 0.07432519043884285
        - 0.0745673706532032
        - 0.011509167852880845
        y:
        - 1.0
        - 0.8333333333333334
        - 0.7142857142857143
        - 0.625
        - 0.5555555555555556
        - 0.5555555555555556
        - 0.46296296296296297
        - 0.04
      

      to

      recorded_data:
        y:
        - 0.08217615673010362
        - 0.07865894848996766
        - 0.07763404630172312
        - 0.07704347951556274
        - 0.07564041787627875
        - 0.07432519043884285
        - 0.0745673706532032
        - 0.011509167852880845
        x:
        - 1.0
        - 0.8333333333333334
        - 0.7142857142857143
        - 0.625
        - 0.5555555555555556
        - 0.5555555555555556
        - 0.46296296296296297
        - 0.04
      
    • Save and exit with ctrl-x

  4. Re-analyze the curve with:
    pio calibrations analyze --device od --name <name of calibration>
    
    Choose a new degree (can be 1 through 5) and confirm and exit.

We’re working on a script to automate this, too!

Hi @DonalC, a hot fix release is out:

It’s targeted to only fix the OD cal problem: both fix existing OD calibrations, and fixes for future OD calibrations. You’ll still need to run

pio calibrations analyze --device od --name <cal-name>

on each worker to recreate the curves.


The problem only exists for calibrations made after the 25.1.21 release.