Growth rate charts questions and bug

rafik.n · January 23, 2024, 6:04am

Hey,

I have some questions about the growth rate charts. Keep in mind I’m sampling every 5 minutes (not seconds). The growth rate at the start and middle of the chart has high errors due to my messing around with the reactor.

Which of the parameters below should I change to smoothen the growth rate considering my slower sampling rate?
Do I need to restart the growth rate measurement function after changing the config file?
Does it recalculate previously calculated growth rates when changed?
Is the daily growth rate calculated by multiplying the hourly growth rate by 24? Can it be further smoothed?

[growth_rate_kalman]

obs_std ↑ smooths growth rate, rate_std ↑ more responsive growth rate

acc_std=0.0008
obs_std=3.0
od_std=0.005
rate_std=0.20

The image below is from the past experiments tab

The need for growth rate smoothing is exaggerated in the dashboard charts due to downsampling. I may turn it off

Noticed a bug in the chart x-axis in all images above. Label switches from weekdays to months then back to weekdays.

PS: I am aware I can fix the chart axis range. will do it tomorrow

CamDavidsonPilon · January 23, 2024, 6:34pm

Which of the parameters below should I change to smoothen the growth rate considering my slower sampling rate?

hm, I don’t know which ones to tweak yet. Let me think more about this and do some experiments.

Do I need to restart the growth rate measurement function after changing the config file?

Yea, config options are almost always read at the time to job starts. Restarting a job will use the latest config.

Does it recalculate previously calculated growth rates when changed?

No, unfortunately. I’d like to see this changed in the future though. Or a “calculator” in the UI that you can modify the params from an existing OD time series and see the resulting GR curve. This’ll probably happen sooner rather than later.

Is the daily growth rate calculated by multiplying the hourly growth rate by 24? Can it be further smoothed?

Yea, the line here is what you are looking for: pioreactorui/contrib/charts/02_implied_daily_growth_rate.yaml at master · Pioreactor/pioreactorui · GitHub - that’s a js function that’s run ontop of the “raw” growth rate. To further smooth it might not be possible.

rafik.n · January 25, 2024, 5:52am

Spent the evening thinking about this. Torn between these 2 theories:

I realized that this may be a case of garbage in garbage out. I have ema turned off so maybe the not so smooth OD readings entering the filter create disturbances in nOD that is exaggerated in growth rate calculation. This theory is supported by the fact that when the raw OD readings calm down on Monday the 22nd in the previous chart so does the nOD and the growth rate.
On the other hand the whole point of the kf is to take in noisy data and shoot out something stable. Leading me to believe that this can be tuned.

Such a feature would make it much easier to tune because you’d only have to rerun the script instead of reruning an experiment. I attempted to write a python script based on streaming_calculations.py and growth_rate_calculating that takes in the optical density csv file from export data tab and runs the ekf. I am then comparing the results with the kalman filter outputs csv file to see if it matches up. So far no luck will have to look at it again next couple days. I wanted to verify I have the correct understanding of the series of events taking place between raw OD → growth rates:

90 and REF raw OD readings captured
OD = RAW / (EMA(REF) is calculated and output is what is reported via “optical density” graph and export csv
OD value measured in step 2 is normalized by dividing with the initially recorded 30 value average and is fed into EKF
EKF spits out nOD and growth rate which are reported via “normalized optical density” and “Implied growth rate” graphs and export csv files

Ill share my python script when I’m a bit more confident in output

rafik.n · January 26, 2024, 4:00pm

I managed to Frankenstein this code together from the GitHub. Here is what it does: takes in optical density CSV as exported from UI (renamed readings.csv) → calculates needed statistics → normalizes based on these statistics → calculates EKF estimates of OD, growth rate, and acceleration → saves and plots

Of course, I only needed to retune because I’ve made various changes including different photodiodes, turned off ema, and reduced sampling. Nonetheless, it is a useful tool for anyone wanting to see the impact of changing ekf parameters on output smoothing.

Here is what my results looked like with the original params, and after changing obs_std from 3 → 50

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from typing import Sequence
from statistics import variance, mean

global obs_std, od_std, rate_std, acc_std, dt
obs_std = 3.0
od_std = 0.005
rate_std = 0.20
acc_std = 0.0008
dt = 5/60  # Time interval set to 5 mins

# Delete all columns in the optical density CSV file beyond column 6
pd.read_csv('readings.csv', usecols=range(6)).to_csv('readings.csv', index=False)

def trimmed_variance(x: Sequence) -> float:
    x = list(x) 
    max_, min_ = max(x), min(x)
    x.remove(max_)
    x.remove(min_)
    return variance(x)

def trimmed_mean(x: Sequence) -> float:
    x = list(x) 
    max_, min_ = max(x), min(x)
    x.remove(max_)  
    x.remove(min_)
    return mean(x)

def ekf_update(state, covariance, observation, od_variance, od_normalization_factor):

    # Nonlinear state prediction
    od, rate, acc = state
    state_pred = np.array([od * np.exp(rate * dt), rate + acc * dt, acc])

    # Jacobian of the state prediction
    F = np.array([[np.exp(rate * dt), od * np.exp(rate * dt) * dt, 0],
                  [0, 1, dt],
                  [0, 0, 1]])

    # Covariance prediction
    Q = np.diag([(od_std*dt)**2, (rate_std*dt)**2, (acc_std*dt)**2])
    covariance_pred = F @ covariance @ F.T + Q

    # Measurement update
    H = np.array([[1, 0, 0]])
    scaling_obs_variances = np.array(
        [od_variance/ (od_normalization_factor) ** 2]
    )
    R = obs_std**2 * np.diag(scaling_obs_variances)

    # Residual state
    y = observation - H @ state_pred
    # Residual covariance
    S = H @ covariance_pred @ H.T + R
    # Kalman gain
    K = np.linalg.solve(S.T, (H @ covariance_pred.T)).T
    # Update state and covariance
    state_updated = state_pred + K @ y 
    covariance_updated = (np.eye(3) - K @ H) @ covariance_pred

    return state_updated, covariance_updated

def process_optical_density(csv_file):
    # Load the CSV file
    data = pd.read_csv(csv_file)

    # Extract relevant data for calculating od_variance and od_normalization_factor
    relevant_data = data.iloc[9:40, 4]

    # Calculate od_variance and od_normalization_factor using trimmed functions
    od_variance = trimmed_variance(relevant_data)
    od_normalization_factor = trimmed_mean(relevant_data)
    print(f"Calculated OD Variance: {od_variance}")
    print(f"Calculated OD Normalization Factor: {od_normalization_factor}")

    # Normalize the optical density observations
    normalized_observations = data.iloc[:, 4] / od_normalization_factor

    # Initialize EKF parameters
    state = np.array([1, 0, 0])  # Initial state (OD, growth rate, acceleration)
    covariance = 1e-4*np.eye(3)  # Initial covariance matrix

    # Process each optical density reading and store results
    results = []
    normalized_values = []
    for observation in normalized_observations:
        normalized_values.append(observation)
        state, covariance = ekf_update(state, covariance, observation, od_variance, od_normalization_factor)
        results.append(state)

    # Combine the normalized values with the results
    results_df = pd.DataFrame(results, columns=['ekf OD', 'ekf Growth Rate', 'ekf Acc'])
    normalized_df = pd.DataFrame(normalized_values, columns=['Normalized OD'])
    updated_data = pd.concat([data, normalized_df, results_df], axis=1)

    # Save updated data to CSV
    updated_data.to_csv(csv_file, index=False)

   # Plotting
    timestamps = pd.to_datetime(data.iloc[:, 0])  

    plt.figure(figsize=(12, 10))
    dt_hours = dt * 60  # Convert dt from hours to minutes
    title_str = (f"EKF Parameters - obs_std: {obs_std}, od_std: {od_std}, "
                 f"rate_std: {rate_std}, acc_std: {acc_std}, dt: {dt_hours} mins")

    plt.suptitle(title_str)
    plt.tight_layout(rect=[0, 0.03, 1, 0.95])
    
    # Plot 1: Normalized OD and ekf OD
    plt.subplot(4, 1, 1)
    plt.plot(timestamps, normalized_values, label='Normalized OD')
    plt.plot(timestamps, updated_data['ekf OD'], label='ekf OD')
    plt.xlabel('Timestamp')
    plt.ylabel('Optical Density')
    plt.title('Normalized and ekf Optical Density vs Time')
    plt.legend()

    # Plot 2: ekf Growth Rate
    plt.subplot(4, 1, 2)
    plt.plot(timestamps, updated_data['ekf Growth Rate'], label='ekf Growth Rate')
    plt.xlabel('Timestamp')
    plt.ylabel('Growth Rate')
    plt.title('ekf Growth Rate vs Time')
    plt.legend()

    # Plot 3: ekf Daily Growth Rate
    ekf_daily_growth_rate = updated_data['ekf Growth Rate'] * 24
    plt.subplot(4, 1, 3)
    plt.plot(timestamps, ekf_daily_growth_rate, label='ekf Daily Growth Rate')
    plt.xlabel('Timestamp')
    plt.ylabel('Daily Growth Rate')
    plt.title('ekf Daily Growth Rate vs Time')
    plt.legend()

    # Plot 4: ekf Acceleration
    plt.subplot(4, 1, 4)
    plt.plot(timestamps, updated_data['ekf Acc'], label='ekf Acc')
    plt.xlabel('Timestamp')
    plt.ylabel('Acceleration')
    plt.title('ekf Acceleration vs Time')
    plt.legend()

    plt.tight_layout()
    plt.show()

# Process the readings.csv file
process_optical_density('readings.csv')```

CamDavidsonPilon · January 26, 2024, 4:18pm

Nice! If the code works, is it really “frankenstein”?

Some thoughts:

the default KF parameters were based on 5s intervals (and we used a similar script to choose decent values). There may be a “time” factor that is missing somewhere in our original code that extends better to 5m intervals. You did seem to get good results with increasing obs_std, so maybe it’s around there.
In theory, I should be able to take an existing 5s OD time series, subsample by taking every 60th measurement (that’s 5s → 5m) and get a vert similar growth curve. Similar, not exact, because since there is less data coming in, the KF will be “less” confident in its prediction.
Increasing a term like obs_std tells the model “trust the internal dynamics more”, which will smooth the graph. Increasing rate_std tells the model “trust the obs more”. Your non-pioreactor vial holder setup likely (?) produces noisier measurements, so increasing obs_std makes sense.
OTOH, less observations => more reliant on individal observations, so maybe increasing rate_std would help balance that, too.

rafik.n · January 26, 2024, 5:12pm

I just meant Frankenstein in essence because it’s a combination of different functions from different scripts.

I see what you’re saying. Confidence level per hour decreases considering fewer observations and therefore fewer corrections. Makes sense to have a time factor built in.

Another issue with changing the sampling time on growth rate is that it takes way longer to calculate the mean and standard deviation stats. when it’s 5 seconds it takes 2.5 minutes, when it is 5 mins it takes 2.5 hours. This results in unrepresentative statistics of the “initial” we normalize off of. A quick workaround I’ll do for the next experiment is to run the growth rate at 5-second intervals initially until it initializes then stop it and then go back to 5 minutes. This has a big impact on nOD.

Yes, this is not a pioreactor-vial setup. I just tried messing with rate_std. When I reduce it from 0.2->0.05 it removes small ripples in growth rate but throws off the response rate of estimated OD for sudden changes

CamDavidsonPilon · January 29, 2024, 2:08am

If you want, I’ve added the number of samples needed as a config parameter - as 35 is the default. Upgrade with pio update app -b develop, and add samples_for_od_statistics=35 under [growth_rate_calculating.config] in you config, ex:

[growth_rate_calculating.config]
# these next two parameters control the length and magnitude
# of the variance shift that our Kalman filter performs after a dosing event
ekf_variance_shift_post_dosing_minutes=0.40
ekf_variance_shift_post_dosing_factor=2500
samples_for_od_statistics=35

rafik.n · February 1, 2024, 3:02am

Thank you for the updated feature. Tested it and works as intended.

After updating I noticed some minor differences in od_reading.py file which I have been playing around with. Is there any documentation about differences implemented in the “develop” branch of the code?

One additional software issue I had relates to the ADC offset calculated and implemented via od_reading.py. It started with me troubleshooting why my sensor configuration saturated at the culture density it did. After some debugging, I found that when I disabled the offset feature (setting it to 0) by editing od_reading.py I could get the sensor to detect light far past the mentioned sensor saturation point.

I found this by luck. At first, I kept my light source on during the offset calculation. This may have a similar effect to the blank feature.

I wanted to run the code as intended. But, when I ran the code as intended by allowing it to turn on the light to get the gain and then off to get the offset value, the calculated offset value was larger than the initial OD readings at a low culture density. I found from the debug statements that the calculated offset is a magnitude higher than the initial OD value. This causes the OD results to default to 0 until culture density increases to surpass the offset value. I don’t actually have to wait for culture density to increase to find this out because I have my light source connected to an external variable PWM controller. So I decrease the amount of light reaching the sensor by decreasing the light strength instead of waiting for the culture to get darker.

Does this make sense? I understand the purpose of sensor offset but the way it’s applied here causes these mentioned issues.

CamDavidsonPilon · February 1, 2024, 7:20pm

develop should be nearly identical to the latest version of software. It may have been that your version was a bit more behind - but even then, there hasn’t been significant changes to the od_reading logic any time recently.

BTW, I don’t know how your editing od_reading.py. I’m guessing editing the source files on the Pi - that’s fine if so. An alternative approach is to fork the GitHub - Pioreactor/pioreactor: Hardware and software for accessible, extensible, and scalable bioreactors. Built on Raspberry Pi. repository, and use the pio update app command to specify your forked version: pio update app --repo rafiksgithubname -b master

Anyways, on the the issue. Just so we get some terminology right:

sensor saturation: do you mean the sensor hitting 3.3V (the upper limit), or when the culture becomes so dense, that light starts to not make it though and you see a drop in OD.

the calculated offset value was larger than the initial OD readings at a low culture density. I found from the debug statements that the calculated offset is a magnitude higher than the initial OD value

that’s surprising! I can’t think of why that might be. If I understand correctly:

your light source is off during offset calculation,
your light source is on for the rest of normal OD readings

Maybe you found this, but running OD reading with the following:

DEBUG=1 pio run od_reading

produces granular signal data. Can you share the first 10s or so of logs after starting od_reading?

rafik.n · February 2, 2024, 5:19am

Yeah, I was just ssh-ing into the Python code and sudo saving changes. This approach would make it a lot easier. I will do that.

Well I wasn’t sure what was causing it when it first happened. The culture is visibly very dense and I didnt have the voltage readout of the 90 degree PD because I was using REF so I only see the resulting OD from dividing them.

Yes, you understand correctly. These were the conditions when the calculated offset value was larger than the initial OD readings at a low culture density.

Should we not expect a larger voltage value seeing as the offset is calculated at 0 light condition which is similar to approaching a very high OD (at least for the 90 PD)?

The test is being run with channel 1 REF OD just looking at light source and channel 2 OD has a very dense culture between it and light source. Here are the debug logs for the 2 test conditions.
1. Code running as intended (light off during offset calculation)

2024-02-01T23:54:37-0500 DEBUG  [ir_led_ref] Using PD channel 1 as IR LED reference.
2024-02-01T23:54:37-0500 DEBUG  [calibration_transformer] No calibration available for channel 2, angle 90, skipping.
2024-02-01T23:54:37-0500 DEBUG  [od_reading] Init.
2024-02-01T23:54:38-0500 WARNING [od_reading] The value for the IR LED, 100.0%, is very high. We suggest a value 90% or less to avoid damaging the LED.
2024-02-01T23:54:38-0500 DEBUG  [od_reading] Starting od_reading with PD channels {'2': '90'}, with IR LED intensity 100.0% from channel A.
2024-02-01T23:54:41-0500 DEBUG  [adc_reader] Using ADC class Pico_ADC.
2024-02-01T23:54:41-0500 DEBUG  [adc_reader] ADC ready to read from PD channels 2, 1, with gain 1.
2024-02-01T23:54:46-0500 DEBUG  [adc_reader] AC hz estimate: 60.0
2024-02-01T23:54:46-0500 DEBUG  [adc_reader] timestamps={'2': [4.890599666396156e-05, 0.02764456099976087, 0.05594922099407995, 0.08386023799539544, 0.11246583399770316, 0.14059143299527932, 0.1682427639971138, 0.19665804899705108, 0.2246074509967002, 0.2533061189969885, 0.2815366649956559, 0.30931940199661767, 0.3378260399986175, 0.3658659619977698, 0.39344896099646576, 0.42180674599512713, 0.4496868770002038, 0.4783224729981157, 0.5064966129939421, 0.5342114329978358, 0.5626479159982409, 0.5906183069964754, 0.6193735380002181, 0.6476465829982772, 0.6754541639966192, 0.7040202279968071, 0.7321233269976801, 0.7597524189986871, 0.7881212979991687, 0.8160710639931494, 0.844745669994154, 0.8729459039968788], '1': [0.0013934809976490214, 0.02880637599446345, 0.05713640099565964, 0.0851355419945321, 0.11369197199383052, 0.14169585299532628, 0.16932088199973805, 0.19783215599454707, 0.22571020399482222, 0.25445439299801365, 0.28262749099667417, 0.3105079359957017, 0.33890118899580557, 0.3669487679944723, 0.39451765199919464, 0.42301559199404437, 0.45079989099758677, 0.47947199699410703, 0.5076615539946943, 0.5353776759948232, 0.5637186899984954, 0.5916923629993107, 0.6205807689984795, 0.6487602219931432, 0.6765777499967953, 0.7052088669952354, 0.7332392579992302, 0.7608309539937181, 0.7892389469998307, 0.8172773589976714, 0.8459017039931496, 0.8740285530002438]}
2024-02-01T23:54:46-0500 DEBUG  [adc_reader] aggregated_signals={'2': [1419, 1573, 901, 1199, 1853, 1156, 1070, 1837, 1292, 818, 1644, 1539, 807, 1455, 1618, 842, 1344, 1826, 1038, 1146, 1850, 1182, 864, 1712, 1465, 807, 1503, 1597, 820, 1360, 1821, 1002], '1': [1328, 1428, 1296, 1294, 1443, 1341, 1263, 1430, 1360, 1240, 1371, 1414, 1248, 1335, 1438, 1279, 1319, 1453, 1328, 1290, 1443, 1349, 1245, 1391, 1403, 1248, 1344, 1426, 1273, 1323, 1453, 1314]}
2024-02-01T23:54:47-0500 DEBUG  [adc_reader] timestamps={'2': [5.677100125467405e-05, 0.027667684997140896, 0.05604750099882949, 0.08392893499694765, 0.11251255199749721, 0.1406446609980776, 0.1683021899953019, 0.19669148499815492, 0.22464791799575323, 0.25335106599959545, 0.281580048998876, 0.3094045039979392, 0.33792379799706396, 0.3659771059974446, 0.39356177100125933, 0.42189325399522204, 0.44978156299475813, 0.47841403300117236, 0.5065723919979064, 0.5342785149987321, 0.5627167679995182, 0.590691899000376, 0.6194100979992072, 0.6476786119965254, 0.6754947340014041, 0.7040808509991621, 0.7321782209983212, 0.7598298640004941, 0.7882052529967041, 0.8161202799965395, 0.8447934279975016, 0.8730031929953839], '1': [0.002031288997386582, 0.028929916996276006, 0.05734952399507165, 0.08518481200007955, 0.11365160699642729, 0.14182975799485575, 0.16938285999640357, 0.19777324899769155, 0.22585218099993654, 0.25454991199512733, 0.28268332299921894, 0.31066376699891407, 0.33915540499583585, 0.36706480699649546, 0.39465108700096607, 0.42299772599653807, 0.45097743999940576, 0.4795378799972241, 0.5076756139969802, 0.5354480899986811, 0.5638062919970253, 0.5917699130004621, 0.620495454997581, 0.6488330840002163, 0.6766116539947689, 0.7052836029979517, 0.7333251919990289, 0.7610321479951381, 0.7892863919987576, 0.817231314998935, 0.8460059200006071, 0.8741328210016945]}
2024-02-01T23:54:47-0500 DEBUG  [adc_reader] aggregated_signals={'2': [1091, 1811, 1059, 1104, 1837, 1290, 991, 1792, 1413, 816, 1537, 1615, 848, 1329, 1715, 912, 1232, 1840, 1152, 1064, 1826, 1299, 832, 1616, 1551, 822, 1378, 1685, 873, 1282, 1837, 1118], '1': [1276, 1451, 1329, 1281, 1440, 1364, 1249, 1419, 1390, 1248, 1347, 1436, 1280, 1315, 1447, 1296, 1303, 1452, 1341, 1273, 1433, 1367, 1245, 1361, 1415, 1266, 1327, 1442, 1289, 1310, 1448, 1342]}
2024-02-01T23:54:47-0500 DEBUG  [adc_reader] ADC offsets: {'2': 1326, '1': 1348}, and in voltage: {'2': 0.06678571428571428, '1': 0.0678937728937729}
2024-02-01T23:54:50-0500 INFO   [od_reading] Ready.
2024-02-01T23:54:50-0500 DEBUG  [od_reading] od_reading is blocking until disconnected.
2024-02-01T23:54:51-0500 DEBUG  [adc_reader] timestamps={'2': [4.9531001423019916e-05, 0.02771018500061473, 0.056005001002631616, 0.08385560100578004, 0.112422135003726, 0.1405287750021671, 0.1682229180005379, 0.1966374210023787, 0.22473130100115668, 0.2536617910009227, 0.28197139800613513, 0.309828509001818, 0.33844545900501544, 0.36659001600492047, 0.39420869100285927, 0.4225568920010119, 0.4504862419998972, 0.47923412800446386, 0.5073360810056329, 0.5350363180041313, 0.5634945180063369, 0.5915351700023166, 0.620286388999375, 0.6485465180012397, 0.6763648270061822, 0.7049393810011679, 0.7330213860041113, 0.7606626130000222, 0.7890500850044191, 0.8170496950042434, 0.8456797170001664, 0.8738542740029516], '1': [0.0020454550030990504, 0.02887496900075348, 0.05714978500327561, 0.08501288600382395, 0.11350124200544087, 0.14160606000223197, 0.16947265000635525, 0.19778251800016733, 0.22608483400108526, 0.2549642820013105, 0.2832686300025671, 0.3111477720012772, 0.3397878980031237, 0.368072558005224, 0.3954849319998175, 0.42372521800280083, 0.4518133689998649, 0.4804622970041237, 0.5084962300024927, 0.536129123000137, 0.5648250830054167, 0.5927097450039582, 0.6214295590034453, 0.6496786980060278, 0.6775702880040626, 0.7060402070055716, 0.7341062230043462, 0.7618543240023428, 0.7902492440043716, 0.81832958200539, 0.8467743460059864, 0.8750594740049564]}
2024-02-01T23:54:51-0500 DEBUG  [adc_reader] aggregated_signals={'2': [1369, 1319, 1358, 1359, 1438, 1420, 1308, 1472, 1376, 1306, 1370, 1217, 1328, 1350, 1328, 1364, 1361, 1464, 1431, 1295, 1456, 1345, 1303, 1373, 1216, 1329, 1338, 1308, 1360, 1358, 1447, 1425], '1': [981, 1015, 996, 982, 1023, 992, 980, 1008, 975, 992, 990, 993, 993, 980, 1013, 997, 978, 1018, 987, 984, 997, 973, 988, 987, 988, 992, 988, 1012, 995, 983, 1021, 986]}
2024-02-01T23:54:51-0500 DEBUG  [od_reading] IR Reference is 0.0. Is it connected correctly? Is the IR LED working?
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/timing.py", line 147, in _execute_function
    self.function(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 983, in record_from_adc
    od_reading_by_channel = self._read_from_adc_and_transform()
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 1088, in _read_from_adc_and_transform
    return self.calibration_transformer(self.ir_led_reference_tracker(batched_readings))
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 619, in __call__
    return {
           ^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 620, in <dictcomp>
    ch: self.transform(od_signal)
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 629, in transform
    raise ValueError("IR Reference is 0.0. Is it connected correctly? Is the IR LED working?")
ValueError: IR Reference is 0.0. Is it connected correctly? Is the IR LED working?
2024-02-01T23:54:51-0500 ERROR  [od_reading] IR Reference is 0.0. Is it connected correctly? Is the IR LED working?
2024-02-01T23:54:56-0500 DEBUG  [adc_reader] timestamps={'2': [4.713499947683886e-05, 0.027695600998413283, 0.055988385996897705, 0.08382195499871159, 0.1123928109955159, 0.14052356599859195, 0.1681851569956052, 0.1965837219977402, 0.22454197800107067, 0.2533318429996143, 0.281675460995757, 0.3096709559977171, 0.3383085300010862, 0.36684933400101727, 0.3945273829958751, 0.42295245899731526, 0.4509273809962906, 0.47962693400040735, 0.5078013859965722, 0.535504695995769, 0.5639425839981413, 0.5919405789973098, 0.6206763289956143, 0.6489429159992142, 0.6767517459957162, 0.7053028629961773, 0.7333940339958644, 0.7610254169994732, 0.7893960660003358, 0.817308124002011, 0.8459958539970103, 0.8742442639995716], '1': [0.0014066049989196472, 0.028997779998462647, 0.05713754500175128, 0.08494127100129845, 0.11352144999545999, 0.14175246399827302, 0.16932363899832126, 0.19769725699734408, 0.22577222999825608, 0.25466079299803823, 0.2831437319982797, 0.31143422499735607, 0.33962357299606083, 0.36819437699887203, 0.395925446995534, 0.4243956790014636, 0.4522455499973148, 0.4809048419992905, 0.5089588789996924, 0.536657552998804, 0.5650239309979952, 0.5930783840012737, 0.6218158529954962, 0.6501029610008118, 0.6778426759992726, 0.7063909799981047, 0.7345473079985823, 0.7621179099951405, 0.790494861001207, 0.8183963459960069, 0.8472348040013458, 0.8755020159951528]}
2024-02-01T23:54:56-0500 DEBUG  [adc_reader] aggregated_signals={'2': [1338, 1337, 1346, 1358, 1378, 1314, 1447, 1427, 1280, 1444, 1255, 1297, 1336, 1354, 1350, 1359, 1432, 1424, 1281, 1472, 1381, 1287, 1406, 1217, 1297, 1328, 1331, 1343, 1360, 1403, 1412, 1281], '1': [1014, 994, 986, 1015, 998, 976, 1019, 984, 982, 992, 986, 991, 990, 1013, 993, 977, 1018, 981, 985, 1012, 976, 986, 987, 988, 990, 990, 1007, 994, 982, 1022, 992, 979]}
2024-02-01T23:54:56-0500 DEBUG  [od_reading] IR Reference is 0.0. Is it connected correctly? Is the IR LED working?
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/timing.py", line 147, in _execute_function
    self.function(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 983, in record_from_adc
    od_reading_by_channel = self._read_from_adc_and_transform()
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 1088, in _read_from_adc_and_transform
    return self.calibration_transformer(self.ir_led_reference_tracker(batched_readings))
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 619, in __call__
    return {
           ^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 620, in <dictcomp>
    ch: self.transform(od_signal)
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 629, in transform
    raise ValueError("IR Reference is 0.0. Is it connected correctly? Is the IR LED working?")
ValueError: IR Reference is 0.0. Is it connected correctly? Is the IR LED working?
2024-02-01T23:54:56-0500 ERROR  [od_reading] IR Reference is 0.0. Is it connected correctly? Is the IR LED working?```

2. Edited code setting offset to 0

2024-02-02T00:13:22-0500 DEBUG  [ir_led_ref] Using PD channel 1 as IR LED reference.
2024-02-02T00:13:22-0500 DEBUG  [calibration_transformer] No calibration available for channel 2, angle 90, skipping.
2024-02-02T00:13:22-0500 DEBUG  [od_reading] Init.
2024-02-02T00:13:22-0500 WARNING [od_reading] The value for the IR LED, 100.0%, is very high. We suggest a value 90% or less to avoid damaging the LED.
2024-02-02T00:13:22-0500 DEBUG  [od_reading] Starting od_reading with PD channels {'2': '90'}, with IR LED intensity 100.0% from channel A.
2024-02-02T00:13:25-0500 DEBUG  [adc_reader] Using ADC class Pico_ADC.
2024-02-02T00:13:26-0500 DEBUG  [adc_reader] ADC ready to read from PD channels 2, 1, with gain 1.
2024-02-02T00:13:30-0500 DEBUG  [adc_reader] AC hz estimate: 60.0
2024-02-02T00:13:30-0500 DEBUG  [adc_reader] timestamps={'2': [5.0677001127041876e-05, 0.027640685002552345, 0.055923292005900294, 0.08378725700458745, 0.11236449700663798, 0.1404729390051216, 0.16812836300232448, 0.1965605530058383, 0.22450857900548726, 0.2531877460060059, 0.28141207300359383, 0.309173642002861, 0.3377399970049737, 0.365785471003619, 0.39338756200595526, 0.4217233990057139, 0.44958819600287825, 0.47821048900368623, 0.5064199720000033, 0.5341635720033082, 0.5626784180058166, 0.5906799850054085, 0.6194172770046862, 0.6476894150036969, 0.6754923910048092, 0.7041253600036725, 0.732218177006871, 0.7598415180036682, 0.7882466770024621, 0.8161751720035682, 0.8448267350031529, 0.8730216870026197], '1': [0.001398479005729314, 0.028799166000680998, 0.05703140900004655, 0.0850068830040982, 0.11345360400446225, 0.1415548060031142, 0.16926023000269197, 0.1977993980035535, 0.2256220090057468, 0.25427169700560626, 0.2825470130046597, 0.31025045700516785, 0.3390184260060778, 0.3668695260057575, 0.39459536600043066, 0.42281651600205805, 0.4506980840014876, 0.4793854280069354, 0.5079920960051822, 0.5354365840030368, 0.5639478350058198, 0.5919007580014295, 0.6205333620018791, 0.6487799280002946, 0.6765702480042819, 0.7053996220056433, 0.7333150960039347, 0.7609140109998407, 0.7895249490029528, 0.8172817790036788, 0.8459176130054402, 0.8741036060018814]}
2024-02-02T00:13:30-0500 DEBUG  [adc_reader] aggregated_signals={'2': [1309, 913, 1768, 1423, 797, 1580, 1568, 802, 1440, 1796, 959, 1201, 1854, 1155, 1057, 1848, 1248, 836, 1689, 1488, 799, 1518, 1734, 899, 1265, 1855, 1101, 1100, 1841, 1229, 854, 1718], '1': [1373, 1236, 1407, 1385, 1238, 1357, 1424, 1256, 1334, 1452, 1310, 1296, 1447, 1343, 1264, 1439, 1358, 1241, 1360, 1407, 1247, 1346, 1451, 1296, 1307, 1450, 1334, 1279, 1438, 1359, 1242, 1396]}
2024-02-02T00:13:31-0500 DEBUG  [adc_reader] timestamps={'2': [5.385399708757177e-05, 0.02767698700336041, 0.05597745800332632, 0.08389986000111094, 0.11250230899895541, 0.1406428859991138, 0.1682979460019851, 0.19727195399900666, 0.22523107400047593, 0.25391612599923974, 0.28222342100343667, 0.3100099380026222, 0.33854660599899944, 0.36659484000119846, 0.39417797300120583, 0.4225287049994222, 0.4504077219971805, 0.4790240240035928, 0.507250226000906, 0.5350601280006231, 0.5635567959980108, 0.5915425310013234, 0.6202921139993123, 0.6485634709970327, 0.6763673320019734, 0.7049932180016185, 0.7330949410024914, 0.7607345849974081, 0.7891371389996493, 0.8170662069969694, 0.8457254259992624, 0.8739162109995959], '1': [0.0048420480015920475, 0.02894213399849832, 0.05708849200163968, 0.08519469400198432, 0.11378198699821951, 0.14178912700299406, 0.16938377099722857, 0.19851178899989463, 0.22648898199986434, 0.25501304600038566, 0.28351231799751986, 0.3112232630010112, 0.33973519100254634, 0.36767894699733006, 0.39525504899938824, 0.4237153110007057, 0.4515017759986222, 0.48012667299917666, 0.508607559997472, 0.5364765760023147, 0.5648255370033439, 0.5926659600008861, 0.6215132509969408, 0.6496545569971204, 0.6774696680004126, 0.7062868029970559, 0.7343709740016493, 0.7619903570011957, 0.79028931799985, 0.8183461980006541, 0.8468253660030314, 0.8750030779992812]}
2024-02-02T00:13:31-0500 DEBUG  [adc_reader] aggregated_signals={'2': [907, 1665, 835, 1312, 1835, 1032, 1168, 1838, 1295, 837, 1630, 1532, 810, 1464, 1616, 832, 1365, 1814, 1048, 1105, 1842, 1201, 849, 1712, 1463, 805, 1466, 1620, 825, 1344, 1821, 1007], '1': [1243, 1441, 1277, 1313, 1454, 1325, 1295, 1433, 1363, 1240, 1364, 1412, 1252, 1342, 1436, 1278, 1324, 1453, 1328, 1277, 1441, 1347, 1246, 1387, 1404, 1248, 1333, 1440, 1274, 1315, 1455, 1317]}
2024-02-02T00:13:31-0500 DEBUG  [adc_reader] ADC offsets: {'2': 0, '1': 0}, and in voltage: {'2': 0.0, '1': 0.0}
2024-02-02T00:13:35-0500 INFO   [od_reading] Ready.
2024-02-02T00:13:35-0500 DEBUG  [od_reading] od_reading is blocking until disconnected.
2024-02-02T00:13:36-0500 DEBUG  [adc_reader] timestamps={'2': [4.994700429961085e-05, 0.027613600999757182, 0.055939281002792995, 0.0837846520007588, 0.11237861100380542, 0.1404964800021844, 0.16816601900063688, 0.1965737810023711, 0.22460863000014797, 0.2533413900018786, 0.2815651440032525, 0.3093248380027944, 0.3378549960034434, 0.3659162510011811, 0.3935118320005131, 0.42183495900098933, 0.44972585100185825, 0.4783769450004911, 0.5065124700049637, 0.5342065400036518, 0.5626713329984341, 0.5906593070030794, 0.6193768590019317, 0.6476485290040728, 0.6754814000014449, 0.7040274430037243, 0.7321285409998382, 0.759776465005416, 0.7882224050044897, 0.8161443900025915, 0.844824701998732, 0.8730506960055209], '1': [0.0018975380007759668, 0.028826613001001533, 0.05730062500515487, 0.08491245600453112, 0.11352620700199623, 0.1416804820037214, 0.16941371800203342, 0.19774695000523934, 0.2258786210004473, 0.25459263100492535, 0.2826819070032798, 0.3104165490003652, 0.3389556129986886, 0.36709322100068675, 0.39461359399865614, 0.4229331809983705, 0.4509145920019364, 0.47962146699865116, 0.5075910040031886, 0.5352980419993401, 0.5638686680031242, 0.5917615910002496, 0.6204606009996496, 0.6487943540050765, 0.6766949330049101, 0.705115664000914, 0.7332685330038657, 0.7609475510034827, 0.7894878639999661, 0.8172983920012484, 0.8460004750013468, 0.8742823010034044]}
2024-02-02T00:13:36-0500 DEBUG  [adc_reader] aggregated_signals={'2': [1216, 1312, 1360, 1216, 1329, 1330, 1314, 1367, 1361, 1465, 1437, 1279, 1446, 1313, 1287, 1427, 1271, 1311, 1345, 1215, 1341, 1332, 1417, 1415, 1305, 1464, 1376, 1284, 1435, 1293, 1296, 1344], '1': [992, 991, 987, 989, 992, 989, 1007, 1002, 978, 1014, 980, 980, 993, 970, 986, 990, 973, 992, 988, 993, 992, 986, 1012, 990, 977, 996, 971, 986, 992, 970, 990, 989]}
2024-02-02T00:13:41-0500 DEBUG  [adc_reader] timestamps={'2': [5.3385003411676735e-05, 0.02769167400401784, 0.05597860400303034, 0.08380928700353252, 0.11240225700021256, 0.1405272089978098, 0.16824424699734664, 0.19668633300170768, 0.22465243199985707, 0.2534061819969793, 0.28164014300273266, 0.30944629599980544, 0.33796228699793573, 0.3660260929973447, 0.3936086539979442, 0.4219448020012351, 0.4498207979995641, 0.4784302780026337, 0.5065959069979726, 0.5343042989989044, 0.562748989003012, 0.5907255560014164, 0.6195920660029515, 0.647858319003717, 0.6756671800030745, 0.7042263990006177, 0.7323539040007745, 0.7599955260011484, 0.7883861009977409, 0.8163007940020179, 0.8449833470003796, 0.8731952250018367], '1': [0.0013823859990225174, 0.029052600999420974, 0.05712104400299722, 0.0849205289996462, 0.11358178000227781, 0.14170975299930433, 0.16963652899721637, 0.19794476099923486, 0.2258446109990473, 0.2546812240034342, 0.28274591600347776, 0.3106481619979604, 0.33913769500213675, 0.367137127002934, 0.3946857810005895, 0.4231019290018594, 0.45091552899975795, 0.47956287400302244, 0.5077218360020197, 0.5354759050023858, 0.5638287199981278, 0.5918056529990281, 0.6207796090020565, 0.6489494569977978, 0.6768006089987466, 0.7053797240005224, 0.7335765509997145, 0.7611085910029942, 0.789576508999744, 0.8174195879983017, 0.8462008900023648, 0.8743124570028158]}
2024-02-02T00:13:41-0500 DEBUG  [adc_reader] aggregated_signals={'2': [1295, 1447, 1303, 1297, 1374, 1211, 1316, 1332, 1298, 1375, 1348, 1432, 1424, 1272, 1468, 1407, 1289, 1429, 1250, 1297, 1359, 1216, 1351, 1358, 1411, 1421, 1295, 1455, 1423, 1279, 1424, 1250], '1': [984, 991, 969, 986, 981, 990, 992, 987, 1004, 993, 976, 1017, 977, 979, 1006, 971, 986, 991, 977, 989, 979, 992, 992, 980, 1017, 992, 979, 1014, 975, 980, 992, 974]}
2024-02-02T00:13:46-0500 DEBUG  [adc_reader] timestamps={'2': [5.7604003814049065e-05, 0.027711310001905076, 0.056060269998852164, 0.08390782900096383, 0.1124791840047692, 0.14062226100213593, 0.16829174799931934, 0.19669888500357047, 0.2246319640034926, 0.25331217200437095, 0.28160550800384954, 0.3093704630009597, 0.3378981729983934, 0.3659678129988606, 0.39355870600411436, 0.4218823550036177, 0.44975256900215754, 0.47837402800359996, 0.5065279900009045, 0.5342371119986637, 0.5627009680029005, 0.5906864420030615, 0.6194045150041347, 0.6476612370024668, 0.6754771810010425, 0.7040216090026661, 0.7321746860034182, 0.7598105270008091, 0.7882112060033251, 0.8161306910042185, 0.8447817850028514, 0.8729999659990426], '1': [0.0014238439980545081, 0.029034842002147343, 0.057361928003956564, 0.08504641400213586, 0.11357615500310203, 0.14180730400403263, 0.16948293700261274, 0.1977889289992163, 0.22571117500046967, 0.2544110700036981, 0.28290211399871623, 0.3104537370018079, 0.3390627999979188, 0.3671630649987492, 0.3946440630024881, 0.42301177400076995, 0.4508379260005313, 0.4795437069988111, 0.5076033470031689, 0.5353759060017182, 0.5639179909994709, 0.5917781530006323, 0.6204966420045821, 0.6487373749987455, 0.6766601929994067, 0.705097329999262, 0.7333146259989007, 0.7609569769992959, 0.7893804690029356, 0.8172216730017681, 0.8458675590009079, 0.8742312070025946]}
2024-02-02T00:13:46-0500 DEBUG  [adc_reader] aggregated_signals={'2': [1377, 1354, 1446, 1409, 1280, 1467, 1422, 1290, 1434, 1237, 1313, 1355, 1312, 1352, 1334, 1388, 1360, 1323, 1459, 1422, 1279, 1458, 1280, 1299, 1383, 1226, 1328, 1327, 1351, 1358, 1344, 1456], '1': [992, 976, 1014, 992, 976, 998, 976, 986, 992, 978, 991, 983, 1001, 993, 985, 1019, 994, 979, 1010, 971, 980, 994, 971, 990, 982, 993, 992, 986, 1003, 992, 975, 1011]}
2024-02-02T00:13:51-0500 DEBUG  [adc_reader] timestamps={'2': [5.406199488788843e-05, 0.027743599996028934, 0.05613896699651377, 0.0840286609964096, 0.11271470199426403, 0.14088694599922746, 0.1685422139998991, 0.19696132999524707, 0.2248970129949157, 0.25357201299630105, 0.2818618079982116, 0.3096315019938629, 0.33815134699398186, 0.36620343499816954, 0.3938104219996603, 0.4221485499947448, 0.45001652499922784, 0.47862511899438687, 0.506793716995162, 0.5345056509977439, 0.5629440899938345, 0.5909425849968102, 0.6196974799968302, 0.6479705039964756, 0.6757865519975894, 0.7043623339995975, 0.7324640579972765, 0.7600946379970992, 0.7885063590001664, 0.8164277709947783, 0.845080479994067, 0.8732798579949304], '1': [0.0013682709977729246, 0.029037184998742305, 0.057665205997182056, 0.08533656799409073, 0.11404760900040856, 0.1421482909936458, 0.16964100799668813, 0.19816486299532698, 0.2259778909938177, 0.254656952994992, 0.28312940299656475, 0.3107661289977841, 0.3392805059993407, 0.3673106669957633, 0.394985569997516, 0.4233435930000269, 0.45109641299495706, 0.4797117259950028, 0.5079755829938222, 0.5356319969941978, 0.5640306969944504, 0.5921226279970142, 0.6209209089938668, 0.6490702359951683, 0.6769119089949527, 0.7055711799985147, 0.7335852989999694, 0.761181713998667, 0.7897345789970132, 0.8175254710004083, 0.846170106997306, 0.8744263599946862]}
2024-02-02T00:13:51-0500 DEBUG  [adc_reader] aggregated_signals={'2': [1366, 1358, 1441, 1393, 1277, 1456, 1425, 1284, 1455, 1248, 1312, 1370, 1277, 1333, 1325, 1358, 1354, 1345, 1456, 1421, 1280, 1454, 1289, 1296, 1404, 1225, 1327, 1341, 1344, 1353, 1358, 1448], '1': [995, 981, 1011, 993, 980, 1002, 976, 980, 993, 975, 992, 984, 996, 993, 984, 1011, 993, 976, 1010, 990, 978, 994, 969, 990, 981, 992, 991, 983, 1003, 992, 978, 1017]}
^C2024-02-02T00:13:53-0500 DEBUG  [od_reading] Exiting caused by signal Interrupt.
2024-02-02T00:13:53-0500 INFO   [od_reading] Disconnected.
2024-02-02T00:13:53-0500 DEBUG  [od_reading] Disconnected successfully from MQTT.

PS:

ignore the 100% IR brightness warning. I am using the signal to just switch on my light. using external light power source
The timestamp might be slightly different than you expect because I increased sleep from 0.1 to 3 to allow my lights time to turn on as their control system takes a second to turn them on

rafik.n · February 10, 2024, 3:33pm

A couple of days of head-scratching later. Seems that I managed to purchase the only photodiode on the market with a maximum reverse bias voltage lower than 3.3V as I didn’t have a full understanding of how these magical components work at the time. Over time the PD degraded and stopped working as expected.

Got some higher quality properly rated PDs from Digikey and problem solved. Using offset as normal now.

My mental model of absorbance and scatter was mixed up here.

Apologies for the confusion.

I am thinking of tackling a mixed OD model that uses scatter and absorbance. Gotta do some research first to see if they can be combined.

CamDavidsonPilon · February 12, 2024, 12:47am

I am thinking of tackling a mixed OD model that uses scatter and absorbance. Gotta do some research first to see if they can be combined.

That’s interesting! We originally used a combination of 90 scatter and 135 scatter, but decided to drop 135 and use the second PD for REF. However, we left some of the abstraction present. For example, the Kalman Filter code (and I think growth-rate-calculating) will work for an arbitrary number of signals. Generally this is called “sensor fusion” if you want something to google.