Suggestion: Projects section and export summary

Would it be possible to introduce “Project” section in the GUI to encapsulate the connected experiments under one folder?

If the option above is possible/supported, it would be also great to include:

  1. “researcher name” in the meta data of the introduction section.
  2. “summary csv” which refers export ID to experiment name

In the further future, “Projects” may also be useful to develop a visual schematics to show the experimental timeline from the beginning to the final experiment, and thus, give bigger picture of the project?

Hi @sharknaro, this is an interesting suggestion - thank you for sharing! It won’t be a quick addition, however, as lots of UI elements would need to be updated.

Big +1 for custom fields for the experiment metadata, then people can add ‘project’ as a field and then sort/group/filter with a view of that sql table somewhere in the UI. personally I think the ui/metadata is already a bit too opinionated about the data model for the way people run experiments so always excited to have things more modular (e.g. single strain). Would prefer that over a new pattern like ‘Projects’ that people may or may not use

1 Like

Lots to think about, thanks folks!

1 Like

I do not know if there is large demand for the following below, but some of our colleagues also stated the following challenges:

  1. Difficulty to relate export_id with the experiments (in our case, where large export zip are archieved over certain periods; finding the right experiment based on export_id is really challenging).

  2. Dosing automation events not stateing the start/ending of the automation but rather a log for the completion of the cycle (some of the group members would like to use the doising initiation and ending for further automation. This can be done by using the dosing events, but requires more work to determine the initil and final dilution event. Hence, they have been asking if the automation evens can state the initial and final dosing log).

Can you provide an example of what you would like to see?

Yes;

Example 1:
Have DilutionStart, DilutionEnd, DilutionEvent log in the Dosing automation events - i.e. individual log for Latest OD > Target OD, Latest OD < Target OD, Cycle summary (aka DilutionEvent), respectively.

Example 2:
Include timestamp, elapsed time for dilution start and dilution end in the DilutionEvent of the Dosing automation event

The idea is to have start and end datapoints that can be used to separate the growth phases into sections on the OD measurements.

Does this makes sense?

1 Like

Yes, we can do this for the next release. Want to try today? Add this to your ~/.pioreactor/plugins folder (for each worker) - it adds new events for start and stop. (This overwrites the existing built-in turbidostat)

# -*- coding: utf-8 -*-
from __future__ import annotations

from datetime import datetime
from typing import cast
from typing import Optional

from pioreactor import types as pt
from pioreactor.automations import events
from pioreactor.automations.dosing.base import DosingAutomationJob
from pioreactor.config import config
from pioreactor.exc import CalibrationError
from pioreactor.utils import local_persistent_storage
from pioreactor.utils.streaming_calculations import ExponentialMovingAverage


class DilutionStart(events.AutomationEvent):
    pass


class DilutionEnd(events.AutomationEvent):
    pass


class Turbidostat(DosingAutomationJob):
    """
    Turbidostat mode - try to keep cell density constant by dosing whenever the target is surpassed.
    Note: this has a small "duration" param to run the algorithm-check constantly.
    """

    automation_name = "turbidostat"
    published_settings = {
        "exchange_volume_ml": {"datatype": "float", "settable": True, "unit": "mL"},
        "target_normalized_od": {"datatype": "float", "settable": True, "unit": "AU"},
        "target_od": {"datatype": "float", "settable": True, "unit": "OD"},
        "duration": {"datatype": "float", "settable": False, "unit": "min"},
    }
    target_od = None
    target_normalized_od = None

    def __init__(
        self,
        exchange_volume_ml: float | str,
        target_normalized_od: Optional[float | str] = None,
        target_od: Optional[float | str] = None,
        **kwargs,
    ) -> None:
        super().__init__(**kwargs)

        with local_persistent_storage("active_calibrations") as cache:
            if "media_pump" not in cache:
                raise CalibrationError("Media pump calibration must be performed first.")
            elif "waste_pump" not in cache:
                raise CalibrationError("Waste pump calibration must be performed first.")

        if target_normalized_od is not None and target_od is not None:
            raise ValueError("Only provide target nOD or target OD, not both.")
        elif target_normalized_od is None and target_od is None:
            raise ValueError("Provide a target nOD or target OD.")

        if target_normalized_od is not None:
            self.target_normalized_od = float(target_normalized_od)
        elif target_od is not None:
            self.target_od = float(target_od)

        self.exchange_volume_ml = float(exchange_volume_ml)
        self.ema_od = ExponentialMovingAverage(
            config.getfloat("turbidostat.config", "od_smoothing_ema", fallback=0.5)
        )

    def set_duration(self, value: float | None):
        # force duration to always be 0.25 - we want to check often.
        super().set_duration(0.25)

    @property
    def is_targeting_nOD(self) -> bool:
        return self.target_normalized_od is not None

    @property
    def _od_channel(self) -> pt.PdChannel:
        return cast(
            pt.PdChannel,
            config.get("turbidostat.config", "signal_channel", fallback="2"),
        )

    def execute(self) -> Optional[events.DilutionEvent]:
        if self.is_targeting_nOD:
            return self._execute_target_nod()
        else:
            return self._execute_target_od()

    def set_target_normalized_od(self, new_target: float) -> None:
        if not self.is_targeting_nOD:
            self.logger.warning("You are currently targeting OD, and can only change that.")
        else:
            self.target_normalized_od = float(new_target)

    def set_target_od(self, new_target: float) -> None:
        if self.is_targeting_nOD:
            self.logger.warning("You are currently targeting nOD, and can only change that.")
        else:
            self.target_od = float(new_target)

    def _execute_target_od(self) -> Optional[events.DilutionEvent]:
        assert self.target_od is not None
        smoothed_od = self.ema_od.update(self.latest_od[self._od_channel])
        if smoothed_od >= self.target_od:
            self.ema_od.clear()  # clear the ema so that we don't cause a second dosing to occur right after.
            latest_od_before_dosing = smoothed_od
            target_od_before_dosing = self.target_od

            data = {
                "latest_od": latest_od_before_dosing,
                "target_od": target_od_before_dosing,
                "exchange_volume_ml": self.exchange_volume_ml,
                "volume_actually_moved_ml": 0,
            }
            self.latest_event = DilutionStart("Starting dilution", data=data)

            results = self.execute_io_action(
                media_ml=self.exchange_volume_ml, waste_ml=self.exchange_volume_ml
            )

            data['volume_actually_moved_ml'] =  results["media_ml"]
            self.latest_event = DilutionEnd("Stopping dilution", data=data)

            return events.DilutionEvent(
                f"Latest OD = {latest_od_before_dosing:.2f} ≥ Target OD = {target_od_before_dosing:.2f}; cycled {results['media_ml']:.2f} mL",
                data
            )
        else:
            return None

    def _execute_target_nod(self) -> Optional[events.DilutionEvent]:
        assert self.target_normalized_od is not None
        if self.latest_normalized_od >= self.target_normalized_od:
            latest_normalized_od_before_dosing = self.latest_normalized_od
            target_normalized_od_before_dosing = self.target_normalized_od

            data = {
                "latest_normalized_od": latest_normalized_od_before_dosing,
                "target_normalized_od": target_normalized_od_before_dosing,
                "exchange_volume_ml": self.exchange_volume_ml,
                "volume_actually_moved_ml": 0,
            }
            self.latest_event = DilutionStart("Starting dilution", data=data)

            results = self.execute_io_action(
                media_ml=self.exchange_volume_ml, waste_ml=self.exchange_volume_ml
            )

            data['volume_actually_moved_ml'] =  results["media_ml"]
            self.latest_event = DilutionEnd("Stopping dilution", data=data)

            return events.DilutionEvent(
                f"Latest Normalized OD = {latest_normalized_od_before_dosing:.2f} ≥ Target  nOD = {target_normalized_od_before_dosing:.2f}; cycled {results['media_ml']:.2f} mL",
                data
            )
        else:
            return None

Thanks for swift response! Already tested it but got stuck with the following error:

mqtt to db streaming Encountered error in saving to DB: Invalid value ‘DilutionStart’ - at $.event_name.

I tried to change the code but did not manage to come up with a solution yet.

I think you just need to restart mqtt-to-db-streaming on the leader

sudo systemctl restart pioreactor_startup_run@mqtt_to_db_streaming.service

Thanks for the hint. I also figured that the code was also not correctly distributed to the workers. At the end, it worked well and I am satisfied with the outcome! I hope others also find this structure useful!