New Pioreactor plugin: pioled_display_plugin

Hello! I’m happy to announce a new plugin + hardware attachment for your Pioreactor: the pioled_display_plugin!

You can attach an Adafruit PiOLED display to your Pioreactor and have your Pioreactor’s important stats available at a glance:

Note you’ll also need a Right Angle Female Header.

And install our plugin, available in your UI:

2 Likes

Thanks for the update to the plugin, with my suggestion to report back the pioreactor’s IP address. Can I add the init.py from the plugin to rc.local, so that it runs at startup? Do I need to tweak that file, so that it defaults “on” if I can load it that way?

I can’t connect to the pioreactor using the hostname on our college wifi, so need to use the IP address, which is of course a dynamic address and periodically changes.

Hi @drsingleton!

With the latest update, the display (with the IP) should start automatically when the Pioreactor is powered up. That’s accomplished by these two lines. If you power-cycle your Pioreactor, is the display not starting (maybe wait a minute, too)?

If not, and your able to SSH in, I wonder what the output the following produces:

sudo systemctl status pioreactor_startup_run@pioled_display.service

Thanks, I will try that out tomorrow and report back. When I first played with the PiOLED display, I installed the stats.py from the adafruit HOWTO. When I then added your plugin, I quickly found that they didn’t play well together and ended up starting off with a fresh install to clean everything out.

Oh I see! That displays the IP!

Yea, probably only a single “thing” can be controlling the display, hence why the plugin and stats.py won’t work well together.

I like having the IP displayed. If you find you want to see other information on the display, let me know too!

@CamDavidsonPilon, I verified that it displays the IP on startup. Previously, the pioreactor growth stats were toggled on/off through the UI, however I don’t see that anymore under “Activities” after updating to the current version of the plugin. Can that be toggled from the command line?

Unrelated to this, I am playing with the media pumps I got from you a few weeks ago. Ran the pioreactor as a turbidometer using my phone to connect while on the other side of campus for the day. I switched the target nOD every 30 minutes or so. Works great!

Yea, I removed having to start it from the UI. I thought it was a better user-experience to always have it on, and stays on even between experiments (similar to how normal displays work: always on).

You can still turn it off with

sudo systemctl stop pioreactor_startup_run@pioled_display.service

Is that something you expect to do often?


Ran the pioreactor as a turbidometer using my phone to connect while on the other side of campus for the day.

haha great!

There was a bug: starting a new experiment from the UI would stop the display. I’ve fixed that bug in the latest release of the plugin. You’ll need to uninstall-reinstall to get the latest version!

Nice plugin. I just posted an Issue on Github for it though, simple fix but it seems like the Dec OS image is missing some dev library dependencies

cheers
Eiki

Is there any difference in running the display on the worker image vs the leader? I cannot get the worker to find the hardware and if I have the display plugged in the self tests fail. See logs below. Same issue happens if I switch the displays (using the known working display)

2024-01-04T19:30:31+0000 [stirring] INFO Disconnected.
2024-01-04T19:30:32+0000 [stirring] DEBUG Disconnected successfully from MQTT.
2024-01-04T19:30:33+0000 [self_test] DEBUG test_positive_correlation_between_rpm_and_stirring: ✅
2024-01-04T19:30:33+0000 [self_test] INFO 1 failed test ❌
2024-01-04T19:31:49+0000 [pioled_display] DEBUG Init.
2024-01-04T19:31:51+0000 [pioled_display] ERROR Unable to find PiOLED hardware. Is it attached?
2024-01-04T19:31:52+0000 [pioled_display] INFO Disconnected.
2024-01-04T19:31:52+0000 [pioled_display] DEBUG Disconnected successfully from MQTT.
2024-01-04T19:34:47+0000 [pioled_display] DEBUG Init.
2024-01-04T19:34:47+0000 [pioled_display] ERROR Unable to find PiOLED hardware. Is it attached?
2024-01-04T19:34:48+0000 [pioled_display] INFO Disconnected.
2024-01-04T19:34:48+0000 [pioled_display] DEBUG Disconnected successfully from MQTT.
2024-01-04T19:37:06+0000 [monitor] DEBUG Exiting caused by signal Terminated.
2024-01-04T19:37:06+0000 [monitor] INFO Disconnected.
2024-01-04T19:37:06+0000 [monitor] DEBUG Disconnected successfully from MQTT.
2024-01-04T19:37:22+0000 [systemd] DEBUG load_rp2040.service successful
2024-01-04T19:37:25+0000 [pioled_display] DEBUG Init.
2024-01-04T19:37:25+0000 [monitor] DEBUG Init.
2024-01-04T19:37:25+0000 [monitor] DEBUG Pioreactor software version: 23.12.11
2024-01-04T19:37:25+0000 [monitor] DEBUG Pioreactor HAT version: 1.2
2024-01-04T19:37:28+0000 [monitor] DEBUG Pioreactor firmware version: 0.0
2024-01-04T19:37:28+0000 [monitor] DEBUG Pioreactor HAT serial number: 051d041c-2fc3-42cc-8ed8-b04603e1418a
2024-01-04T19:37:28+0000 [monitor] DEBUG IPv4 address: 10.0.0.157
2024-01-04T19:37:28+0000 [monitor] DEBUG WLAN MAC address: d8:3a:dd:61:05:be
2024-01-04T19:37:29+0000 [pioled_display] ERROR Unable to find PiOLED hardware. Is it attached?
2024-01-04T19:37:29+0000 [pioled_display] INFO Disconnected.
2024-01-04T19:37:30+0000 [monitor] DEBUG Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/adcs.py", line 102, in read_from_channel
    self.i2c.writeto_then_readfrom(
  File "/usr/local/lib/python3.11/dist-packages/busio.py", line 224, in writeto_then_readfrom
    return self._i2c.writeto_then_readfrom(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/adafruit_blinka/microcontroller/generic_linux/i2c.py", line 98, in writeto_then_readfrom
    readin = self._i2c_bus.read_i2c_block_data(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/Adafruit_PureIO/smbus.py", line 264, in read_i2c_block_data
    ioctl(self._device.fileno(), I2C_RDWR, request)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/monitor.py", line 138, in __init__
    self.self_checks()
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/monitor.py", line 214, in self_checks
    self.check_for_power_problems()
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/monitor.py", line 511, in check_for_power_problems
    is_rpi_having_power_probems, voltage = self.rpi_is_having_power_problems()
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/monitor.py", line 500, in rpi_is_having_power_problems
    voltage_read = voltage_in_aux(precision=0.05)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/hardware.py", line 123, in voltage_in_aux
    adc.from_raw_to_voltage(adc.read_from_channel(ADC_CHANNEL_FUNCS["aux"])) / slope,
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/adcs.py", line 107, in read_from_channel
    raise exc.HardwareNotFoundError(
pioreactor.exc.HardwareNotFoundError: Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
2024-01-04T19:37:30+0000 [pioled_display] DEBUG Disconnected successfully from MQTT.
2024-01-04T19:37:33+0000 [monitor] NOTICE pioworker1 is online and ready.
2024-01-04T19:37:36+0000 [monitor] INFO Ready.
2024-01-04T19:37:36+0000 [monitor] DEBUG monitor is blocking until disconnected.
2024-01-05T11:53:24+0000 [monitor] DEBUG Running `nohup pio run self_test >/dev/null 2>&1 &` from monitor job.
2024-01-05T11:53:26+0000 [self_test] INFO Starting self-test. Running 10 tests.
2024-01-05T11:53:26+0000 [self_test] DEBUG test_pioreactor_HAT_present: ✅
2024-01-05T11:53:29+0000 [adc_reader] ERROR The internal ADC is not responding. Exiting.
2024-01-05T11:53:29+0000 [self_test] DEBUG The internal ADC is not responding. Exiting.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 407, in _run
    test(client, logger, unit, experiment_name)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 135, in test_all_positive_correlations_between_pds_and_leds
    ).setup_adc()
      ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 197, in setup_adc
    raise exc.HardwareNotFoundError("The internal ADC is not responding. Exiting.")
pioreactor.exc.HardwareNotFoundError: The internal ADC is not responding. Exiting.
2024-01-05T11:53:29+0000 [self_test] ERROR The internal ADC is not responding. Exiting.
2024-01-05T11:53:30+0000 [self_test] DEBUG test_all_positive_correlations_between_pds_and_leds: ❌
2024-01-05T11:53:30+0000 [self_test] DEBUG Heater PCB is not connected, or i2c is not working.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 407, in _run
    test(client, logger, unit, experiment_name)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 309, in test_detect_heating_pcb
    assert is_heating_pcb_present(), "Heater PCB is not connected, or i2c is not working."
AssertionError: Heater PCB is not connected, or i2c is not working.
2024-01-05T11:53:30+0000 [self_test] ERROR Heater PCB is not connected, or i2c is not working.
2024-01-05T11:53:31+0000 [self_test] DEBUG test_detect_heating_pcb: ❌
2024-01-05T11:53:33+0000 [adc_reader] ERROR The internal ADC is not responding. Exiting.
2024-01-05T11:53:34+0000 [self_test] DEBUG The internal ADC is not responding. Exiting.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 407, in _run
    test(client, logger, unit, experiment_name)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 225, in test_ambient_light_interference
    adc_reader.setup_adc()
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 197, in setup_adc
    raise exc.HardwareNotFoundError("The internal ADC is not responding. Exiting.")
pioreactor.exc.HardwareNotFoundError: The internal ADC is not responding. Exiting.
2024-01-05T11:53:34+0000 [self_test] ERROR The internal ADC is not responding. Exiting.
2024-01-05T11:53:34+0000 [self_test] DEBUG 
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 407, in _run
    test(client, logger, unit, experiment_name)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 315, in test_positive_correlation_between_temperature_and_heating
    assert is_heating_pcb_present()
AssertionError
2024-01-05T11:53:35+0000 [self_test] DEBUG test_ambient_light_interference: ❌
2024-01-05T11:53:35+0000 [self_test] ERROR 
2024-01-05T11:53:35+0000 [self_test] DEBUG test_positive_correlation_between_temperature_and_heating: ❌
2024-01-05T11:53:37+0000 [self_test] DEBUG Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/adcs.py", line 102, in read_from_channel
    self.i2c.writeto_then_readfrom(
  File "/usr/local/lib/python3.11/dist-packages/busio.py", line 224, in writeto_then_readfrom
    return self._i2c.writeto_then_readfrom(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/adafruit_blinka/microcontroller/generic_linux/i2c.py", line 98, in writeto_then_readfrom
    readin = self._i2c_bus.read_i2c_block_data(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/Adafruit_PureIO/smbus.py", line 264, in read_i2c_block_data
    ioctl(self._device.fileno(), I2C_RDWR, request)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 407, in _run
    test(client, logger, unit, experiment_name)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 336, in test_aux_power_is_not_too_high
    assert voltage_in_aux() <= 18.0
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/hardware.py", line 123, in voltage_in_aux
    adc.from_raw_to_voltage(adc.read_from_channel(ADC_CHANNEL_FUNCS["aux"])) / slope,
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/adcs.py", line 107, in read_from_channel
    raise exc.HardwareNotFoundError(
pioreactor.exc.HardwareNotFoundError: Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
2024-01-05T11:53:37+0000 [self_test] ERROR Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
2024-01-05T11:53:38+0000 [self_test] DEBUG test_aux_power_is_not_too_high: ❌
2024-01-05T11:53:38+0000 [adc_reader] ERROR The internal ADC is not responding. Exiting.
2024-01-05T11:53:39+0000 [self_test] DEBUG The internal ADC is not responding. Exiting.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 407, in _run
    test(client, logger, unit, experiment_name)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 252, in test_REF_is_lower_than_0_dot_256_volts
    ).setup_adc()
      ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 197, in setup_adc
    raise exc.HardwareNotFoundError("The internal ADC is not responding. Exiting.")
pioreactor.exc.HardwareNotFoundError: The internal ADC is not responding. Exiting.
2024-01-05T11:53:39+0000 [self_test] ERROR The internal ADC is not responding. Exiting.
2024-01-05T11:53:39+0000 [self_test] DEBUG test_REF_is_lower_than_0_dot_256_volts: ❌
2024-01-05T11:53:40+0000 [stirring] DEBUG Init.
2024-01-05T11:53:42+0000 [stirring] ERROR Heating PCB must be present to measure RPM.
2024-01-05T11:53:43+0000 [stirring] INFO Disconnected.
2024-01-05T11:53:43+0000 [stirring] DEBUG Disconnected successfully from MQTT.
2024-01-05T11:53:44+0000 [self_test] DEBUG Heating PCB must be present to measure RPM.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 407, in _run
    test(client, logger, unit, experiment_name)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 73, in test_REF_is_in_correct_position
    with stirring.start_stirring(
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/stirring.py", line 475, in start_stirring
    stirrer = Stirrer(
              ^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 98, in __call__
    obj = type.__call__(cls, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/stirring.py", line 214, in __init__
    raise exc.HardwareNotFoundError("Heating PCB must be present to measure RPM.")
pioreactor.exc.HardwareNotFoundError: Heating PCB must be present to measure RPM.
2024-01-05T11:53:44+0000 [self_test] ERROR Heating PCB must be present to measure RPM.
2024-01-05T11:53:44+0000 [self_test] DEBUG test_REF_is_in_correct_position: ❌
2024-01-05T11:53:45+0000 [ir_led_ref] DEBUG Using PD channel 1 as IR LED reference.
2024-01-05T11:53:45+0000 [calibration_transformer] DEBUG Not using any calibration.
2024-01-05T11:53:45+0000 [od_reading] DEBUG Init.
2024-01-05T11:53:45+0000 [od_reading] DEBUG Starting od_reading with PD channels {'2': '90'}, with IR LED intensity 50.0% from channel A.
2024-01-05T11:53:46+0000 [led_intensity] DEBUG Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/dacs.py", line 66, in set_intensity_to
    self.i2c.writeto(hardware.DAC, bytes([channel, eight_bit]))
  File "/usr/local/lib/python3.11/dist-packages/busio.py", line 207, in writeto
    return self._i2c.writeto(address, buffer, stop=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/adafruit_blinka/microcontroller/generic_linux/i2c.py", line 60, in writeto
    self._i2c_bus.write_bytes(address, buffer[start:end])
  File "/usr/local/lib/python3.11/dist-packages/Adafruit_PureIO/smbus.py", line 303, in write_bytes
    self._device.write(buf)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/led_intensity.py", line 174, in led_intensity
    dac.set_intensity_to(getattr(dac, channel), intensity)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/dacs.py", line 68, in set_intensity_to
    raise HardwareNotFoundError(
pioreactor.exc.HardwareNotFoundError: Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
2024-01-05T11:53:46+0000 [led_intensity] ERROR Unable to find I²C for LED driver. Is the Pioreactor HAT attached to the Raspberry Pi? Is the firmware loaded?
2024-01-05T11:53:48+0000 [led_intensity] DEBUG Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/dacs.py", line 66, in set_intensity_to
    self.i2c.writeto(hardware.DAC, bytes([channel, eight_bit]))
  File "/usr/local/lib/python3.11/dist-packages/busio.py", line 207, in writeto
    return self._i2c.writeto(address, buffer, stop=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/adafruit_blinka/microcontroller/generic_linux/i2c.py", line 60, in writeto
    self._i2c_bus.write_bytes(address, buffer[start:end])
  File "/usr/local/lib/python3.11/dist-packages/Adafruit_PureIO/smbus.py", line 303, in write_bytes
    self._device.write(buf)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/led_intensity.py", line 174, in led_intensity
    dac.set_intensity_to(getattr(dac, channel), intensity)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/dacs.py", line 68, in set_intensity_to
    raise HardwareNotFoundError(
pioreactor.exc.HardwareNotFoundError: Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
2024-01-05T11:53:48+0000 [led_intensity] ERROR Unable to find I²C for LED driver. Is the Pioreactor HAT attached to the Raspberry Pi? Is the firmware loaded?
2024-01-05T11:53:50+0000 [led_intensity] DEBUG Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/led_intensity.py", line 49, in change_leds_intensities_temporarily
    yield
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 907, in __init__
    self.start_ir_led()
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 1038, in start_ir_led
    raise OSError("IR LED could not be started. Stopping OD reading.")
OSError: IR LED could not be started. Stopping OD reading.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/dacs.py", line 66, in set_intensity_to
    self.i2c.writeto(hardware.DAC, bytes([channel, eight_bit]))
  File "/usr/local/lib/python3.11/dist-packages/busio.py", line 207, in writeto
    return self._i2c.writeto(address, buffer, stop=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/adafruit_blinka/microcontroller/generic_linux/i2c.py", line 60, in writeto
    self._i2c_bus.write_bytes(address, buffer[start:end])
  File "/usr/local/lib/python3.11/dist-packages/Adafruit_PureIO/smbus.py", line 303, in write_bytes
    self._device.write(buf)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/led_intensity.py", line 174, in led_intensity
    dac.set_intensity_to(getattr(dac, channel), intensity)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/dacs.py", line 68, in set_intensity_to
    raise HardwareNotFoundError(
pioreactor.exc.HardwareNotFoundError: Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
2024-01-05T11:53:50+0000 [led_intensity] ERROR Unable to find I²C for LED driver. Is the Pioreactor HAT attached to the Raspberry Pi? Is the firmware loaded?
2024-01-05T11:53:51+0000 [self_test] DEBUG IR LED could not be started. Stopping OD reading.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 407, in _run
    test(client, logger, unit, experiment_name)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 287, in test_PD_is_near_0_volts_for_blank
    with start_od_reading(
         ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 1234, in start_od_reading
    return ODReader(
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/base.py", line 98, in __call__
    obj = type.__call__(cls, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 907, in __init__
    self.start_ir_led()
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/od_reading.py", line 1038, in start_ir_led
    raise OSError("IR LED could not be started. Stopping OD reading.")
OSError: IR LED could not be started. Stopping OD reading.
2024-01-05T11:53:51+0000 [self_test] ERROR IR LED could not be started. Stopping OD reading.
2024-01-05T11:53:52+0000 [self_test] DEBUG test_PD_is_near_0_volts_for_blank: ❌
2024-01-05T11:53:54+0000 [self_test] DEBUG 
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 407, in _run
    test(client, logger, unit, experiment_name)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/self_test.py", line 343, in test_positive_correlation_between_rpm_and_stirring
    assert is_heating_pcb_present()
AssertionError
2024-01-05T11:53:54+0000 [self_test] ERROR 
2024-01-05T11:53:55+0000 [self_test] DEBUG test_positive_correlation_between_rpm_and_stirring: ❌
2024-01-05T11:53:55+0000 [self_test] INFO 9 failed tests ❌
2024-01-05T11:53:57+0000 [monitor] DEBUG Exiting caused by signal Terminated.
2024-01-05T11:53:57+0000 [monitor] INFO Disconnected.
2024-01-05T11:53:57+0000 [monitor] DEBUG Disconnected successfully from MQTT.
2024-01-05T11:54:13+0000 [systemd] DEBUG load_rp2040.service successful
2024-01-05T11:54:16+0000 [pioled_display] DEBUG Init.
2024-01-05T11:54:16+0000 [monitor] DEBUG Init.
2024-01-05T11:54:16+0000 [monitor] DEBUG Pioreactor software version: 23.12.11
2024-01-05T11:54:16+0000 [monitor] DEBUG Pioreactor HAT version: 1.2
2024-01-05T11:54:16+0000 [pioled_display] ERROR Unable to find PiOLED hardware. Is it attached?
2024-01-05T11:54:16+0000 [monitor] DEBUG Pioreactor firmware version: 0.2
2024-01-05T11:54:16+0000 [monitor] DEBUG Pioreactor HAT serial number: 051d041c-2fc3-42cc-8ed8-b04603e1418a
2024-01-05T11:54:16+0000 [monitor] DEBUG IPv4 address: 10.0.0.157
2024-01-05T11:54:16+0000 [monitor] DEBUG WLAN MAC address: d8:3a:dd:61:05:be
2024-01-05T11:54:16+0000 [monitor] DEBUG PWM power supply at ~5.05V.
2024-01-05T11:54:16+0000 [monitor] DEBUG Power status okay.
2024-01-05T11:54:16+0000 [monitor] DEBUG Disk space at 6%.
2024-01-05T11:54:16+0000 [monitor] DEBUG CPU usage at 17%.
2024-01-05T11:54:16+0000 [monitor] DEBUG Memory usage at 25%.
2024-01-05T11:54:16+0000 [monitor] DEBUG CPU temperature at 39 ℃.
2024-01-05T11:54:16+0000 [monitor] DEBUG Heating PCB temperature at 22 ℃.
2024-01-05T11:54:17+0000 [pioled_display] INFO Disconnected.
2024-01-05T11:54:18+0000 [pioled_display] DEBUG Disconnected successfully from MQTT.
2024-01-05T11:54:20+0000 [monitor] NOTICE pioworker1 is online and ready.
2024-01-05T11:54:23+0000 [monitor] INFO Ready.
2024-01-05T11:54:23+0000 [monitor] DEBUG monitor is blocking until disconnected.

Hm, there shouldn’t be any difference between the leader and worker.

Your logs suggest an issue with i2c (a communication protocol that the OLED and our internal sensors uses - they all talk on the same “line”). Can you remove the OLED and try self-test again. If that still fails, try a power-cycle and then a self-test?

Dumb question because I’m getting self test issues on the leader and the worker. Reference photo diode issues for example. The self test ui says “Add a closed vial with water and stirbar into the Pioreactor.”. I have the higher precision temp probes in the vials. These are metal and likely highly reflective. Should I not be testing with them in the vial? Instructions are unclear…

I think I can answer my own dumb question. Every test passes on both leader and worker when I put in a vial with no temp probe in it (only water).

Ok to answer your question finally @CamDavidsonPilon . All tests work when the pioled is not connected. I shutdown and started again with the pioled connected and during the startup I get this error below. One thought, has the pioled been tested on systems with the temp probe addon? Could they be conflicting? FYI I upgraded to the latest 2024 jan release before doing all of this:

2024-01-11T16:56:09+0000 [monitor] DEBUG WLAN MAC address: d8:3a:dd:61:05:be
2024-01-11T16:56:10+0000 [pioled_display] ERROR Unable to find PiOLED hardware. Is it attached?
2024-01-11T16:56:11+0000 [pioled_display] INFO Disconnected.
2024-01-11T16:56:11+0000 [monitor] DEBUG Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/adcs.py", line 102, in read_from_channel
    self.i2c.writeto_then_readfrom(
  File "/usr/local/lib/python3.11/dist-packages/busio.py", line 224, in writeto_then_readfrom
    return self._i2c.writeto_then_readfrom(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/adafruit_blinka/microcontroller/generic_linux/i2c.py", line 98, in writeto_then_readfrom
    readin = self._i2c_bus.read_i2c_block_data(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/Adafruit_PureIO/smbus.py", line 264, in read_i2c_block_data
    ioctl(self._device.fileno(), I2C_RDWR, request)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/monitor.py", line 138, in __init__
    self.self_checks()
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/monitor.py", line 214, in self_checks
    self.check_for_power_problems()
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/monitor.py", line 511, in check_for_power_problems
    is_rpi_having_power_probems, voltage = self.rpi_is_having_power_problems()
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/background_jobs/monitor.py", line 500, in rpi_is_having_power_problems
    voltage_read = voltage_in_aux(precision=0.05)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/hardware.py", line 129, in voltage_in_aux
    adc.from_raw_to_voltage(adc.read_from_channel(ADC_CHANNEL_FUNCS["aux"])) / slope,
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/adcs.py", line 107, in read_from_channel
    raise exc.HardwareNotFoundError(
pioreactor.exc.HardwareNotFoundError: Unable to find i2c channel 48. Is the HAT attached? Is the firmware loaded?
2024-01-11T16:56:12+0000 [pioled_display] DEBUG Disconnected successfully from MQTT.
2024-01-11T16:56:15+0000 [monitor] NOTICE pioworker1 is online and ready.
2024-01-11T16:56:17+0000 [monitor] INFO Ready.
2024-01-11T16:56:17+0000 [monitor] DEBUG monitor is blocking until disconnected.

One thought, has the pioled been tested on systems with the temp probe addon? Could they be conflicting?

Let me try locally. I don’t see a reason why they may conflict however¹

With the OLED plugged in, can you run the following:

i2cdetect -y 1 && bash /usr/local/bin/load_rp2040.sh

And show me the output?

¹ Now that I look at it, the OLED sits on the temp expansion kit’s HAT now, right? That’s interesting. Still should be okay, but I’ll check here.

Don’t bother @CamDavidsonPilon. After upgrading both pioreactors and trying again to swap the pioleds one of the works consistently while the other does not. It seems to be a HW issue with the pioled!

I ran the test anyway but I’m convinced the pioled must be dead on arrival from Adafruit.

pioreactor@pioworker1:~ $ i2cdetect -y 1 && bash /usr/local/bin/load_rp2040.sh
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --
+ set -e
+ export LC_ALL=C
+ LC_ALL=C
+ openocd -f interface/raspberrypi-swd.cfg -f target/rp2040.cfg -c init -c 'reset halt' -c 'load_image /usr/local/bin/main.elf' -c 'resume 0x20000000' -c exit
Open On-Chip Debugger 0.11.0-g228ede4-dirty (2022-10-14-12:25)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
adapter speed: 1000 kHz

Info : Hardware thread awareness created
Info : Hardware thread awareness created
Info : RP2040 Flash Bank Command
Info : BCM2835 GPIO JTAG/SWD bitbang driver
Info : clock speed 1001 kHz
Info : SWD DPIDR 0x0bc12477
Info : SWD DLPIDR 0x00000001
Info : SWD DPIDR 0x0bc12477
Info : SWD DLPIDR 0x10000001
Info : rp2040.core0: hardware has 4 breakpoints, 2 watchpoints
Info : rp2040.core1: hardware has 4 breakpoints, 2 watchpoints
Info : starting gdb server for rp2040.core0 on 3333
Info : Listening on port 3333 for gdb connections
target halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ea msp: 0x20041f00
target halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ea msp: 0x20041f00
9020 bytes written at address 0x20000000
downloaded 9020 bytes in 0.154833s (56.891 KiB/s)

yea this suggests that it has shorted the i2c communications. Ah that’s a shame!