Optical density not being recorded, although it is "Started"

Software User interface
1

Hello!

I saw there was another post from July 2023 asking about a similar issue (link) but it is still not working correctly for our pioreactors after trying what you recommended, so wanted to make a new post, maybe someone has more input now :slight_smile:

So I was running yesterday 4 reactors with no automation, just keeping stirring and constant temperature. 2 of my reactors would not plot the OD data in the UI although in the Settings it said it was “started”. Today I repeated the experiment, creating a new experiment, re starting everything, and the same happened for the same 2 pioreactors. I run a self test for these 2 and one of them failed at "Photodiode measures near zero signal for clear water”, and the other failed at the same check and also at “Heating is responsive”. Could this be the reason why no OD data is being recorded? Although OD readings are activated in both of them and no error message comes out.

Before starting the experiment, I blanked each reactor with the media un-inoculated. Since the selftest failed at “Photodiode measures near zero signal for clear water” I then tried to erase this blank calibration and start OD readings again with no blank provided, but still no OD data is being recorded.

Then I found the old post where this same issue was mentioned.
When I try “pio run od_reading” it does activate it in the UI but no OD data appears. But after trying some other commands you wrote to see the logs it started to work, just not sure exactly how or why… Then I tried with the other reactor which is still failing and wrote

pio run od_reading

I get this, so it does seem to work and gets switched on in the UI but no data is plotted.

auto-p03 pio run od_reading
auto-p03 pio run od_reading2048Ă—387 257 KB

pio run od_reading dv

If i run

pio run od_reading >/dev/null 2>&1 & disown

then I do not see in the logs anything about that. But if i run it then i stopped it in the UI and re start OD reading again in the UI then, when i run pio logs i see the line below, but still no OD data being recorded.

 [monitor] DEBUG Running `nohup pio run od_reading >/dev/null 2>&1 &` from Monitor job.

I saw that the version of the software was improtant and then realized I had different versions in each pioreactor. However I updated it some days ago directly from the UI and there it says it is updated to the last version, but when i write pio version -v i was getting different versions for each reactor, so i updated them all from the CL.

After having all pioreactors to the last version, i tried again running pio run od_reading and seems like it is still not recording the OD in that one reactor. Will try other stuff, but wanted to post in case someone already had these issues and tried other things.

Thanks!

2

My first suggestion would have been to have all the Pioreactors on the recent version (there was a breaking change in 24.1.26 that could have caused this issue if your leader was >= 24.1.26 but your workers were < 24.1.26).

You may need to restart your leader for some changes to take effect. Start with that.

Next, try to see if data is being produced from the workers. Start OD reading in all the Pioreactors (stop and start if it’s already running), and try this on the command line on the leader:

pio mqtt -t "pioreactor/+/+/od_reading/od2"

Do you see data from each Pioreactor in your cluster?

3

Hello!

Yes now the 4 reactors appear, thanks!

So I updated the software version from the UI directly before, but then it seemed like not all of them had the same version. Is updating it directly from the button in the UI only updating the version of the UI, or the whole software? Is it better to update each piroeactor individually through the CL instead?

4

So I updated the software version from the UI directly before, but then it seemed like not all of them had the same version. Is updating it directly from the button in the UI only updating the version of the UI, or the whole software? Is it better to update each piroeactor individually through the CL instead?

Ah, I think I know what happened: the update-from-the-ui button only updates you to the next release per pioreactor, not the most recent. Due to this behaviour, it’s possible for your cluster to be in a state where different pioreactors have different versions.

The reason for this behaviour is a bit out-dated, and I may remove this behaviour soon.

At this point, I am more often recommending the update-via-archive approach: Updating the Pioreactor software | Pioreactor Docs

1 Like
5

Hi Cameron,

I think I am having a similar issue here, and as suggested I updated all of the Pioreactor workers to the latest release version. If I go to the command line on the leader and type

pio mqtt -t "pioreactor/+/+/od_reading/od2"

I can see that the data from each worker. However, the OD measurements appear only when I keep the website on the screen. If I go out of overview or refresh the page, all the data points disappear.

Do you have an idea about what can be the cause?

In the meantime, I will try to see if the data points are recorded to export and visualize

EDIT: The export zip file does not include any registered data points

6

The issue seems to be fixed!

I think the problem possibly caused by me setting up the leader i2c off in the config.txt! i.e.,

dtparam=i2c_arm=off

I turned the setting back on in the leader and rebooted the leader. I also updated the app and ui through leader command line, so perhaps that can be also the reason?

The reason for turning i2c off is that, I dedicated one raspberry pi as leader of the cluster, and hence, did not attach raspberry pi HAT on it, since we are not going to connect anything to the leader. However, I kept receiving an warning message that the HAT is missing the leader raspberry pi. Hence, I decided to set the i2c off, thus causing data not to be registered.

However, since I posted this, perhaps I can ask:

  1. What is the recommended leader setup (with/without HAT)?
  2. Does it matter if the leader raspberry pi has HAT? if no, how can we turn of the warning message for i2c missing HAT of the leader?
7

Ah, I should share that we (almost) have dedicated images for just this usecase (leader-only, no worker software): Add leader-only to custopize.yml · Issue #12 · Pioreactor/CustoPiZer · GitHub

Next release, I’ll turn this on.

For now, you can remove the leader’s name from the [cluster.inventory] section in the config.ini (and you may need to reboot).

1 Like
9

Hi again @CamDavidsonPilon,

So I am having the same issue. I recently updated a leader only local access point to the most recent version, and I confirmed that both the workers and leader are up to date with the version.

However, the readings illustrated in the UI disappear once I change the screen. I tried to update everything and I fond this problem:

pioreactor@L02:~ $ pio update firmware
2024-06-10T11:16:55+0200 DEBUG  [update_firmware] sudo wget -O /usr/local/bin/main.elf https://github.com/Pioreactor/pico-build/releases/download/0.2/main.elf
2024-06-10T11:16:56+0200 DEBUG  [update_firmware] sudo bash /usr/local/bin/load_rp2040.sh
2024-06-10T11:16:56+0200 DEBUG  [update_firmware] bash: /usr/local/bin/load_rp2040.sh: No such file or directory

2024-06-10T11:16:56+0200 ERROR  [update_firmware] Update failed. See logs.
Aborted!

Do you think this can be the issue for the data points not being recorded?

10

Leader-only images have this bug currently (fixed in the next release): you’ll have to use the manual fix here: Leader only image bug - HardwarePWM not found - #3 by CamDavidsonPilon

I suspect the next release will come early this week with the fix (and other changes)

pio update firmware

I don’t expect this to work, since the firmware lives on the HAT (plus you need the internet to pull) - what you probably are interested in is pio update app or pio update ui. However, if you want to the latest development versions, you can use pio update app -b develop and pio update ui -b develop` (both these require internet connection).

11

Thanks for the answer @CamDavidsonPilon! Though, I am not clear if running the firmware manual update will solve the issue with:

So when you state

Do you refer to the firmware issue or UI data plots disappearance ?

12

UI data plots disappearance

Thats the bug that my linked answer will fix.

The firmware update issue is unrelated. We haven’t released new firmware in months (years?), so there’s no need to try to update firmware.

1 Like