Export dataset not working - and graphs aren't loading correctly

Software
1

Hello, i am a new user of the Pioreactor setup i run my first experiment and now i am trying to extract the data but i keep getting this message: “Server error occurred. Check UI logs.”

And on the UI logs:
[32m2023-12-21T19:02:36+0100 INFO [export_experiment_data] Starting export of tables: pioreactor_unit_activity_data, od_readings.e[0m Traceback (most recent call last): File “/usr/local/bin/pio”, line 8, in sys.exit(pio()) ^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1157, in call return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/click/core.py”, line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/pioreactor/actions/leader/export_experiment_data.py”, line 165, in click_export_experiment_data export_experiment_data(experiment, output, partition_by_unit, tables) File “/usr/local/lib/python3.11/dist-packages/pioreactor/actions/leader/export_experiment_data.py”, line 73, in export_experiment_data with zipfile.ZipFile(output, mode=“w”, compression=zipfile.ZIP_DEFLATED) as zf, closing( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3.11/zipfile.py”, line 1281, in init self.fp = io.open(file, filemode) ^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: ‘/var/www/pioreactorui/static/exports/export_Demo experiment_20231221190235.zip’

Not sure if related: my growth curve looked fantastic when i checked 12h in, but after i had navigated to a different tab of the UI and then returned to the Overview it was not there anymore, only a few OD measurements were shown. And then again a few hours later the datapoints collected after that incident looked nice and smooth and when i refreshed the page they were gone.

Any idea what’s the problem?

2

Hi @Eleni_M!

I think the export problem your experiencing is a previous bug in the software. It’s an easy fix. Can you try this fix: New Pioreactor release: 23.11.18 - #2 by CamDavidsonPilon and let me know if it doesn’t work?

For your second point: I don’t believe it’s related to the bug above. Don’t be alarmed though: the data is safe. What I think is happening is an weird edge-case with how we populate the graphs. As a summary: when you open the overview, we populate the graph from the database, and while the page is open, we populate from a real-time service. If you switch tabs, the graphs should still populate (might be browser-specific), but something funky might be going on that causes data to not be populated. I’ll investigate more.

If you refresh and wait a few seconds, do the graphs fully populate? How long has your experiment been running in total?

3

Hi Cam,
Thanks for the coming back with help so fast. The export issue is solved!!
For the second part no refreshing makes it worse. The graph is populated with the the new measurements as long as i’m on this tab. When i refresh the new measurements are gone and i can only see 3 measurements. Normally I am measuring OD every 20min.
I am not too worried about this issue, its just nice to see how the experiment goes in real time.

anydesk00000
anydesk000001920×1080 240 KB

4

I think this is a consequence of two things: a moderately long experiment, and the 20m gap between measurements. Our charts assume a short-ish duration (5s is the default). So when an experiment has been going on for many hours, it’s normally inefficient to display every data point - so we downsample the data (i.e. retrieve every Nth data point) as a function of how long the experiment has been going on. In your case, downsampling isn’t required, ever.

There’s an easy fix to remove downsampling. SSH back into your leader Pioreactor. We’ll make changes to the charts by editing some chart files:

nano /var/www/pioreactorui/contrib/charts/01_implied_growth_rate.yaml

This opens up the growth-rate chart. To this file, add the following:

downsample: false

So it should look something like:

---
chart_key: implied_growth_rate
data_source: growth_rates # SQL table
title: Implied growth rate
mqtt_topic: growth_rate_calculating/growth_rate
source: app
payload_key: growth_rate
y_axis_label: Growth rate, h ^a
interpolation: stepAfter
y_axis_domain: [-0.02, 0.1]
lookback: 100000
fixed_decimals: 2
down_sample: false

Exit with ctrl-x, and save when prompted. Try refreshing the UI (there’s a cache, so it may take ~30 seconds)

You can do the same to other charts, nOD and OD respectively:

nano /var/www/pioreactorui/contrib/charts/04_normalized_od.yaml

nano /var/www/pioreactorui/contrib/charts/05_od.yaml

If this doesn’t work, let me know, as it may imply something else is going on!

5

Amazing! I just tried it and it solved the problem! Thank you :slight_smile: