Error after update to 25.2.11

TurboBK · February 28, 2025, 2:38pm

Dear Pioreactor Team,

We recently encountered an issue after updating from 24.12.10 → 25.1.21 → 25.2.11 → 25.2.20 using the ZIP file. We are operating in local access point mode due to security regulations preventing internet access.

The update worked well for our leader and 7 workers, but one worker remains on 25.2.11.

After the update:

We can still connect to this worker via SSH. but not with the UI.
However, when we try to run the pio command, it does not respond (no error message, just no output).

Also, when we tried to re-update via UI, it showed server error code 503.

Could you kindly advise on how we should proceed to resolve this issue?

Thank you for your time and support!

Best regards,
Turbo

CamDavidsonPilon · March 1, 2025, 1:05am

Hi @TurboBK,
I’ve seen this before, but I can’t recall our exact solution. Let me try to replicate again and get back to you shortly. Sorry about the troubles!

jb1404 · March 15, 2025, 11:03am

Hi @CamDavidsonPilon,
Any update on this matter?
Thanks

CamDavidsonPilon · March 15, 2025, 1:45pm

@jb1404 sorry! This got lost but thank you for bumping it.

Can you try something? SSH into the worker that’s not working as expected, and run:

python3 -c "import pioreactor"

Any error?

jb1404 · March 17, 2025, 10:35am

Hi! Thanks for your answer.

No error message but no output either:

Captura de pantalla 2025-03-17 a las 11.34.50

jb1404 · March 17, 2025, 10:59am

I tried to download the packages using pip download pioreactor ,they downloaded and then I ran

cp pioreactor-*.whl *.whl huey-2.5.2.tar.gz JSON-log-formatter-0.5.1.tar.gz pioreactor@BIO-PR-W08.local:~

Then,
pip install --no-index --find-links=~ pioreactor-*.whl *.whl huey-2.5.2.tar.gz JSON-log-formatter-0.5.1.tar.gz

and I obtained the error: ERROR: MarkupSafe-2.1.5-cp312-cp312-macosx_10_9_universal2.whl is not a supported wheel on this platform

I am using Python 3.12.7 and MacOS 14.0

CamDavidsonPilon · March 17, 2025, 12:45pm

Hi @jb1404,

Okay, hm, try this instead:

On your Mac, download the Pioreactor version 25.2.20 Python wheel:

wget https://files.pythonhosted.org/packages/59/20/4ccc908331d9c4d3b1575724e0a9098b57062b9df01adfcc2ce62389f149/pioreactor-25.2.20-py3-none-any.whl

Copy it to your worker:

scp pioreactor-*.whl pioreactor@BIO-PR-W08.local:~

Then SSH into the worker and try:

sudo pip install --no-index pioreactor-25.2.20-py3-none-any.whl --force-reinstall --no-deps

Then try pio and see if that works.

jb1404 · March 17, 2025, 1:16pm

Thanks for your answer!
I tried to do so but obtained this error:

jb1404 · March 17, 2025, 1:40pm

I managed to install the software I can access the pio command now in the terminal. I realized that the 25.3.5 was downloaded instead of the 25.2.20.

However, the blue light that should appear before installing the Pioreactor in the UI is not blinking and this error appears in the UI. I already rebooted the worker.

“Unable to complete connection. The following error occurred: Internal server error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application. - see logs for more.”

CamDavidsonPilon · March 17, 2025, 1:51pm

Okay, hm, that sounds like an error on the leader. Can you SSH into your leader, try pio logs and see if there a recent error there?

Are other parts of your UI working, or is it just this that is showing an error?

jb1404 · March 17, 2025, 1:58pm

Everything else in the UI is working fine. The leader is online and responding. This is what pio logs gave as output. The first error is about a plugin that stopped working after updating to 25.2.11.

jb1404 · March 17, 2025, 2:02pm

Also, in the UI I found these errors a couple of days ago, but I don’t think this has to do with the workers connection

CamDavidsonPilon · March 17, 2025, 2:07pm

Got it.

That plugin is deprecated, and 25.2.11 introduced the calibration standards option natively in Pioreactor. You can delete it from the workers with (from leader):
```
pios rm /home/pioreactor/.pioreactor/plugins/od_calibration_from_standards.py
```
Looks like I’m not catching an error well in the backend. Anyways, try this on your leader:
```
pio workers add BIO-PR-W08 -v 1.1 -m pioreactor_20ml
```

Let me know what that returns.

jb1404 · March 17, 2025, 2:30pm

It is working again now!! I am able to see and control the worker from the UI.

Thank you for your quick and valuable help!!

CamDavidsonPilon · March 17, 2025, 2:33pm

Great. I think the issue was that you were inputting into the UI the name BIO-PR-W08.local, when it should have been just BIO-PR-W08 (no .local). Anyways, this is fixed and reported back to the user in the next version.

jb1404 · March 17, 2025, 2:37pm

I really appreciate it! Thanks!