Pumps unresponsive in UI and command line

Software
1

I have been having issues with my pumps not working similarly to Cycle media in dosing does not work. I recently updated my cluster to 24.10.29 as the changelog made it seem like there were relevant changes to mitigate this bug, but I still am unable to turn my pumps on through the UI (clicking the button doesn’t lead to anything happening in the logs either). Additionally, when I try running the pumps from the command line with pio run add_media --continuously, I get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/pio", line 8, in <module>
    sys.exit(pio())
             ^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/pump.py", line 576, in click_add_media
    return add_media(
           ^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/actions/pump.py", line 268, in _pump_action
    with utils.managed_lifecycle(
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/__init__.py", line 204, in __enter__
    with JobManager() as jm:
         ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/__init__.py", line 548, in __init__
    self._create_tables()
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/utils/__init__.py", line 582, in _create_tables
    self.cursor.executescript(create_table_query)
sqlite3.OperationalError: no such column: job_name

So it seems like there may be a breaking change with sqlite3 with something new/changed in this most recent update. Any ideas?

2

Ah, hm, maybe there is a bug with the update, I’ll investigate. Two easy solutions:

  1. Reboot the cluster, or
  2. On each machine, run
sqlite3 /tmp/local_intermittent_pioreactor_metadata.sqlite

opens up a database shell, and run:

ALTER TABLE pio_job_metadata RENAME COLUMN name TO job_name;

and

ALTER TABLE pio_job_metadata ADD COLUMN is_long_running_job INTEGER NOT NULL DEFAULT 0;
3

Hi @CamDavidsonPilon , happy new year!

We have run into this problem too when starting jobs which I believe calls the same util function. The fix of rebooting the cluster works. We were wondering whether there was a more permanent bug fix planned? Low priority given there’s a workaround but just wanted to check whether it was still on your radar. We’re on version 24.12.10.

Many thanks for your help,

Vicky

4

Does that not permanently work? If your problem is the same as above, then a reboot (including the leader) should have solved it completely.

However, it sounds like you’re still experiencing a problem? Can you describe it in more detail?

5

Hi Cameron, thanks for getting back to me. Sorry for not being clearer. Rebooting the cluster worked until we started another experiment, at which point we encountered the error again. Rebooting the cluster at this point got rid of the error. This made me think that one has to do this every time we begin a new experiment… However, the column name in the tmp database seems to be correct and we haven’t experienced this issue starting jobs after this. Will let you know if we have any trouble again but will otherwise close this. Thank you!

Screenshot 2025-01-14 at 08.58.36

1 Like