Install plugin from source with dependencies

Hello,

I’d like to install a plugin from source but with dependencies being installed via PyPI. I think perhaps the easiest way to do this is to provide an index-url in the pip3 install:

sudo pip3 install --force-reinstall --no-index "$source" to

sudo pip3 install "$source" --force-reinstall --ignore-installed --index-url https://pypi.org/simple/

Currently the pio plugins install predict-fluorescence --source predict_fluorescence-1.0.0-py3-none-any.whl fails:

2024-11-14T13:59:13+0000 DEBUG  [install_plugin] Installing plugin predict-fluorescence.
2024-11-14T13:59:13+0000 DEBUG  [install_plugin] bash /usr/local/bin/install_pioreactor_plugin.sh predict-fluorescence predict_fluorescence-1.0.0-py3-none-any.whl
2024-11-14T13:59:14+0000 ERROR  [install_plugin] Failed to install plugin predict-fluorescence. See logs.
2024-11-14T13:59:14+0000 DEBUG  [install_plugin] b'Processing ./predict_fluorescence-1.0.0-py3-none-any.whl\nINFO: pip is looking at multiple versions of predict-fluorescence to determine which version is compatible with other requirements. This could take a while.\n'
2024-11-14T13:59:14+0000 DEBUG  [install_plugin] b"+ export LC_ALL=C\n+ LC_ALL=C\n+ plugin_name=predict-fluorescence\n+ source=predict_fluorescence-1.0.0-py3-none-any.whl\n+ clean_plugin_name=predict-fluorescence\n+ clean_plugin_name_with_dashes=predict-fluorescence\n+ clean_plugin_name_with_underscores=predict_fluorescence\n++ python3 -c 'import site; print(site.getsitepackages()[0])'\n+ install_folder=/usr/local/lib/python3.11/dist-packages/predict_fluorescence\n++ crudini --get /home/pioreactor/.pioreactor/config.ini cluster.topology leader_hostname\n+ leader_hostname=lead1\n++ hostname\n+ '[' lead1 = lead1 ']'\n+ am_i_leader=true\n+ '[' -n predict_fluorescence-1.0.0-py3-none-any.whl ']'\n+ sudo pip3 install --force-reinstall --no-index predict_fluorescence-1.0.0-py3-none-any.whl\nERROR: Could not find a version that satisfies the requirement schedule==1.2.2 (from predict-fluorescence) (from versions: none)\nERROR: No matching distribution found for schedule==1.2.2\n"
Traceback (most recent call last):
  File "/usr/local/bin/pio", line 8, in <module>
    sys.exit(pio())
             ^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/plugin_management/install_plugin.py", line 44, in click_install_plugin
    install_plugin(name_of_plugin, source)
  File "/usr/local/lib/python3.11/dist-packages/pioreactor/plugin_management/install_plugin.py", line 33, in install_plugin
    raise BashScriptError(f"Failed to install plugin {name_of_plugin}. See logs.")
pioreactor.exc.BashScriptError: Failed to install plugin predict-fluorescence. See logs.

Happy to raise a PR if that’s more helpful.

Thanks,

Vicky

I think this is the problem =\

It’s been a while since I looked at this bash script. I’ll take a fresh look make improvements, including handling your use case.


Unrelated to the main issue, but the pioreactor software has a similar function to what scheduale provides. See pioreactor.utils.timing.RepeatedTimer - but don’t let this stop you from progressing!

That would be great, thank you! I’ll take a look at the RepeatedTimer - that may be a way to make progress. Hope paternity leave is treating you well!

PS I have recreated the shell script but without the --no-index and without the broadcasting to the cluster (it’s a leader-only background job). The graph appears on the Overview page, the buttons appear on the “Manage Pioreactors” card and the new table appears in the sqlite database but I can’t get the job to run i.e. if I click on the “Start” button on the “Manage Pioreactors” card, it doesn’t start and if I do pio run --help on the command line, it is not one of the jobs that appears. If I run python predict_fluorescence.py in the plugins folder, it runs fine (publishing topics to the mqtt). I have rebooted the leader and the cluster. Do you have any tips?

From the run function, I can see that I need a click command. I have this in my background job:

@click.command(name="predict_fluorescence", help=__plugin_summary__)
def click_predict_fluorescence() -> None:
    """
    Predict fluorescence.
    """

    job = PredictFluorescenceJob(
        unit="pio01", 
        experiment=UNIVERSAL_EXPERIMENT, 
    )
    job.block_until_disconnected()


if __name__ == "__main__":
    click_predict_fluorescence()

NB I’ve put pio01 even though it’s leader only - the leader has the leader-only image and so it didn’t recognise the leader as part of the cluster. The job needs historical data and so queries the sqlite database for all pioreactors and publishes them to separate MQTT topics.

Here is the shell script:


#!/bin/bash

# Check if the package name is provided
if [ -z "$1" ]; then
    echo "Error: Package name not provided."
    echo "Usage: ./install_on_lead.sh <PACKAGE_DIR> <PACKAGE_UNDERSCORE>"
    exit 1
fi

PACKAGE_DIR="$1"
PACKAGE_UNDERSCORE="$2"

# Define the plugin directory
PLUGIN_DIR="/home/pioreactor/.pioreactor/plugins"

# Navigate to the dist directory of the package
cd "$PLUGIN_DIR/$PACKAGE_DIR/dist" || {
    echo "Error: Directory $PLUGIN_DIR/$PACKAGE_DIR/dist does not exist."
    exit 1
}

echo "Changed to directory: $(pwd)"

# Find the wheel file
WHEEL_FILE=$(ls *.whl | tail -n 1)
echo "WHEEL_FILE is $WHEEL_FILE"

if [ -n "$WHEEL_FILE" ]; then
    echo "Installing $WHEEL_FILE"
    pip3 install "$WHEEL_FILE" --force-reinstall --ignore-installed --index-url https://pypi.org/simple/ || true
    echo "Installation complete, proceeding to check for LEADER_ONLY file"

    # Check if LEADER_ONLY file exists
    if [ -f "$PLUGIN_DIR/$PACKAGE_DIR/$PACKAGE_UNDERSCORE/LEADER_ONLY" ]; then
        echo "LEADER_ONLY file exists"

        # Merge new config.ini if it exists
        if [ -f "$PLUGIN_DIR/$PACKAGE_DIR/$PACKAGE_UNDERSCORE/additional_config.ini" ]; then
            echo "Merging additional config"
            crudini --merge /home/pioreactor/.pioreactor/config.ini < "$PLUGIN_DIR/$PACKAGE_DIR/$PACKAGE_UNDERSCORE/additional_config.ini"
        fi

        # Add any new SQL and restart mqtt_to_db job if SQL exists
        if [ -f "$PLUGIN_DIR/$PACKAGE_DIR/$PACKAGE_UNDERSCORE/additional_sql.sql" ]; then
            echo "Adding additional SQL and restarting service"
            sqlite3 "$(crudini --get /home/pioreactor/.pioreactor/config.ini storage database)" < "$PLUGIN_DIR/$PACKAGE_DIR/$PACKAGE_UNDERSCORE/additional_sql.sql"
            sudo systemctl restart pioreactor_startup_run@mqtt_to_db_streaming.service
        fi

        # Merge UI contributions if contrib directory exists
        if [ -d "$PLUGIN_DIR/$PACKAGE_DIR/$PACKAGE_UNDERSCORE/ui/contrib/" ]; then
            echo "Merging UI contributions"
            rsync -a "$PLUGIN_DIR/$PACKAGE_DIR/$PACKAGE_UNDERSCORE/ui/contrib/" /home/pioreactor/.pioreactor/plugins/ui/contrib/
        fi
    else
        echo "LEADER_ONLY file does not exist"
    fi
else
    echo "No .whl file found for installation"
fi

Many thanks in advance!

Hm, if you execute pio run --help, is there an error message at the top of the output? Any errors when a plugin loads are captured and spit out with pio x is run.

Otherwise, that looks okay to me. Can you share the entire script?

Hi Cameron, thanks for taking a look. When running pio run --help, I get the error:

pio_data_daemon plugin load error: No module named 'pio_data_daemon'

And when I run pio plugins list, I get a similar error:

pio_data_daemon plugin load error: No module named 'pio_data_daemon' pio_data_daemon plugin load error: No module named 'pio_data_daemon'

pio_data_daemon is another plugin we’re working on. I’ve deleted the directory from the .pioreactor/plugins directory and tried pio plugins uninstall pio_data_daemon (which errors as it can’t find it) followed by a reboot but it looks like it’s not uninstalling properly… This might be the issue but the other installed plugins show up in the pio run --help as subcommands (and I’m reasonably sure it was a problem before my colleague started working on the other plugin).

I’ll email across the package and the shell script to you too in case there’s anything I’m doing that’s unexpected. Many thanks!

Does your Python plugin code have something like:

import pio_data_daemon
# or from pio_data_daemon import ...

at the top / in it somewhere? This would cause an error on loading the plugin, too.

But yea, please send the Python code over email, and we’ll test it locally.

Great! Thanks - I’ve sent it to hello@pioreactor.com.

Small update:

  • we’ve run pip uninstall pio-data-daemon and the errors I mentioned above have disappeared
  • we tried testing installing a community plugin log2discord to try to work out whether it was something specific with how we’re setting up plugins. We installed it via the UI, rebooted and that one doesn’t appear in the pio plugins list either.

Will keep trying a few things and keep you updated here if we see anything different. Thanks!