open
flexure
.org

OFEP4: Install extensions using the Command Line Interface

OFEP 4
Author Samuel McDermott & Richard Bowman
Created 01-Mar-2021
Status Approved, not implemented
Approved date 08-May-2024
Requires implementation Yes
Implemented date N/A
Updated dates (post-approval) N/A

1. Introduction

At the moment, downloading and installing extensions is complicated.

From a developer point of view, the current state is frustrating and potentially risky. Many external developers have also complained about the difficulties in deploying their extensions (especially around user permissions and dependency management.)

It is too complicated to expect a standard user to follow the required steps to use an extension.

As we are starting to see more extensions being developed internally and externally, we should aim for a universally simple experience for downloading and installing extensions, with clear documentation for developers, and easy instructions for users.

2. State before this OFEP

Users need to download the files for the extension from the repo, copy the files into the /var/openflexure/extensions/microscope_extensions directory (for which the standard pi user does not have the correct permissions). They then need to manually install any dependencies, and enable any other functionality (e.g. SPI). Then once the application is restarted (either from CLI or by restarting the Pi) the extension can be used.

If the extension has been updated, there is no way to trigger an update apart from by doing this manually. This means if the extensions was cloned from a repository, the repository would need to be downloaded again.

Some extensions are published to pypi, and so can be installed from there, but it is still a requirement to have a file inside that directory. Installing from a Python package has many advantages (proper version and dependency management) but it is still a requirement to have a file in the microscope_extensions directory and it is not clear how to install an extension.

3. Implementation details.

The best approach would be through the GUI. This is the most user friendly way of interacting with the microscope and a long term goal should be that extensions can be added through OpenFlexure Connect.

However, a more immediate goal should use the existing command line interface program. This avoids the need to restart the web server from the web interface, is more secure, and could be implemented much more simply.

3.1 User experience

More advanced users should be familiar with the command line ofm instruction set. A new command should added to this interface to allow users to download an extensions, install any dependencies, and restart the server with a simple command:

sudo ofm install-extension camera-stage-mapping

or

sudo ofm install-extension git+https://gitlab.com/openflexure/microscope-extensions/openflexure-videoplugin.git

This should be able to work with either a python package on PyPI (in which case only the package name, and optionally version, is required) or a Python package in a git repository (in which case the repository address is specified).

If system configuration or installation of additional non-Python dependencies is required, the user should be prompted to do so, ideally just requiring a "yes" for it to happen.

3.2 Developer experience

When developers are developing extensions, they should be required to package their extension such that it is an installable Python module, and encouraged to publish their extension to PyPI. In addition to the Python code, we need the following information:

  • A script to run (optional) to install any dependencies that cannot be installed with pip
  • Any Python dependencies to install (specified in the usual way, and installed by pip or pipenv)
  • The extension class(es) included within the package that should be loaded
  • A post-install script (optional). This would allow interfaces to be enabled (eg. SPI) or other configuration changes to be made.

Rather than reinvent packaging metadata for Python yet again, we propose to use the "entry points" mechanism. This allows the Python package to declare that it contains one or more OFM extensions, and to specify the submodule and class(es) to load.

Pre-install scripts are tricky because we'd need a way to extract the metadata before the package is installed. However, the main motivation for including this is to allow dependencies to be installed with apt. If such dependencies are missing, often the Python module can be installed, but it won't run because its imports raise an error. I therefore propose we handle this a different way: if the module can't load because of unmet non-pip dependencies, it should fail with an ImportError. If there's a script that would remedy the problem (e.g. apt install libmydependencies0-dev), we can include it as an attribute of the Exception object. This mechanism isn't beautiful, but it could be picked up fairly easily by the extension loader, and allow the user to fix extensions that currently can't load.

More elaborate system configuration (e.g. enabling GPIO or similar) could be done with a method of the extension object. I propose we add a method to the BaseExtension class configuration_required() that will return True if system configuration is needed. The extension installer could then query the status of each available extension, and prompt the user to run a configuration script (e.g. extension.configure_extension()). The base implementation should simply return False.

Because package names are not necessarily the same as module names, it's non-trivial to know which package we are currently installing. I therefore suggest that once the package has been installed, we simply check all available extensions; first, running any install scripts that are suggested by modules that can't load, and second running any configuration scripts by modules that claim not to be configured. This also works nicely for upgrading or reinstalling modules, as it will re-run installation/configuration scripts whenever needed, rather than just on the first installation of a package. Similarly, if pipenv is used to manage a project and installs a number of extensions without running ofm install-extension, we could use this mechanism to ensure the relevant scripts are run at a later time.

3.3 The install script

Steps the program should do:

  1. Stop the OFM server application.
  2. Activate the OFM virtual environment
  3. Install the python package. This will be done with pipenv or whatever tool is managing server dependencies, so that dependency resolution happens and we don't accidentally break the server by installing an extension. The package (and version if specified) should be added to the project's Pipfile.
  4. In the future it would be great to be able to verify the hash of any Python depedencies that are specified, but this isn't planned initially.
  5. Attempt to load all available extensions, and prompt for any install scripts suggested by modules that are currently unable to load
  6. Check if any available extensions report configuration is required, and if so prompt the user to run the configuration scripts
  7. Retain the manifest file or its URL, to allow the extension to be reinstalled if needed (this may not be implemented initially)
  8. Restart the server application.

Further command line arguments should be passed to pipenv (this would allow multiple packages to be installed, etc.)

3.4 Template repository

We should create a template repository that developers can copy to make their own repo. This should have a template Python module that could be installed either from PyPI or from the git repository. It could include an example .gitlab-ci.yml to automatically publish the extension to PyPI. It should demonstrate the use of an install script (though most users won't need this) and the configuration methods. This has been started on GitLab.

3.5 Other considerations

LabThings currently handles loading and finding extensions so this would require an upstream change - though actually it could be implemented quite simply in the openflexure microscope server, as we need only pass a list of extension objects to its extension loader. If the proposed mechanism works well, it could easily be adopted by LabThings.

The mechanism as proposed does not include provision for disabling installed extensions. It may be that a Python package that provides an extension is installed, but we don't actually want the extension. A mechanism in the future should be provided to rectify this (for example, the configuration JSON file could contain a list of disabled extensions). This is not required for implementation of this OFEP. Enabling/disabling extensions will be considered as part of the configuration overhaul, and will affect the implementation of this slightly.

4 Alternatives and rejected options

Previous versions of this OFEP suggested the use of a "manifest" file to specify pre/post install scripts, Python dependencies, and code to load the extension from the module. I think the mechanisms described above are cleaner and simpler.