Plugin Development Walk-through¶

In this guide, we’ll walk through developing a custom plugin for PixStor Search.

Introduction
Plugin template
Plugin check tool
Filling in the template
- Handles
- Schema
- Review
Extracting metadata
Enabling the plugin
Perform an ingest
Adding more metadata
Summary
Appendix: Complete Plugin

Introduction ¶

The plugin we’ll develop will be a metadata plugin, which will extract sheet names from Excel spreadsheets

This guide will cover:

writing a plugin

validating the plugin

installing and enabling the plugin

performing an ingest with the plugin

This is a relatively high level, practical guide. See the Plugin Development Overview for a more detailed description of plugin development.

See also Asynchronous Proxy Plugin Walk-Through for a description of developing an asynchronous, proxy-generating plugin.

Plugin template ¶

We’ll start with a plugin template.

Create a new file in the user plugins directory

$ mkdir -p /opt/arcapix/usr/share/apsearch/plugins/available/user
$ vim /opt/arcapix/usr/share/apsearch/plugins/available/user/excel_plugin.py

We’ve put the plugin under available, not enabled - we’re not ready to enable it yet.

Warning

It’s best not to enable the plugin until we’re happy with the schema. Once the schema is registered, it’s hard to change.

The plugin can’t be used in ingests until it is enabled.

Paste the following code into the new file

from arcapix.search.metadata.plugins.base import Plugin, PluginStatus
from arcapix.search.metadata.helpers import Metadata


class SamplePlugin(Plugin):

    def namespace(self):
        """Return the namespace for the metadata for this plugin."""
        return 'sample'

    def is_async(self):
        """Return whether this plugin does any operations asynchronously."""
        return False

    def handles(self, ext=None, mimetype=None):
        """Return whether the plugin can handle a given file, based on extension/mimetype

        :param ext: file extension, including a leading '.'
        :param mimetype: file mime type, e.g. image/png

        :returns: True if the plugin needs to process this file, false otherwise
        """
        return False

    def schema(self):
        """Return the schema for the metadata produced by this plugin.

        All metadata will be validated against this before inserting into the database.
        """
        return []

    def _extract(self, filename):
        """Private worker function to extract metadata from a file.

        :param filename: file to extract data from

        :return: dict of metadata
        """
        raise NotImplementedError("TODO")

    def process(self, id_, file_, fileinfo=None):
        """Extract metadata from a given file and submit an update.

        :param id_: black box identifier which shall be passed to the metadata update functions
        :param file_: full path to the file
        :param fileinfo: dict of information which may have already been gathered and may be of use

        :returns: PluginStatus indicating success/failure
        """
        try:
            data = self._extract(file_)

            if not data:
                # didn't find any metadata
                return PluginStatus.SKIPPED

            if Metadata(id_, self).update(data):
                return PluginStatus.SUCCESS

            return PluginStatus.ERRORED

        except Exception as exc:
            self.logger.error(
                "An exception was raised while processing '%s' (%s): %s",
                file_, id_, exc
            )
            return PluginStatus.FATAL

Plugin check tool ¶

As we develop this plugin, we’ll use the searchctl plugins check tool to check for issues.

Let’s try it now. To specify our plugin, we give it the full path and the plugin (class) name

For brevity, let’s define an environment variable for the plugin path

$ export PLUGIN_PATH=/opt/arcapix/usr/share/apsearch/plugins/available/user/excel_plugin.py

$ searchctl plugins check $PLUGIN_PATH::SamplePlugin

WARNING:arcapix.search.metadata.plugins.validation:plugin: plugin doesn't appear to be installed in plugin path
ERROR:arcapix.search.metadata.plugins.validation:sample: field isn't registered
WARNING:arcapix.search.metadata.plugins.validation:PxS middleware must be restarted after a plugin is installed:
WARNING:arcapix.search.metadata.plugins.validation:    $ systemctl restart apsearch-middleware
WARNING:arcapix.search.metadata.plugins.validation:No test files given. Skipping file-based checks. Some issues may be missed

SamplePlugin :: server :: sample
❌ field isn't registered
plugin
❗ plugin doesn't appear to be installed in plugin path

As we can see, it’s detected that the plugin isn’t enabled and installed. Since we’re not ready to enable the plugin yet, we can ignore this for now.

Extracting metadata ¶

To extract our metadata we’re going to use a 3rd-party library: openpyxl

from openpyxl import load_workbook

class ExcelPlugin(Plugin):

    # ...

    def _extract(self, filename):
        workbook = load_workbook(filename=filename)

        return {'sheets': workbook.sheetnames}

sheets is the field name we defined in the schema. workbook.sheetnames returns a list of strings, so there’s nothing more to do.

Let’s check it

$ searchctl plugins check $PLUGIN_PATH::ExcelPlugin /mmfs1/data/admin/expenses.xlsx
searchctl error: No module named openpyxl

Oops..!

Installing dependencies ¶

Since our plugin relies on a 3rd-party library, we need to install it.

PixStor Search lives in it’s own virtual environment, it doesn’t have access to libraries installed outside of that environment (e.g. using yum or the system python)

So we need to install the dependency inside of Search’s virtual environment

$ /usr/share/arcapix/apsearch/bin/pip install openpyxl

Review ¶

Here’s the complete plugin, updated to include the extract implementation

from openpyxl import load_workbook

from arcapix.search.metadata.plugins.base import Plugin, PluginStatus
from arcapix.search.metadata.helpers import Metadata


EXTENSIONS = ['.xls', '.xlsx']
MIME_TYPES = [
    'application/vnd.ms-excel',
    'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
]


class ExcelPlugin(Plugin):

    def namespace(self):
        """Return the namespace for the metadata for this plugin."""
        return 'excel'

    def is_async(self):
        """Return whether this plugin does any operations asynchronously."""
        return False

    def handles(self, ext=None, mimetype=None):
        """Return whether the plugin can handle a given file, based on extension/mimetype

        :param ext: file extension, including a leading '.'
        :param mimetype: file mime type, e.g. image/png

        :returns: True if the plugin needs to process this file, false otherwise
        """
        return ext in EXTENSIONS or mimetype in MIME_TYPES

    def schema(self):
        """Return the schema for the metadata produced by this plugin.

        All metadata will be validated against this before inserting into the database.
        """
        return [
            {
                "name": "sheets",
                "prompt": "Names of sheets in this workbook",
                "value": {
                    "datatype": "[String]"
                }
            }
        ]

    def _extract(self, filename):
        """Private worker function to extract metadata from a file.

        :param filename: file to extract data from

        :return: dict of metadata
        """
        workbook = load_workbook(filename=filename)

        return {'sheets': workbook.sheetnames}

    def process(self, id_, file_, fileinfo=None):
        """Extract metadata from a given file and submit an update.

        :param id_: black box identifier which shall be passed to the metadata update functions
        :param file_: full path to the file
        :param fileinfo: dict of information which may have already been gathered and may be of use

        :returns: PluginStatus indicating success/failure
        """
        try:
            data = self._extract(file_)

            if not data:
                # didn't find any metadata
                return PluginStatus.SKIPPED

            if Metadata(id_, self).update(data):
                return PluginStatus.SUCCESS

            return PluginStatus.ERRORED

        except Exception as exc:
            self.logger.error(
                "An exception was raised while processing '%s' (%s): %s",
                file_, id_, exc
            )
            return PluginStatus.FATAL

Now that we’re happy that our plugin works, let’s install and enable it.

Perform an ingest ¶

Now we can ingest some files using our plugin.

Let’s try ingesting our test file

$ APLOGLEVEL=info searchctl add-file /mmfs1/data/admin/expenses.xlsx --plugins excel_plugin::ExcelPlugin
...
INFO:arcapix.search.metadata.broker:Starting ingest of '/mmfs1/data/admin/expenses.xlsx' (3927216287062101581)
INFO:arcapix.search.metadata.broker:/mmfs1/data/admin/expenses.xlsx (3927216287062101581) default::DefaultPlugin SUCCESS
INFO:arcapix.search.metadata.broker:/mmfs1/data/admin/expenses.xlsx (3927216287062101581) excel_plugin::ExcelPlugin SUCCESS
NOTIFY:arcapix.search.metadata.broker:Ingested '/mmfs1/data/admin/expenses.xlsx' (3927216287062101581) with 2 plugins - 2 plugins were successful, 0 failed

We’re using the --plugins flag so the ingest only uses the excel plugin. (default::DefaultPlugin is a special, internal plugin, which is always ingested).

Open up the PixStor Search UI, and verify that the excel metadata is present for our test file.

Now we want to ingest any other excel files on the file system

$ searchctl ingest mmfs1 --include '*.xls,*.xlsx' --plugins excel_plugin::ExcelPlugin

Here, we’re using an --include to only match files with the extensions our plugin handles.

If you have scheduled ingests enabled, and search is in ‘rich’ mode, then the plugin should be used automatically next time the ingest is run.

Adding more metadata ¶

Supposing we want extract more metadata with the plugin - let’s say, whether the spreadsheet is read-only - we simply add the new field to the schema, and update the _extract method

def schema(self):
    return [
        {
            "name": "sheets",
            "prompt": "Names of sheets in this workbook",
            "value": {
                "datatype": "[String]"
            }
        },
        {
            "name": "readonly",
            "prompt": "Indicates whether the workbook is read-only",
            "value": {
                "datatype": "Boolean"
            }
        }
    ]

def _extract(self, filename):
    workbook = load_workbook(filename=filename)

    return {
        'sheets': workbook.sheetnames,
        'readonly': workbook.read_only,
    }

Check the new metadata

$ APLOGLEVEL=notify searchctl plugins check excel_plugin::ExcelPlugin /mmfs1/data/admin/expenses.xlsx
...
ERROR:arcapix.search.metadata.plugins.validation:readonly: field isn't registered
WARNING:arcapix.search.metadata.plugins.validation:PxS middleware must be restarted after a plugin is installed:
WARNING:arcapix.search.metadata.plugins.validation:    $ systemctl restart apsearch-middleware
NOTIFY:arcapix.search.metadata.plugins.validation:Checking file "/mmfs1/data/sample_data/expenses.xlsx"
NOTIFY:arcapix.search.metadata.plugins.validation:File "/mmfs1/data/sample_data/expenses.xlsx" with extension ".xlsx" and mimetype "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" is handled by ExcelPlugin
NOTIFY:arcapix.search.metadata.plugins.validation:Plugin process returned <PluginStatus.SUCCESS: 1>
NOTIFY:arcapix.search.metadata.plugins.validation:Plugin processing took 0.318276 seconds
NOTIFY:arcapix.search.metadata.plugins.validation:Extracted metadata:
{
    "readonly": false,
    "sheets": [
        "Summary",
        "Expenses",
        "Mileage"
    ]
}
NOTIFY:arcapix.search.metadata.plugins.validation:Some issues were found with ExcelPlugin

ExcelPlugin :: server :: excel :: readonly
❌ field isn't registered

Once again, we have to restart the Search server to register the new field

$ systemctl restart apsearch-middleware

$ APLOGLEVEL=notify searchctl plugins check excel_plugin::ExcelPlugin /mmfs1/data/admin/expenses.xlsx
...
NOTIFY:arcapix.search.metadata.plugins.validation:Extracted metadata:
{
    "readonly": false,
    "sheets": [
        "Summary",
        "Expenses",
        "Mileage"
    ]
}
✔️ No issues found!

Adding new metadata is as easy as above.

Changing or removing existing metadata is hard. It requires either rebuilding the DB from scratch, or reindexing.

Summary ¶

In this walk-through we developed a plugin which extracts metadata from Excel files.

We used searchctl plugins check to validate our progress as we went along.

Finally, we installed and enabled our plugin, and ingested some files with it.

For a discussion of more advanced topics, including developing plugins for generating proxies, and submitting asynchronous jobs, see the Plugin Development Overview and Asynchronous Proxy Plugin Walk-Through

Appendix: Complete Plugin ¶

Here is the complete ExcelPlugin we developed over the course of this walk-through

from openpyxl import load_workbook

from arcapix.search.metadata.plugins.base import Plugin, PluginStatus
from arcapix.search.metadata.helpers import Metadata


EXTENSIONS = ['.xls', '.xlsx']
MIME_TYPES = [
    'application/vnd.ms-excel',
    'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
]


class ExcelPlugin(Plugin):

    def namespace(self):
        """Return the namespace for the metadata for this plugin."""
        return 'excel'

    def is_async(self):
        """Return whether this plugin does any operations asynchronously."""
        return False

    def handles(self, ext=None, mimetype=None):
        """Return whether the plugin can handle a given file, based on extension/mimetype

        :param ext: file extension, including a leading '.'
        :param mimetype: file mime type, e.g. image/png

        :returns: True if the plugin needs to process this file, false otherwise
        """
        return ext in EXTENSIONS or mimetype in MIME_TYPES

    def schema(self):
        """Return the schema for the metadata produced by this plugin.

        All metadata will be validated against this before inserting into the database.
        """
        return [
            {
                "name": "sheets",
                "prompt": "Names of sheets in this workbook",
                "value": {
                    "datatype": "[String]"
                }
            },
            {
                "name": "readonly",
                "prompt": "Indicates whether the workbook is readonly",
                "value": {
                    "datatype": "Boolean"
                }
            }
        ]

    def _extract(self, filename):
        """Private worker function to extract metadata from a file.

        :param filename: file to extract data from

        :return: dict of metadata
        """
        workbook = load_workbook(filename=filename)

        return {
            'sheets': workbook.sheetnames,
            'readonly': workbook.read_only,
        }

    def process(self, id_, file_, fileinfo=None):
        """Extract metadata from a given file and submit an update.

        :param id_: black box identifier which shall be passed to the metadata update functions
        :param file_: full path to the file
        :param fileinfo: dict of information which may have already been gathered and may be of use

        :returns: PluginStatus indicating success/failure
        """
        try:
            data = self._extract(file_)

            if not data:
                # didn't find any metadata
                return PluginStatus.SKIPPED

            if Metadata(id_, self).update(data):
                return PluginStatus.SUCCESS

            return PluginStatus.ERRORED

        except Exception as exc:
            self.logger.error(
                "An exception was raised while processing '%s' (%s): %s",
                file_, id_, exc
            )
            return PluginStatus.FATAL

Plugin Development Walk-through¶

Introduction ¶

Plugin template ¶

Plugin check tool ¶

Filling in the template ¶

Handles ¶

Schema ¶

Review ¶

Extracting metadata ¶

Installing dependencies ¶

Validate metadata ¶

Review ¶

Enabling the plugin ¶

Perform an ingest ¶

Adding more metadata ¶

Summary ¶

Appendix: Complete Plugin ¶

Extension	Mime type
.xls	application/vnd.ms-excel
.xlsx	application/vnd.openxmlformats-officedocument.spreadsheetml.sheet