Changelog

1.2.0 - 2024-02-01

Bug fixes:

  • RNDEAS-1475: Validation error for valid GeoPoint data

Feature enhancements:

  • HUB2-1172: Tags plugin

1.1.0 - 2023-10-26

Bug fixes:

  • RNDEAS-1460: Handle unicode error for invalid DPX film edge codes
  • RNDEAS-1462: Remove unsupported float values from extracted metadata
  • RNDEAS-1464: Remove invalid GPS values from the location plugin
  • RNDEAS-1467: Last ingest time improperly formatted in verify-file
  • RNDEAS-1471: 500 error from projection on unknown field

Improvements:

  • RNDEAS-1465: Ability to configure the google cloud plugin poll interval

Feature enhancement:

  • RNDEAS-1473: Support for per-field AND filters

1.0.0 - 2022-05-26

Refactoring:

  • RNDEAS-1446: Migrated to Python 3
  • Deprecated commands (finder, etc.) have been removed

Bug fixes:

  • RNDEAS-1451: elasticsearch service status reporting on PixStor 6

Improvements:

  • RNDEAS-1434: Change video preview audio encoding to vorbis

Breaking Changes:

  • Renamed plugin ‘async’ method to ‘is_async’
  • Any user plugins which are not compatible with Python 3 will no longer work. See Migrate to PixStor Search 1.0 for more details.

0.15.0 - 2022-05-17

Bug fixes:

  • RNDEAS-1426: UnicodeDecodeError in prune directory plugin
  • RNDEAS-1435: Ingest auth token expires after 24 hours

Feature enhancement:

  • RNDEAS-1360: Extract more DPX metadata
  • RNDEAS-1363: Plugin to extract endian information
  • RNDEAS-1365: Adobe Premier project plugin
  • RNDEAS-1374: Plugin to extract Maya fileinfo metadata
  • RNDEAS-1427: Plugin development walk-throughs
  • RNDEAS-1430: Add plugin check for missing schema
  • RNDEAS-1432: Searchctl command to verify ingest status of individual files
  • RNDEAS-1448: Ngenea metadata plugins
  • Vantage video proxy plugin

Improvements:

  • RNDEAS-1431: Sort plugins with the same priority alphabetically

0.14.0 - 2021-03-30

Updated EULA

Bug fixes:

  • RNDEAS-1376: More informative error when trying to ingest single file
  • RNDEAS-1377: Reason not listed in searchctl jobs for memory errors
  • RNDEAS-1386: Exiftool can’t identify some formats with -fast3
  • RNDEAS-1398: Revert filters link name in C+J
  • RNDSRCHUI-249: TypeError for aggregation of non-range numeric filter

Feature enhancements:

  • RNDEAS-1344: Support for generating null-terminated lists from verify-ingest
  • RNDEAS-1385: Use policy output processor during ingest
  • RNDEAS-1390: Colorized searchctl jobs table
  • RNDEAS-1392: Searchctl command for checking service statuses

Refactoring:

  • RNDEAS-1391: Update to use rights instead of roles

0.13.0 - 2021-01-08

Bug fixes:

  • RNDEAS-1185: Prune directory error if no ‘core.inode’ data
  • RNDEAS-1242: Paths aren’t properly quoted in HATEOAS links
  • RNDEAS-1300: Unicode error for ingest with unicode target path
  • RNDEAS-1303: Don’t allow stopping all jobs in searchctl
  • RNDEAS-1304: Memory error from verbose ingest logging
  • RNDEAS-1328: Improve error reporting for broker user config
  • RNDEAS-1330: Rephrase cleandb logging messages
  • RNDEAS-1339: Dynamic plugins errors from unicode paths
  • RNDEAS-1349: Unicode error in searchctl jobs

Feature enhancements:

  • RNDEAS-985: Support for multiple ingest locks
  • RNDEAS-1211: Manpages for searchctl single-file tools
  • RNDEAS-1261: Support for setting ImageMagick temp dir in search.yaml
  • RNDEAS-1269: Support for running stat-only in parallel with rich ingest
  • RNDEAS-1279: Record logging settings in searchctl jobs
  • RNDEAS-1291: Document how to install plugin dependencies
  • RNDEAS-1304: Show ingest logging in searchctl ingest
  • RNDEAS-1313: Sort jobs by status change time in searchctl
  • RNDEAS-1326: Plugin for curating duplicate entries by path
  • RNDEAS-1351: Add plugin schema endpoint to Swagger docs
  • RNDEAS-1358: Mark locking errors as SKIPPED in searchctl jobs

0.12.0 - 2020-09-25

Bug fixes:

  • RNDEAS-1310: Support pre 1900 dates in metadata fields
  • RNDEAS-1311: Don’t skip exiftool mimetype for large files
  • RNDEAS-1312: Wrong encoding of non-ascii characters in retry lists

Feature enhancements:

  • RNDEAS-1091: Support for ‘follow’ mode in searchctl logs
  • RNDEAS-1287: Dynamic plugin filter

0.11.0 - 2020-08-21

New Features:

  • RNDEAS-1228: Ability to set a size limit for plugins
  • RNDEAS-1257: Include site name in email alert subject line

Improvements:

  • RNDEAS-937: SIGTERM / SIGINT on apsearch-ingest should propagate to all children
  • RNDEAS-950: Show headers when using ‘pxs_file_list’ search CLI utility
  • RNDEAS-1045: Change ingest output to report multiple invalid plugins filters
  • RNDEAS-1049: Validate plugin - support for dynamic plugins
  • RNDEAS-1095: Ability to keep policy lists
  • RNDEAS-1145: Tweaks for searchctl plugins list
  • RNDEAS-1148: Searchctl ingest could inform of log at start of ingest
  • RNDEAS-1163: Implement plugin ingest timeout
  • RNDEAS-1171: searchctl filter jobs statuses
  • RNDEAS-1172: searchctl add total ingest time
  • RNDEAS-1200: Support APLOGTARGET in search.yaml
  • RNDEAS-1201: Ensure mimetype tools do not read huge files
  • RNDEAS-1213: Ensure ingest worklists are contained within their own directory
  • RNDEAS-1241: Ability to change column widths in searchctl jobs
  • RNDEAS-1254: Increase speed and robustness of PDF Proxying
  • RNDEAS-1273: rpmvenv deployment for middleware
  • RNDEAS-1298: Search 0.11 is converting TIFs via ImageMagick, not OpenImageIO
  • RNDEAS-1217: Ability to match plugins on mimetype only
  • RNDEAS-1218: Handle single frame videos
  • RNDEAS-1219: Xsym plugin
  • RNDEAS-1220: Exiftool processes are left running when ingest dies
  • RNDEAS-1230: Invalid metadata values from exiftool
  • RNDEAS-1235: Handle video files with no video stream
  • RNDEAS-1237: Camera plugin error for invalid flash value
  • RNDEAS-1238: Image fix orientation error for invalid orientation
  • RNDEAS-1239: Investigate cause of 422 errors on EG+
  • RNDEAS-1274: wsgi-searchserver.py needs to be in $PATH
  • RNDEAS-1275: Bundle modified systemd unit in RPM
  • RNDEAS-1277: Binary stripping causes ELF errors
  • RNDEAS-1278: Update sudoers to match new paths
  • RNDEAS-1209: Support for yaml with searchctl ingest

Bug fixes:

  • RNDEAS-627: Unallowed white balance
  • RNDEAS-948: pxs_file_list –filter search CLI utility does not work
  • RNDEAS-989: Condor job put on hold if files that require async job are ingested by root user
  • RNDEAS-1075: Unicode compatible paths are ingested with no issue but do not appear in UI
  • RNDEAS-1103: Unicode error for pxs_file_list
  • RNDEAS-1143: proxy: mimetype of generated proxy “video/mp4” doesn’t match mimetype specified by plugin “video/mpeg”
  • RNDEAS-1146: Log message states to retry directory None
  • RNDEAS-1164: Regex searching returns 500
  • RNDEAS-1167: searchctl plugins check does not work alike validate_plugin
  • RNDEAS-1173: ConflictError: TransportError(409, u’version_conflict_engine_exception
  • RNDEAS-1182: VideoThumbnail - IOError: cannot identify image file <cStringIO.StringI object
  • RNDEAS-1193: double CTRL-C required to stop ingest
  • RNDEAS-1196: Per plugin logs print out unicode, summary log prints out proper filename
  • RNDEAS-1199: CommonAttributesPlugin common.creator: must be of string type
  • RNDEAS-1206: PhotoshopMetaDataPlugin - TypeError: Struct() argument 1 must be string, not unicode
  • RNDEAS-1215: Imagmagick tmp can fill OS filesystem
  • RNDEAS-1216: Assorted plugin errors at EG+
  • RNDEAS-1222: Quote paths in queries for prunedirectory
  • RNDEAS-1227: Prune directory query error when path contains double quotes
  • RNDEAS-1236: Broken pipe after exiftool crash
  • RNDEAS-1243: Setting security utils to true breaks the DB
  • RNDEAS-1255: 0.11b3 cannot import name abc
  • RNDEAS-1258: Handle exceptions in videopreview
  • RNDEAS-1266: Remove video/x-ms-asf from imagepreview
  • RNDEAS-1270: PDF Preview thumbnail aspect stretched
  • RNDEAS-1272: systemd service file has wrong variable names
  • RNDEAS-1281: Ensure non-root users can ingest with searchctl
  • RNDEAS-1290: find_duplicates not working on 10.132.0.175

Breaking changes:

  • RNDEAS-948: rename pxs_file_list flags –single-field to –output-field
    and –field to –filter-by-field

0.10.1 - 2020-06-18

Bug fixes:

  • RNDEAS-1223: Change searchctl shebang to use env Python 2.7
  • RNDEAS-1224: Change wsgi script shebang to use env Python 2.7

0.10.0 - 2020-05-14

Bug fixes:

  • RNDEAS-1085: Truncate logging of ffmpeg output
  • RNDEAS-1101: Don’t use tuples in plugin docs and core plugins
  • RNDEAS-1108: Error when searchctl logs level isn’t set
  • RNDEAS-1109: apsearch-ingest returned exit code 1 on success
  • RNDEAS-1110: Ignore trailing slash when filtering jobs by target
  • RNDEAS-1113: Some MXF proxies are transparent
  • RNDEAS-1117: Profile plugin backwards compatibility
  • RNDEAS-1165: Error when PDF metadata plugin doesn’t extract any data

Feature enhancements:

  • RNDEAS-823: Prevent ingest from changing file timestamps (atime, mtime, ctime)
  • RNDEAS-956: Tool for recommending ingest configurations for stat-only ingest
  • RNDEAS-957: Manpages for searchctl tools
  • RNDEAS-994: Support for benchmarking plugins against multiple files
  • RNDEAS-1016: Tool for listing and stopping active ingests
  • RNDEAS-1037: Support for resetting file access times (atime) on ingest
  • RNDEAS-1063: Generate proxies for more image and video types
  • RNDEAS-1071: Plugin filter for ‘core’ plugins
  • RNDEAS-1087: Convert existing CLI tools to searchctl commands
  • RNDEAS-1088: Tool for listing enabled plugins
  • RNDEAS-1093: Display searchctl logs in a pager
  • RNDEAS-1099: PDF metadata and proxy plugins
  • RNDEAS-1100: Add MXF plugin to Lab
  • RNDEAS-1104: Document how to perform range queries
  • RNDEAS-1106: Dynamic plugin to show duplicate files
  • RNDEAS-1107: Add descriptions for all core plugins
    • RNDEAS-1155 Parent navigation with Nav plugin
  • RNDSRCHUI-202: Dynamic Flags plugin to show real-time on/offline file status

Refactoring:

  • The existing CLI tools (finder, etc) are deprecated in favour of equivalent ‘searchctl’ tools. The old tools still exist in this release for compatibility, but will be removed in a future release.
  • RNDEAS-1137: Create Arcapix Lab plugin location

0.9.0 - 2020-02-10

Bug fixes:

  • RNDEAS-993: Prune directory fails for directories bigger than 10k files
  • RNDEAS-999: Async proxy update fails if proxy already exists
  • RNDEAS-1010: Validation error for creation time of filesystem mountpoint
  • RNDEAS-1019: Don’t warn about root ingest for stat-only
  • RNDEAS-1031: Error when filtering core plugins by mimetype
  • RNDEAS-1032: More descriptive error for bad plugin filters
  • RNDEAS-1033: Bad error message from metadata validation error
  • RNDEAS-1035: Don’t try to calculate hash for directories
  • RNDEAS-1040: apsearch-ingest doesn’t show full logging output
  • RNDEAS-1042: Validate plugin tool doesn’t recognise ‘remote’ type
  • RNDEAS-1065: Make sure ‘broker’ user can ‘see’ deleted files in search
  • RNDEAS-1069: Skip paths containing non-unicode bytes

Feature enhancements:

  • RNDEAS-242: Support for ingest with selected plugins
  • RNDEAS-863: Support for hiding proxies from selected users
  • RNDEAS-995: Method for checking if a core plugin already generated proxies for a file
  • RNDEAS-996: OpenAPI (Swagger) config for the REST API
  • RNDEAS-1008: Infer date parse order from locale + support for explicit configuration
  • RNDEAS-1030: File access security takes into account user roles
  • RNDEAS-1054: Set ingest syslog identifier to ‘arcapix.search.tools.finder’
  • RNDEAS-1064: Support for OPTIONS requests to check endpoint permissions
  • RNDEAS-1053: Support for incremental ingest using file change time (ctime)

Refactoring:

  • RNDEAS-1039: Make use of Eve roles configs

0.8.0 - 2019-11-22

Bug fixes:

  • RNDEAS-585: Plugin validate - supress misleading tracebacks
  • RNDEAS-805: Reduce unnecessary logger warnings
  • RNDEAS-933: Mimetype incorrectly identified for directories
  • RNDEAS-963: Validation error when sha512 hash can’t be calculated
  • RNDEAS-988: Prune directory plugin deletes renamed files
  • RNDEAS-1003: Tweak to offline file check

Feature enhancements:

  • RNDEAS-428: Optimised single-file ingest
  • RNDEAS-442: Support for multi-node ingest
  • RNDEAS-625: Support for exiftool ‘large file support’
  • RNDEAS-651: Plugin validate - check value precision
  • RNDEAS-654: Plugin validate - check schema against elasticsearch
  • RNDEAS-800: Support for ‘wand’ as an image converter
  • RNDEAS-882: Plugin validate - add profiling information
  • RNDEAS-907: Record per-plugin last successful ingest times
  • RNDEAS-935: Support for ingesting files from list
  • RNDEAS-940: Support for only re-ingesting failed and out-dated plugins
  • RNDEAS-943: Flexible last ingest timestamp support
  • RNDEAS-975: Support for recording lists of files which failed to ingest

Performance enhancements:

  • RNDEAS-251: Use bulk inserts in ingest
  • RNDEAS-441: Use asynchronous REST calls in ingest
  • RNDEAS-883: Use persistent ExifTool
  • RNDEAS-906: Implement mime.types as a method for determining mimetype
  • RNDEAS-942: Support for specifying preferred image converters
  • RNDEAS-972: Plugin profiling tool

Refactoring:

  • RNDEAS-824: apsearch-ingest script re-written to use yaml config
  • RNDEAS-844: Incremental ingest no longer uses snapshots
  • RNDEAS-895: Move file_security to apcore-utils

Incompatible changes:

  • PillowThumbnail was renamed to CoreImageThumbnail.
    Any plugins which import it - including ExtendedImageThumbnail - will need updating

0.7.1 - 2019-08-05

Bug fixes:

  • RNDEAS-874: Allow proxies to inherit acls
  • RNDEAS-913: Import error for async user plugins
  • RNDEAS-916: Only handle mov files based on mimetype
  • RNDEAS-918: Don’t error on lock cleanup if lock file is missing
  • RNDEAS-919: Recursion error from cleandb HTTPError
  • RNDEAS-920: Provide logging output for cleandb and other tools

Feature enhancements:

  • RNDEAS-901: Tool for locating proxies on the filesystem
  • RNDEAS-922: Safety check to verify mimetype for video plugins

0.7.0 - 2019-05-30

Bug fixes:

  • RNDEAS-571: Handle max results error nicely
  • RNDEAS-632: Improve results reporting from finder
  • RNDEAS-705: Cleandb error when directory contains more than 10,000 entries
  • RNDEAS-730: Sudoers file to specify what commands need to run as root
  • RNDEAS-768: Rename common plugin to reduce confusion re whether it is optional
  • RNDEAS-787: Handle KeyboardInterrupt nicely in find_duplicates
  • RNDEAS-843: 500 error when proxy wasn’t found
  • RNDEAS-845: Improved error reporting when ETAG can’t be fetched during ingest
  • RNDEAS-856: Support for PSD, PSB, and EXR file types
  • RNDEAS-870: Stat-only ingest doesn’t change access time (atime)
  • RNDEAS-894: Allow users with admin role to see all files

Feature enhancements:

  • Add profiling information for plugins

Incompatible changes:

  • RNDEAS-796: Logging module was removed. Standalone ‘aplogging’ package is required instead.

0.6.2 - 2019-02-28

Documentation Update:

  • Correct URL’s in documentation after change to /api

0.6.1 - 2019-02-25

Bug fixes:

  • RNDEAS-793: Error on desktop connector dynamic import

0.6.0 - 2019-02-19

Feature enhancements:

  • RNDEAS-707: Support for Dynamic Plugins and Desktop Connector
  • RNDEAS-688: Support for boolean operators in ‘_all’ queries
  • RNDEAS-738: Support for incoming Collection+JSON
  • Client tools for EDL Export, File List Export, and Attribute adjustment (BETA)

Refactoring:

  • RNDEAS-640: Move non-stat plugins out of core

0.5.0 - 2018-11-14

Bug fixes:

  • RNDEAS-655: Finder stop tries to read pid from lock file
  • RNDEAS-675: Error when snapshot history file in manually cleared
  • RNDEAS-676: Refine finder include/exclude matching
  • RNDEAS-679: Permission error when checking if files are offline

Feature enhancements:

  • Support for combining database queries

0.4.0 - 2018-10-03

Bug fixes:

  • RNDEAS-474: File security check error for unicode filenames
  • RNDEAS-559: Text fields being sorted by tokenised values
  • RNDEAS-562: PickleError from ‘finder remove’
  • RNDEAS-575: Make file security checks nGeanea-safe
  • RNDEAS-599: Make sure async job engine can import user plugins
  • RNDEAS-618: Add missing config documentation
  • RNDEAS-621: Exiftool fails for large video files
  • RNDEAS-628: Ingest dies if file can’t be read
  • RNDEAS-629: Security checker randomly fails
  • RNDEAS-630: Validate plugin reports ‘hint’ field as unknown field error

Feature Enhancements:

  • RNDEAS-175: Support for offline files
  • RNDEAS-447: Tool for cleaning up orphaned proxy files
  • RNDEAS-519: Support for interfacing with external resources
  • RNDEAS-576: Add file-type (core.type) metadata field
  • RNDEAS-637: Search recursively for plugins in plugin path

Refactoring:

  • RNDEAS-504: APConfig and APAuth split into separate packages
  • RNDEAS-523: Configurable lock path
  • RNDEAS-534: Remove misleading logging of exceptions on proxy move
  • RNDEAS-557: Increase pagination limit to 1000
  • RNDEAS-558: Make date aggregations return ints rather than floats
  • RNDEAS-573: Configurable proxy-gen work directory
  • RNDEAS-579: Remove default value for proxy-gen work directory - must be configured explicitly
  • RNDEAS-591: Convert classmethod generate_temp_filename to standalone utility function
  • RNDEAS-613: Async jobs only run on HTCondor nodes configured for apsearch
  • RNDEAS-622: Remove misleading logging of exception when Pillow can’t open an image
  • RNDEAS-625: Improved handling of errors from exiftool
  • RNDEAS-649: Change type of core.user.id and core.group.id to long

0.3.1 - 2018-04-13

Bug fixes:

  • RNDEAS-565: issues generating docs for rpm build

Feature enhancement:

  • RNDEAS-487: move example plugins to public repo
  • RNDEAS-569: example plugin for extended image thumbnail support

0.3.0 - 2018-02-01

Bug fixes:

  • RNDGPA-234: fix uid/gid lookup on SSSD
  • RNDEAS-445: proxygen fails when proxy already exists

Feature enhancements:

  • RNDEAS-465: add plugin validation tool
  • RNDEAS-499: support for incremental ingest (scan) without snapshots
  • RNDEAS-501: add ingest locking
  • RNDEAS-501: add support for email alert on ingest error
  • RNDEAS-501: add stop command to kill ingest gracefully
  • RNDEAS-502: improve snapshot rotation for ingest
  • RNDEAS-506: database cleanup tool

Incompatible changes:

  • RNDEAS-268: logging controlled via APDEBUGLOG and APDEBUGLEVEL environment variables
  • RNDEAS-500: rename monitor to finder
  • Various database schema changes - NOTE: you will need to delete any existing Elastic DB
  • Integrated eve-easysearch package - NOTE: you may need to remove the existing package manually

0.2 - 2017-07-14

  • Initial release