8.5.1. TRANSPARENT_RECALL¶

8.5.1.1. SYNOPSIS¶

transparent_recall file [--config CONF]

8.5.1.2. DESCRIPTION¶

transparent_recall is a tool for recalling individual files via Ngenea Hub

Performing recalls via Ngenea Hub allows for monitoring progress via the Ngenea Hub UI. Individual recall tasks performed on demand, but for reporting are grouped together into one job per hour.

transparent_recall can be called directly to recall files, but typically would be installed as a filesystem policy rule. See TRANSPARENT RECALL POLICY for more info.

8.5.1.3. OPTION SUMMARY¶

      file          One or more directory to export events from

    --config CONF   Alternative configuration file location
-h, --help          Print help message and exit

8.5.1.3.1. OPTIONS¶

--config

The path to an alternative configuration file.

If not specified, the default configuration paths will be used. The default paths are in the user's HOME directory $HOME/.config/ngenea/ngenea-client.conf, and the global configuration at /etc/ngenea/ngenea-client.conf

See the CONFIGURATION section below and ngenea-client.conf(5) for more information.

Command line options take precedence over any corresponding config file settings.
-h, --help

Prints the help message.

8.5.1.4. CONFIGURATION¶

transparent_recall requires authentication to be able to perform recalls via Ngenea Hub. To authenticate, a valid client_key must be placed in the configuration file.

A client key can be generated via the Ngenea Hub REST API, or using the ngclient(1) authenticate command.

Minimally, the configuration must include this client_key, as well as the site where recalls are performed.

[settings]
client_key = ...
site = thissite

The site must match the the node where the recall was triggered.

transparent_recall will respect any ngclient(1) recall configuration options, except for recursive. This includes timeout; by default it will wait indefinitely for the recall to complete.

See ngenea-client.conf(5) for more information on the configuration format and additional options.

8.5.1.5. TRANSPARENT RECALL POLICY¶

Transparent recall, by definition, is intended to be triggered automatically when an offline file is opened for reading or writing.

To enable transparent recall functionality using transparent_recall, the following rules can be added to the filesystem placement policy.

/* BEGIN: NGENEA TRANSPARENT RECALL POLICY */

define(xattr_stbsz,[INTEGER(XATTR('dmapi.APXstbsz'))])

RULE FileOpen EVENT 'OPEN'
     ACTION(SetDataEvent(0, OP_READ, CASE WHEN xattr_stbsz IS NULL THEN 0 ELSE xattr_stbsz END) AND
            SetDataEvent(1, OP_WRITE+OP_TRUNC, 0))
     WHERE LENGTH(XATTR('dmapi.APXguuid'))>0 AND CountSubstr(MISC_ATTRIBUTES,'V')>0

RULE FileOpen_else EVENT 'OPEN' DIRECTORIES_PLUS

RULE FileData EVENT 'DATA'
     ACTION(system_lw('/usr/bin/python3 /usr/bin/transparent_recall ' || getDetail('path_name') )=0)
     WHERE LENGTH(XATTR('dmapi.APXguuid'))>0 AND CountSubstr(MISC_ATTRIBUTES,'V')>0

RULE FileData_else EVENT 'DATA' DIRECTORIES_PLUS

/* END: NGENEA TRANSPARENT RECALL POLICY */

If transparent recall (EVENT) rules are already installed for ngenea (native), these rules should replace those equivalent rules.

Don't replace any rules besides the ngenea EVENT rules, e.g. don't replace any SET POOL rules.

See mmchpolicy(1) for how to change the filesystem placement policy

8.5.1.5.1. WARNING¶

Due to limitations of GPFS, the above policy will not work on any paths which contain whitespace. Transparent recall will simply error.

An alternative policy substitutes the system_lw line for

ACTION(system('/usr/bin/python3 /usr/bin/transparent_recall ''' || getDetail('path_name') || '''')=0)

This will handle paths containing whitespace, but not those containing single quotes. Again, transparent recall will error.

There is currently no approach which works in all cases. You must chose the version which is most compatible with the sorts of paths you have on your system.

8.5.1.6. TROUBLESHOOTING¶

When attempting to read an offline file, if the read process reports "Operation not permitted", and it's not due to permissions, the most likely cause is that the recall failed.

Logs for the transparent_recall command invocation can be found at /var/adm/ras/mmfs.log.latest

Logs for the transparent recall job can be viewed via Ngenea Hub.

WARNING - if reading a file triggers a recall, the read request will block until recall exits; it can't be interrupted (Ctrl+C) or killed (kill -9). If the recall job is 'stuck' and no timeout is set, the only way to make the read process exit is to kill the recall job via Ngenea Hub.

8.5.2. AVAILABILITY¶

Distributed as part of the ngenea-hub-client rpm, or the ngclient wheel (Python) for non-Red Hat based systems.

Note - transparent_recall makes use of flock(2), so can only be used on Unix-base operating systems.

8.5.2.1. SEE ALSO¶

ngclient(1), ngenea-client.conf(5), ngrecall(1), mmchpolicy(1)

8.5.2.2. LICENSE¶

2021 ArcaPix Limited

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.