Copying¶
The ngcopy tool copies or moves objects from source paths to a destination path within a storage container (E.G. bucket) or from source storage containers to a destination storage container. By default, ngcopy processes the paths recursively.
While copying or moving objects, ngcopy can update remote location extended attributes of local files corresponding to the source paths and make them reference copied or moved objects under the destination path.
ngcopy¶
Synopsis¶
ngcopy [ --move ] ( --update-rmtlc | --no-update-rmtlc )
-E SOURCE_ALIAS:SOURCE_PATH
DESTINATION_ALIAS:DESTINATION_PATH
ngcopy [ --move ] ( --update-rmtlc | --no-update-rmtlc )
( -E RESTRICTION_ALIASES[:RESTRICTION_PATHS] )+
( --endpoint-exclude=EXCLUSION_ALIASES[:EXCLUSION_PATHS] )*
DESTINATION_ALIAS[:DESTINATION_PATH/]
ngcopy [ --move ] ( --update-rmtlc | --no-update-rmtlc )
-F RMTLIST DESTINATION_ALIAS[:DESTINATION_PATH/]
ngcopy [ --move ] ( --update-rmtlc | --no-update-rmtlc )
-L REMOTE_LOC DESTINATION_ALIAS[:DESTINATION_PATH/]
Description¶
Copies or moves remote objects.
Options¶
--all-obj-instances
copy or move all instances of source remote objects.
Default: copy or move only latest object instances.
--config-file=FILE
path to a master configuration file.
Default: /opt/arcapix/etc/ngenea.conf
-E, --endpoint=ALIASES[:PATHS]
restrict the set of storage endpoints for copying or moving
source remote objects and folders to endpoints with aliases
specified by extended glob pattern ALIASES.
Optionally, restrict remote object and folder pathnames at the
endpoints to pathnames matching extended glob pattern PATHS.
By default, restrict remote object and folder pathnames to the
root path. Compatible with: --no-recursion-remote.
--endpoint-exclude=ALIASES[:PATHS]
exclude remote objects and folders with pathnames specified by
extended glob pattern PATHS at storage endpoints with aliases
specified by extended glob pattern ALIASES from copying
or moving.
Compatible with: --no-recursion-remote.
-F RMTLIST process source remote paths read from a file and treat them as
restriction paths.
Every line of the file should have the format: ALIAS:PATH, where
ALIAS is a storage endpoint alias, and PATH is a remote path at
the storage endpoint.
Wildcard characters in ALIAS and PATH are treated as
ordinary characters.
Compatible with: --no-recursion-remote.
--folder-prefix=FOLDER
strip a specified folder prefix from remote object names when
inferring local file names unless the option
`--retain-folder-prefix-in-fnames' is present.
Prepend a specified folder prefix to restriction and exclusion
paths for scanning remote objects and folders.
The prefix is never percent encoded or decoded.
Compatible with: --retain-folder-prefix-in-fnames.
--help display this help and exit.
--lock-level=partial|implicit
DMAPI locking level:
"partial" - explicitly request a DMAPI exclusive access
right for the duration of updating the remote
location xattr of a file;
"implicit" - instruct DMAPI to self-manage access rights when
updating the remote location xattr of a file.
Default: "partial"; can be specified in the configuration file
or overridden by command line.
--log-format=json
log messages in JSON format.
Conflicts with: --log-target=syslog.
--log-target=syslog
redirect all logging to the syslog.
Conflicts with: --log-format=json.
--move move source remote objects and folders to destination remote
objects and folders.
Default: copy source remote objects and folders to destination
remote objects and folders.
--no-flock disable using lock files.
Sets lock level to "implicit" if it is not set.
Conflicts with "partial" lock level.
--no-recursion-remote
disable recursive interpretation of restriction and exclusion
extended glob patterns for remote object and folder pathnames
and remote object and folder pathnames specified in a path
list file.
The recursive interpretation means matching sub-directories at
all nesting levels, whereas non-recursive interpretation means
matching a single directory.
Compatible with: -E, --endpoint; --endpoint-exclude; -F.
--no-update-rmtlc
do not update remote location xattrs of local files
corresponding to copied or moved remote objects to reference
their new destination.
--overwrite-remote
overwrite remote objects and folders if they already exist at a
copy or move destination.
Default: skip copying or moving remote objects and folders
already existing at a copy or move destination.
-P, --param-endpoint=ALIASES:PARAMETER=VALUE
add a parameter with name PARAMETER and value VALUE to
parameters read from configuration files for storage endpoints
with aliases specified by extended glob pattern ALIASES.
If PARAMETER already exists in a configuration file, it takes
a new VALUE.
--perf-dstat=INTERVAL,FILE
record the following information about the running program to a
FILE every INTERVAL seconds: virtual memory size (in
megabytes), resident set size (in megabytes), thread count, and
the number of open file descriptors.
--perf-profile[=all]
dump cumulative times of executing various operations:
"all" - dump cumulative times for all operations executed at
least once (default: hide operations with
insignificant times).
-L, --remote-loc-xattr=REMOTE_LOC
location of a remote object or folder to copy or move.
--retain-folder-prefix-in-fnames
retain a folder prefix in local file names when inferring
files for updating remote location xattrs.
Default: strip the folder prefix.
Compatible with: --folder-prefix; --update-rmtlc.
--show-progress[=yes|no]
displaying the progress of copying/moving a current object:
"yes" - display the progress in classic (not JSON) output
mode if verbosity level is less than 2 even if stdout
is not a terminal (default);
"no" - do not display the progress.
Default: display the progress in classic output mode if
verbostiy level is 1, and stdout is a terminal.
--update-rmtlc update remote location xattrs of local files corresponding to
copied or moved remote objects to reference their
new destination.
Compatible with: --retain-folder-prefix-in-fnames.
-v, --verbose[=LEVEL]
verbosity level:
0 = error and warning messages (also used when this option
is absent);
1 = display copying/moving progress for a current remote
object; print source and destination endpoint aliases and
paths of successfully copied/moved remote
objects and folders; [default]
2 = debug messages, excluding those related to file locking;
3 = enable core dump and debug messages related to file locking;
print PID and current time with microsecond precision.
-V, --version display version information and exit.
Examples¶
Copy a single object to another path in the endpoint "awss3", and update the remote location extended attribute of a local file referencing the object:
ngcopy --update-rmtlc -E awss3:path/to/source/obj awss3:path/to/destination/obj
Move a folder to another one recursively without updating the remote location extended attribute while displaying the names of processed objects and their move progress:
ngcopy -v --move --no-update-rmtlc --folder-prefix=source/folder -E awss3 awss3:destination/folder/