ProjectData
Usage:
icav2 projectdata <command> <args...>
CLI Commands:
archive archive data
create Create data id for a project
delete delete data
download Download a file/folder
downloadurl get download url
folderuploadsession Get details of a folder upload
get Get details of a data
link Link data to a project
list List data
mount Mount project data
temporarycredentials fetch temporal credentials for data
unarchive unarchive data
unlink Unlink data to a project
unmount Unmount project data
update Updates the details of a data
upload Upload a file/folder
Plugin Commands:
ls List data with standard posix ls options
view View a file and parse into stdout
find Find a file / directory based on depth, regex and type (like unix find)
s3-sync-upload Upload a directory to a project folder, calling aws s3 sync underneath
s3-sync-download Download a directory to a project folder, calling aws s3 sync underneath
create-download-script Create a shell script that downloads a project folder via presigned urls
Flags:
-h, --help help for projectanalyses
Global Flags:
-t, --access-token string JWT used to call rest service
-o, --output-format string output format (default "table")
-s, --server-url string server url to direct commands
-k, --x-api-key string api key used to call rest service
Use "icav2 projectdata [command] --help" for more information about a command.
Traverse a file system as if it's a posix file system.
Use the --long-listing, --time, and --reverse commands to sort your output.
Usage:
icav2 projectdata ls help
icav2 projectdata ls [-l | --long-listing]
[-t | --time]
[-r | --reverse]
icav2 projectdata ls <data_path>
[-l | --long-listing]
[-t | --time]
[-r | --reverse]
Description:
List data in directory, similar to ls in a posix file system
Options:
<data_path> Optional, data path, defaults to '/'
-l, --long-listing Optional, use long-listing format to show owner, modification timestamp and size
-t, --time Optional, sort items by time
-r, --reverse Optional, reverse order
Environment variables:
ICAV2_BASE_URL Optional, default set as https://ica.illumina.com/ica/rest
ICAV2_PROJECT_ID Optional, taken from "$HOME/.icav2/.session.ica.yaml" if not set
ICAV2_ACCESS_TOKEN Optional, taken from "$HOME/.icav2/.session.ica.yaml" if not set
Example: icav2 projectdata ls /reference_data/
Use the <tab> command to traverse through files and folders as you would with a regular file system
Stream a file to stdout (or your browser) with the projectdata view subcommand
icav2 projectdata view --help
Usage:
icav2 projectdata view help
icav2 projectdata view <data_path>
[-b | --browser]
Description:
View a file to stdout
Options:
<data_path> Required, path to file
-b, --browser Optional, display in browser
Environment variables:
ICAV2_BASE_URL Optional, default set as https://ica.illumina.com/ica/rest
ICAV2_PROJECT_ID Optional, taken from "$HOME/.icav2/.session.ica.yaml" if not set
ICAV2_ACCESS_TOKEN Required, taken from "$HOME/.icav2/.session.ica.yaml" if not set
BROWSER Optional, required if --browser is set
Example: icav2 projectdata view /output_data/tiny.fastq.gz | zcat | head
Use the <tab> command to traverse through files and folders as you would with a regular file system
Find data in directory, similar to find in a posix file system.
Usage:
icav2 projectdata find help
icav2 projectdata find [-w=<min_depth> | --min-depth=<min_depth>]
[-x=<max_depth> | --max-depth=<max_depth]
[--type=<type>]
[-n=<name> | --name=<name>]
[-l | --long-listing]
[-t | --time]
[-r | --reverse]
icav2 projectdata find <data_path>
[-w=<min_depth> | --min-depth=<min_depth>]
[-x=<max_depth> | --max-depth=<max_depth]
[--type=<type>]
[-n=<name> | --name=<name>]
[-c=<creator_username_or_id> | --creator=<creator_username_or_id>]
[-l | --long-listing]
[-t | --time]
[-r | --reverse]
Description:
Find data in directory, similar to find in a posix file system.
Options:
<data_path> Optional, data path, defaults to '/'
-w=<min_depth>, --min-depth=<min_depth> Optional, minimum depth to search
-x=<max_depth>, --max-depth=<max_depth> Optional, maximum depth to search
--type=<type> Optional, one of "FILE" or "DIRECTORY"
-n=<name>, --name=<name> Optional, name of file or directory, regex expressions are possible here
-c=<creator_id_or_name>, --creator=<creator_id_or_name> Optional, creator username or id
-l, --long-listing Optional, use long-listing format to show owner, modification timestamp and size
-t, --time Optional, sort items by time
-r, --reverse Optional, reverse order
Environment variables:
ICAV2_BASE_URL Optional, default set as https://ica.illumina.com/ica/rest
ICAV2_PROJECT_ID Optional, taken from "$HOME/.icav2/.session.ica.yaml" if not set
ICAV2_ACCESS_TOKEN Optional, taken from "$HOME/.icav2/.session.ica.yaml" if not set
Example: icav2 projectdata find /reference_data/
Use the <tab> command to traverse through files and folders as you would with a regular file system
🚧
Uses the aws s3 sync command to upload and download directories.
User can specify multiple additional s3 parameters to test uploads / downloads OR to upload / download only certain files.
Can also specify a write-script-path so the output script can be copied to a server that does not have access to the icav2 cli.
$ icav2 projectdata s3-sync-upload --help
Usage:
icav2 projectdata s3-sync-upload help
icav2 projectdata s3-sync-upload <upload_path> <data_path>
[-w=<file_path> | --write-script-path=<file_path> ]
[-s=<s3_sync_arg> | --s3-sync-arg=<s3_sync_arg>]...
Description:
Upload a folder to icav2 using aws s3 sync.
Options:
<upload_path> Required, the source directory, directory must exist
<data_path> Required, path to icav2 data folder you wish to upload to
-w=<file_path>, --write-script-path=<file_path> Optional, write out a script instead of invoking aws s3 command
that holds the AWS S3 Sync parameters.
upload path parameter folder does not necessarily need to exist if this option
is set. Parent folder of this parameter must exist.
-s=<s3_sync_arg>, --s3-sync-arg=<s3_sync_arg> Other arguments are sent to aws s3 sync, specify multiple times for multiple arguments
Environment variables:
ICAV2_BASE_URL Optional, default set as https://ica.illumina.com/ica/rest
ICAV2_PROJECT_ID Optional, taken from "$HOME/.icav2/.session.ica.yaml" if not set
ICAV2_ACCESS_TOKEN Required, taken from "$HOME/.icav2/.session.ica.yaml" if not set
Extras:
AWS CLI V2 must be installed
You MUST have write permissions to this project to run this command
Examples: icav2 projectdata s3-sync-upload $HOME/test_inputs/ /test_data/inputs/
icav2 projectdata s3-sync-upload $HOME/test_inputs/ /test_data/inputs/ --s3-sync-arg --dryrun
icav2 projectdata s3-sync-upload $HOME/test_inputs/ /test_data/inputs/ --s3-sync-arg --exclude='*' --s3-sync-arg --include='*.bam'
Usage:
icav2 projectdata s3-sync-download help
icav2 projectdata s3-sync-download <data_path> <download_path>
[-w=<file_path> | --write-script-path=<file_path> ]
[-s=<s3_sync_arg> | --s3-sync-arg=<s3_sync_arg>]...
Description:
Download a folder from icav2 using aws s3 sync.
Options:
<data_path> Required, path to icav2 data folder you wish to download from
<download_path> Required, the local download directory, parent folder must exist
-w=<file_path>, --write-script-path=<file_path> Optional, write out a script instead of invoking aws s3 command
that holds the AWS S3 Sync parameters.
download path parameter parent folder does not necessarily need to exist if this option
is set. Parent folder of this parameter must exist.
-s=<s3_sync_arg>, --s3-sync-arg=<s3_sync_arg> Other arguments are sent to aws s3 sync, specify multiple times for multiple arguments
Environment variables:
ICAV2_BASE_URL Optional, default set as https://ica.illumina.com/ica/rest
ICAV2_PROJECT_ID Optional, taken from "$HOME/.icav2/.session.ica.yaml" if not set
ICAV2_ACCESS_TOKEN Required, taken from "$HOME/.icav2/.session.ica.yaml" if not set
Extras:
AWS CLI V2 must be installed
Due to permission requirements with aws s3 sync, you MUST have write permissions to this project to run this command
Examples: icav2 projectdata s3-sync-download /test_data/outputs/ $HOME/outputs/
icav2 projectdata s3-sync-download /test_data/outputs/ $HOME/outputs/ --s3-sync-arg --dryrun
icav2 projectdata s3-sync-download /test_data/outputs/ $HOME/outputs/ --s3-sync-arg --exclude='*' --s3-sync-arg --include='*.bam'
Use the <tab> command to traverse through files and folders as you would with a regular file system
Create a download script that can download the ICAv2 data from anywhere.
This creates a script from a template that houses the presigned urls.
Use the --public-key, or keybase options to encrypt the presigned urls in the script.
Usage:
icav2 projectdata create-download-script help
icav2 projectdata create-download-script <data_path>
(--name <name_of_script>)
[--output-directory /path/to/output/]
[--public-key <public_key_path> | --keybase-username <keybase_username> | --keybase-team <keybase_team} ]
[--file-regex <regex>]
Description:
Create a script to download a folder from icav2.
This will generate a script that contains a list of presigned urls to download from.
If a public key or keybase username is specified, the presigned urls will be encrypted.
Options:
<data_path> Required, path to icav2 data folder you wish to download from
--name <name> Descriptor of the data being downloaded
--output-directory </path/to/output/dir/> Directory to place the output file ($PWD by default)
--public-key </path/to/output/key.pem> Path to a public key (PEM) used to encrypt the data via
openssl pkeyutl -encrypt -keyform PEM -inkey /path/to/public-key
--keybase-username <your-username> Collects the public key for a keybase user to encrypt data
--keybase-team <your-team> Collects the public key for a keybase team to encrypt data
--file-regex <regex> Only add files to the download script that match a regex
Environment variables:
ICAV2_BASE_URL Optional, default set as https://ica.illumina.com/ica/rest
ICAV2_PROJECT_ID Optional, taken from "$HOME/.icav2/.session.ica.yaml" if not set
ICAV2_ACCESS_TOKEN Required, taken from "$HOME/.icav2/.session.ica.yaml" if not set
Extras:
If --keybase-username or --keybase-team is specified, one MUST have keybase cli installed.
If --public-key is specified, openssl must be installed.
Examples: icav2 projectdata create-download-script /test_data/outputs/
icav2 projectdata create-download-script /test_data/outputs/ --name test-data-outputs --keybase-username alexiswl --file-regex '*.bam'
🚧
Use the <tab> command to traverse through files and folders as you would with a regular file system