Help ?¶
For each and every command, there is a --help option. Please consult it to
grasp the details for a command, its arguments and optional parameters, default
values and settings that can further shape the output.
For example,
Usage: rekx [OPTIONS] COMMAND [ARGS]...
๐พ ๐ฆ Rekx command line interface prototype
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ --version Show the version and โ
โ exit. โ
โ --log Enable logging. โ
โ --install-completion [bash|zsh|fish|powershe Install completion for โ
โ ll|pwsh] the specified shell. โ
โ [default: None] โ
โ --show-completion [bash|zsh|fish|powershe Show completion for the โ
โ ll|pwsh] specified shell, to โ
โ copy it or customize โ
โ the installation. โ
โ [default: None] โ
โ --help Show this message and โ
โ exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Diagnose chunking layout โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ inspect Inspect Xarray-supported data โ
โ shapes Diagnose chunking shapes in multiple Xarray-supported data โ
โ validate-json Validate chunk size consistency along multiple Kerchunk โ
โ reference files How to get available variables? โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Suggest chunking layout โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ suggest Suggest a good chunking shape, ex. '8784,2600,2600' โ
โ Needs a review! โ
โ suggest-alternative Suggest a good chunking shape Merge to suggest โ
โ suggest-symmetrical Suggest a good chunking shape Merge to suggest โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Rechunk data โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ modify-chunks Modify in-place the chunk size metadata in NetCDF files โ
โ Yet not implemented! โ
โ rechunk Rechunk data โ
โ rechunk-generator Generate variations of rechunking commands for multiple โ
โ files โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Create references โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ reference Create Kerchunk JSON reference files โ
โ reference-parquet Create Parquet references to an HDF5/NetCDF file โ
โ Merge to reference โ
โ reference-multi-parquet Create Parquet references to multiple HDF5/NetCDF โ
โ files Merge to reference-parquet โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Combine references โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ combine Combine Kerchunk reference sets (JSONs to JSON) โ
โ combine-to-parquet Combine Kerchunk reference sets into a single โ
โ Parquet store (JSONs to Parquet) โ
โ combine-parquet-stores Combine multiple Parquet stores (Parquets to โ
โ Parquet) โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Select from time series โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ select ๎ซฑ Select time series over a location โ
โ select-fast ๎ซฑ Bare read time series from Xarray-supported data and โ
โ optionally write to CSV โฒ Performance Test โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Select from Kerchunk references โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ select-json ๎ซฑ Select time series over a location from a JSON Kerchunk โ
โ reference set โ
โ select-parquet ๎ซฑ Select data from a Parquet references store โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ โฒ Read performance โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ read-performance ๎ซฑ Measure read and load in-memory operations of a point โ
โ time series from Xarray-supported data โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
The help for the command shapes
Usage: rekx shapes [OPTIONS] SOURCE_DIRECTORY
Diagnose chunking shapes in multiple Xarray-supported data
โญโ Time series โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * source_directory PATH Source directory path [default: None] โ
โ [required] โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ --pattern TEXT Filename pattern โ
โ to match โ
โ [default: *.nc] โ
โ --variable-set [all|coordinates Set of Xarray โ
โ |coordinates-wit variables to โ
โ hout-data|data|m diagnose โ
โ etadata|time] [default: โ
โ XarrayVariableSeโฆ โ
โ --validate-consiโฆ --no-validate-cโฆ [default: โ
โ no-validate-consโฆ โ
โ --common-shapes --no-common-shaโฆ Report common โ
โ maximum chunking โ
โ shape โ
โ [default: โ
โ no-common-shapes] โ
โ --help Show this message โ
โ and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
โญโ Input / Output โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ --csv PATH CSV output filename [default: None] โ
โ --verbose -v INTEGER Show details while executing commands โ
โ [default: 0] โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Verbosity¶
Most of the commands feature an extra --verbose or shortly -v flag. It'll
make rekx to be more communicative about what he did.
For example
check the difference of executing the same command
without -v
and with -v
๐ด Variables are not consistently shaped across all files!
SIS
Variable Shape Files
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
SIS 1 x 1 x 2600 SISin202001010000004231000101MA.nc
SIS 1 x 2600 x 2600 SISin200001010000004231000101MA_1_2600_2600.nc
Logging¶
rekx is growing and learning as we all do, by trial & error :-).
To get some background information on how rekx is crunching data,
we can instruct the --log option right before any subcommand :
SISin202001010000004231000101MA.nc
Vโฆ Shape Chunks Cโฆ Elโฆ Preโฆ โฆ Compโฆ Shโฆ Reaโฆ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Sโฆ 48 x 2600โฆ 1 x 1 โฆ 1โฆ 41โฆ 0.75 - zlib Faโฆ 0.0โฆ
File size: 181550165 bytes, Dimensions: time: 48, lon: 2600, bnds: 2, lat: 2600
* Cache: Size in bytes, Number of elements, Preemption ranging in [0, 1]
A .log is created
containing timestamped details
on the execution of important commands and their output.
Example :