gedih3.cliutils#

Attributes#

`VALID_FORMATS`
`PIPELINE_FORMATS`
`FORMAT_EXTENSIONS`
`INTERNAL_COLUMN_PATTERNS`

Functions#

`detect_dataset_format`(dataset_path)	Detect the file format of a simplified dataset.
`list_dataset_files`(dataset_path[, fmt])	List data files in a simplified dataset directory.
`read_dataset_schema`(filepath, fmt)	Read column names and geometry flag from a dataset file without loading data.
`make_dataset_reader`(fmt[, columns, geo])	Return a callable that reads a single file into a DataFrame or GeoDataFrame.
`add_dask_args`(parser[, profile])	Add Dask-related arguments to an argument parser.
`add_verbosity_args`(parser)	Add verbosity-related arguments to an argument parser.
`add_storage_args`(parser)	Add remote storage credential arguments to an argument parser.
`setup_storage`(args[, logger])	Call `configure_storage()` from parsed CLI arguments.
`add_product_args`(parser[, include_detail_level])	Add GEDI product variable arguments to an argument parser.
`parse_egi_levels`(value)	Parse EGI argument in format 'level' or 'level:partition'.
`setup_logging`(args[, name])	Configure logging based on verbosity flags and return a logger.
`print_banner`(title[, version, logger])	Print a tool banner with centered title.
`print_success`(message[, logger])	Print a success message with banner formatting.
`resolve_path_args`(args, names[, logger])	Absolutize each path-bearing CLI arg listed in `names` on the driver.
`progress_iter`(iterable, *, desc[, total, args, unit])	tqdm wrapper with consistent style + safe logger interleaving.
`cli_exception_handler`(args[, logger])	Standard exception handling context manager for CLI tools.
`configure_database_path`(args[, logger])	Configure database path from args or default.
`get_dataset_index_info`(database)	Get spatial index information from a dataset or database.
`load_data_from_source`(database[, columns, region, ...])	Load data from H3 database, simplified dataset, or parquet directory.
`is_internal_column`(col_name)	Check if a column name matches internal/partition column patterns.
`filter_data_columns`(columns[, exclude_geometry])	Filter out internal/partition columns from a column list.
`get_numeric_columns`(ddf[, exclude_internal])	Get list of numeric columns from a Dask DataFrame.
`get_rasterizable_columns`(ddf)	Get columns suitable for rasterization from a Dask DataFrame.
`get_aggregatable_columns`(df)	Get numeric columns suitable for aggregation from a DataFrame.
`filter_raster_columns`(columns, geodf)	Filter columns suitable for rasterization, excluding internal columns.
`h3_col_name`(level)	Get H3 column name for a given resolution level.
`find_coordinate_column`(columns, base_name)	Find a coordinate column by base name, handling product suffixes.
`parse_aggregation`(agg_str)	Parse aggregation spec from CLI string, JSON file, or text file.
`parse_file_format`(args[, default])
`resolve_product_vars`(args)	Resolve product variables from CLI args.
`parse_gedi_args`(args)	Parse GEDI product args from CLI (legacy wrapper around resolve_product_vars).
`parse_dask_args`(args)
`format_dask_cluster_info`(→ str)	One-line summary of a live dask Client's cluster — workers, total threads, total RAM.
`parse_region`(region_str)	Parse region argument into GeoDataFrame or bbox
`collect_columns`(args[, available_columns])	Collect all requested variables from command line arguments and validate against available columns.
`get_product_quality_conditions`(selected_products, ...)	Return quality filter conditions for the selected products and GEDI version.
`build_query_string`(args[, available_columns])	Build pandas query string from arguments
`safe_query`(df, query_str)	Apply a pandas query, handling column names with '/' that pandas can't parse.

Module Contents#

gedih3.cliutils.VALID_FORMATS = ['parquet', 'feather', 'shp', 'geojson', 'gpkg', 'txt', 'csv', 'h5', 'hdf5']#

gedih3.cliutils.PIPELINE_FORMATS#

gedih3.cliutils.FORMAT_EXTENSIONS#

gedih3.cliutils.detect_dataset_format(dataset_path)[source]#

Detect the file format of a simplified dataset.

Checks DATASET_META_FILENAME for ‘file_format’ field first, then scans directory for known extensions. Defaults to ‘parquet’ for backwards compat.

Parameters:

dataset_pathstr: Path to the dataset directory

Returns:

str: Detected format (‘parquet’, ‘feather’, or ‘gpkg’)

Raises:

GediValidationError: If detected format is not in PIPELINE_FORMATS

gedih3.cliutils.list_dataset_files(dataset_path, fmt=None)[source]#

List data files in a simplified dataset directory.

Parameters:

dataset_pathstr: Path to the dataset directory
fmtstr, optional: File format. If None, auto-detected via detect_dataset_format().

Returns:

list of str: Sorted list of file paths

Raises:

FileNotFoundError: If no matching files found

gedih3.cliutils.read_dataset_schema(filepath, fmt)[source]#

Read column names and geometry flag from a dataset file without loading data.

Parameters:

filepathstr: Path to a single data file
fmtstr: File format (‘parquet’, ‘feather’, or ‘gpkg’)

Returns:

tuple: (column_names: list[str], has_geometry: bool)

gedih3.cliutils.make_dataset_reader(fmt, columns=None, geo=True)[source]#

Return a callable that reads a single file into a DataFrame or GeoDataFrame.

The returned callable supports column selection at read time.

Parameters:

fmtstr: File format (‘parquet’, ‘feather’, or ‘gpkg’)
columnslist, optional: Columns to load
geobool: If True, use geopandas readers (for files with geometry metadata). If False, use pandas readers (for files without geometry metadata).

Returns:

callable: f(filepath) -> DataFrame or GeoDataFrame

gedih3.cliutils.add_dask_args(parser, profile=None)[source]#

Add Dask-related arguments to an argument parser.

Parameters:

parserargparse.ArgumentParser: The argument parser to add arguments to.
profilestr, optional: Resource profile hint. 'build' uses fewer workers with more memory (suitable for HDF5→parquet pipelines). None uses generic defaults.

gedih3.cliutils.add_verbosity_args(parser)[source]#: Add verbosity-related arguments to an argument parser.

gedih3.cliutils.add_storage_args(parser)[source]#

Add remote storage credential arguments to an argument parser.

Covers S3, HTTP/HTTPS, FTP, and SFTP/SSH protocols.

gedih3.cliutils.setup_storage(args, logger=None)[source]#

Call configure_storage() from parsed CLI arguments.

Should be called early in main() before any data access.

gedih3.cliutils.add_product_args(parser, include_detail_level=True)[source]#

Add GEDI product variable arguments to an argument parser.

Supports two mutually exclusive modes: 1. Global: --detail-level <level> applies a detail level to ALL products. 2. Per-product: -l2a <...> -l4a <...> selects specific products.

Per-product flags use nargs='*' so a bare flag (e.g. -l2a) means “dump everything from L2A”, while -l2a default uses the standard variable set.

Parameters:

parserargparse.ArgumentParser
include_detail_levelbool: Include --detail-level flag (only useful for download/build).

gedih3.cliutils.parse_egi_levels(value)[source]#

Parse EGI argument in format ‘level’ or ‘level:partition’.

This function is used by CLI tools to parse EGI level arguments that specify both an index/aggregation level and an optional output partition level.

EGI levels: 1 = finest (~1m), 12 = coarsest (~160km) Note: This is opposite to H3 where higher numbers mean finer resolution.

Examples:: ‘1’ -> (1, 12) # Level 1, partition at level 12 (default) ‘1:12’ -> (1, 12) # Explicit level:partition ‘6:10’ -> (6, 10) # Level 6, partition at level 10

Parameters:

valuestr or None: EGI level specification string

Returns:

tuple or None: (level, partition_level) tuple, or None if value is None

Raises:

argparse.ArgumentTypeError: If the value cannot be parsed or levels are invalid

gedih3.cliutils.setup_logging(args, name=None)[source]#

Configure logging based on verbosity flags and return a logger.

Also configures Dask warning suppression for non-DEBUG modes.

Args:: args: Parsed arguments with ‘quiet’ and ‘verbose’ attributes name: Logger name (defaults to calling module’s __name__)
Returns:: Configured logger instance

gedih3.cliutils.print_banner(title, version=None, logger=None)[source]#

Print a tool banner with centered title.

Args:: title: Tool title string version: Package version (if None, imports from gedih3) logger: Logger to use (if None, uses print)

gedih3.cliutils.print_success(message, logger=None)[source]#: Print a success message with banner formatting.

gedih3.cliutils.resolve_path_args(args, names, logger=None)[source]#

Absolutize each path-bearing CLI arg listed in names on the driver.

Tools that dispatch work to dask workers (gh3_extract, gh3_aggregate, gh3_from_img, gh3_from_polygon) pass paths from args verbatim into map_partitions / client.map. Workers then resolve any relative path against their own CWD — which on a remote scheduler is almost never the same as the user’s interactive shell. The result is silent and brutal: workers write to (or read from) the wrong location with no error surfaced until much later (empty output dir / “FileNotFoundError” mid-aggregation). Absolutizing on the driver before dispatch closes the footgun.

Skips: None / empty, already-absolute paths, and URL-style strings (http://, s3://, /vsicurl/, /vsis3/, gs://, gcs://, /vsigs/). Callers should only pass arg names that are known to be paths — keeps bbox / country-code / query args (region, etc.) untouched.

gedih3.cliutils.progress_iter(iterable, *, desc, total=None, args=None, unit='it')[source]#

tqdm wrapper with consistent style + safe logger interleaving.

Yields a tqdm-wrapped iterator. logging_redirect_tqdm is used so that logger.info / logger.warning calls made from inside the loop body do not clobber the bar.

Parameters:

iterableIterable: The items to iterate over.
descstr: Description shown to the left of the bar.
totalint, optional: Total item count. Inferred via len(iterable) when possible.
argsargparse.Namespace, optional: CLI args; used to read --quiet defensively via getattr(args, 'quiet', False). Pass None to always show.
unitstr: Unit label for the bar (default 'it').

gedih3.cliutils.cli_exception_handler(args, logger=None)[source]#

Standard exception handling context manager for CLI tools.

Provides consistent error handling across CLI tools: - KeyboardInterrupt: Clean exit with message - Other exceptions: Print error message, optionally show traceback in verbose mode

Args:: args: Parsed arguments with ‘verbose’ attribute for traceback control logger: Optional logger (not currently used, reserved for future use)

Usage:

with cli_exception_handler(args):
    # CLI main logic here
    pass

Example:

def main():
    args = get_cmd_args()
    with cli_exception_handler(args):
        # Main CLI logic
        client = Client(**dask_kwargs)
        ...

gedih3.cliutils.configure_database_path(args, logger=None)[source]#

Configure database path from args or default.

Args:: args: Parsed arguments with ‘database’ attribute logger: Optional logger for output
Returns:: Database path string

gedih3.cliutils.get_dataset_index_info(database)[source]#

Get spatial index information from a dataset or database.

Reads metadata to determine the index type (h3 or egi) and level.

Parameters:

databasestr: Path to H3 database or simplified dataset directory

Returns:

dict: Dictionary with keys: - ‘source_type’: ‘h3_database’, ‘simplified_dataset’, or ‘parquet_directory’ - ‘index_type’: ‘h3’ or ‘egi’ (or None if unknown) - ‘index_level’: int (or None if unknown) - ‘partition_level’: int (or None if not applicable) - Other metadata fields from the source

gedih3.cliutils.load_data_from_source(database, columns=None, region=None, query=None, logger=None)[source]#

Load data from H3 database, simplified dataset, or parquet directory.

Auto-detects the data source type and loads accordingly. Delegates to gh3_load() or egi_load() based on index type.

Args:: database: Path to database directory columns: Columns to load region: Spatial filter (GeoDataFrame or bbox) query: Query string for filtering logger: Optional logger for output
Returns:: Dask GeoDataFrame

gedih3.cliutils.INTERNAL_COLUMN_PATTERNS = ['^h3_\\d{2}$', '^egi\\d{2}$', '^_egi_[xy]$', '^shot_number']#

gedih3.cliutils.is_internal_column(col_name)[source]#

Check if a column name matches internal/partition column patterns.

Internal columns include H3 partition columns (h3_XX), EGI index columns (egiXX), internal EGI coordinates (_egi_x, _egi_y), and shot identifiers.

Args:: col_name: Column name to check
Returns:: True if column is internal, False otherwise

gedih3.cliutils.filter_data_columns(columns, exclude_geometry=True)[source]#

Filter out internal/partition columns from a column list.

Args:: columns: List of column names exclude_geometry: If True, also exclude ‘geometry’ column
Returns:: List of user data columns (excluding internal columns)

gedih3.cliutils.get_numeric_columns(ddf, exclude_internal=True)[source]#

Get list of numeric columns from a Dask DataFrame.

Args:: ddf: Dask DataFrame exclude_internal: If True (default), exclude internal/partition columns
Returns:: List of column names with numeric dtypes

gedih3.cliutils.get_rasterizable_columns(ddf)[source]#

Get columns suitable for rasterization from a Dask DataFrame.

This is a convenience function that returns numeric columns excluding internal columns (h3_XX, egiXX, etc.) and geometry.

Args:: ddf: Dask DataFrame
Returns:: List of column names suitable for rasterization

gedih3.cliutils.get_aggregatable_columns(df)[source]#

Get numeric columns suitable for aggregation from a DataFrame.

This is a convenience function that returns numeric columns excluding internal/partition columns (h3_XX, egiXX, _egi_x, _egi_y, shot_number, etc.).

This encapsulates the common pattern:: numeric_cols = df.select_dtypes(include=[np.number]).columns.tolist() filtered_cols = filter_data_columns(numeric_cols)
Args:: df: DataFrame, GeoDataFrame, or Dask DataFrame
Returns:: List of column names suitable for aggregation

gedih3.cliutils.filter_raster_columns(columns, geodf)[source]#

Filter columns suitable for rasterization, excluding internal columns.

Internal columns (egi indices, h3 indices, shot_number) should not be rasterized as bands - they’re metadata, not data values. Also excludes the index column since it will become a column after reset_index().

Args:: columns: List of column names to filter, or None to auto-detect numeric columns geodf: GeoDataFrame to get numeric columns and index name from
Returns:: List of filtered column names suitable for rasterization, or None if empty

gedih3.cliutils.h3_col_name(level)[source]#

Get H3 column name for a given resolution level.

Args:: level: H3 resolution level (0-15)
Returns:: Column name string, e.g. ‘h3_06’ for level 6

gedih3.cliutils.find_coordinate_column(columns, base_name)[source]#

Find a coordinate column by base name, handling product suffixes.

In the H3 database, coordinate columns may have product suffixes (e.g., ‘lon_lowestmode_l2a’ instead of ‘lon_lowestmode’).

Args:

columns: list-like of available column names base_name: Base column name to search for (e.g., ‘lon_lowestmode’)

Returns:

Actual column name if found, None otherwise

Examples:

>>> find_coordinate_column(['lon_lowestmode_l2a', 'lat_lowestmode_l2a'], 'lon_lowestmode')
'lon_lowestmode_l2a'
>>> find_coordinate_column(['lon', 'lat'], 'lon')
'lon'

gedih3.cliutils.parse_aggregation(agg_str)[source]#

Parse aggregation spec from CLI string, JSON file, or text file.

Supports:

Single function: ‘mean’ → ‘mean’

Percentile shorthand: ‘p25’, ‘p50’, ‘p95’ → named percentile callable

List of functions: “[‘mean’, ‘std’, ‘p25’, ‘p75’]” → mixed list

Column-specific dict: “{‘col’:[‘mean’,’p50’]}” → dict with callables

JSON file (.json): parsed as dict or list

Text file: one function name per line → list (single line → string)

Percentile patterns (p25, p90, etc.) are expanded into named callable functions that work with pandas .agg() and produce clean column names like ‘agbd_l4a_percentile_25’.

gedih3.cliutils.parse_file_format(args, default='parquet')[source]#

gedih3.cliutils.resolve_product_vars(args)[source]#

Resolve product variables from CLI args.

Two modes (mutually exclusive for L2A/L2B/L4A/L4C): 1. Global: --detail-level <level> applies level to L2A/L2B/L4A/L4C (not L1B). 2. Per-product: -l2a <...> -l4a <...> selects specific products.

L1B is always specified separately via -l1b and can be combined with either mode.

Per-product flag semantics (with nargs='*'): - Flag absent → args.l2a = None (product not selected) - Bare flag (-l2a) → args.l2a = [] (dump everything) - With keyword: -l2a default → ['default'] - With vars: -l2a rh agbd → ['rh', 'agbd']

Returns:

dict: Product code → variable list mapping.

Raises:

GediValidationError: If --detail-level is combined with non-L1B per-product flags.

gedih3.cliutils.parse_gedi_args(args)[source]#

Parse GEDI product args from CLI (legacy wrapper around resolve_product_vars).

Supports both the new -l global flag and legacy per-product flags. Falls back to legacy parsing when detail_level attribute is absent.

gedih3.cliutils.parse_dask_args(args)[source]#

gedih3.cliutils.format_dask_cluster_info(client) → str[source]#

One-line summary of a live dask Client’s cluster — workers, total threads, total RAM.

Use after Client(…) is up so that external –dask-scheduler connections report the cluster’s real shape instead of the local-cluster CLI defaults.

gedih3.cliutils.parse_region(region_str: str | None)[source]#: Parse region argument into GeoDataFrame or bbox

gedih3.cliutils.collect_columns(args, available_columns=None)[source]#: Collect all requested variables from command line arguments and validate against available columns. Returns: (column_list, product_map)

gedih3.cliutils.get_product_quality_conditions(selected_products, version, available_columns)[source]#

Return quality filter conditions for the selected products and GEDI version.

Parameters:

selected_productslist[str]: Product keys that were requested (e.g. [‘L2A’, ‘L4A’]).
versionint or None: GEDI data version from build log (e.g. 2 or 3). None defaults to 2.
available_columnslist[str]: Columns present in the database (suffixed form e.g. ‘quality_flag_l2a’).

Returns:

list[tuple[str, str]]: Pairs of (column_name, condition_str) to use in the quality filter query, e.g. [('quality_flag_l2a', '== 1'), ('degrade_flag_l2a', '== 0')].

gedih3.cliutils.build_query_string(args, available_columns=None)[source]#: Build pandas query string from arguments

gedih3.cliutils.safe_query(df, query_str)[source]#

Apply a pandas query, handling column names with ‘/’ that pandas can’t parse.

pandas.DataFrame.query() fails on backtick-quoted names containing ‘/’ because it converts the slash to an unresolvable internal token. This function works around the limitation by temporarily renaming such columns.