gedih3.gh3driver#

Functions#

`gh3_set_db_path`([gh3_root_dir])
`gh3_list_files`([gh3_root_dir])
`gh3_list_parts`([gh3_root_dir])
`gh3_read_meta`(var[, gh3_root_dir])
`gh3_select_partitions`(source[, region])	Return the H3 partition cell IDs of a database that may hold shots for `region`.
`gh3_write_meta`(opath, **kwargs)
`gh3_write_dataset_meta`(opath[, index_type, ...])	Write simplified metadata for extracted/aggregated datasets.
`gh3_part_from_df`(df)
`gh3_reindex`(df)
`gh3_aggregate_func`(df, res[, agg, cols])
`gh3_add_geometry`(df)
`gh3_build_bbox_index`([source])	Build the `_bbox_index.parquet` root sidecar for an H3 database.
`refresh_bbox_index_after_build`(gh3_dir[, enabled, logger])	Best-effort bbox-index (re)build for a just-completed build.
`invalidate_bbox_index`(gh3_dir)	Delete the bbox-index sidecar when partition data may have changed.
`gh3_load_hex`(d[, _bbox_index, part_col, _storage_cfg])
`gh3_load`([source, columns, region, query, from_map, ...])	Load H3-indexed GEDI data from any source.
`gh3_aggregate`(gh3_df[, target_res, agg, columns, ...])	Aggregate H3-indexed GEDI data to a coarser H3 resolution.
`gh3_export_part`(df, odir[, fmt, is_file_path, ...])	Export a single partition to file with a simple naming convention.
`gh3_export`(ddf, output[, fmt, merge, show_progress, ...])	Export a Dask DataFrame to simplified flat files with metadata.
`egi_load`([source, columns, region, query, ...])	Load EGI-indexed GEDI data from any source.
`egi_aggregate_func`(df, level[, agg, cols, x_col, y_col])	Aggregate H3-indexed DataFrame to EGI (EASE Grid Index) pixels.
`egi_add_geometry`(df[, polygons])	Add EGI pixel geometry to an EGI-indexed DataFrame.
`egi_aggregate`(gh3_df[, target_level, agg, columns, ...])	Aggregate GEDI data to EGI (EASE Grid Index) square pixels.
`egi_extract`(gh3_df[, index_level, partition_level, ...])	Extract H3-indexed GEDI data with EGI spatial indexing.
`egi_export_part`(df, odir[, fmt, is_file_path, ...])	Export a single EGI partition to file(s).
`is_egi_indexed`(df)	Check if a DataFrame is EGI-indexed.
`get_spatial_index_type`(df)	Determine the spatial index type of a DataFrame.
`gh3_to_raster`(gdf[, columns, output_path, compress])	Convert a single spatially-indexed GeoDataFrame to raster.
`gh3_rasterize`(data, output[, columns, merge, query, ...])	Rasterize a dataset or Dask DataFrame to GeoTIFF in a single call.
`gh3_rasterize_partitions`(ddf, output_dir[, columns, ...])	Rasterize Dask GeoDataFrame partitions to individual GeoTIFF files.
`gh3_sample_raster`(image_path[, data_source, region, ...])	Sample raster pixel values at GEDI shot locations.

Module Contents#

gedih3.gh3driver.gh3_set_db_path(gh3_root_dir=GH3_DEFAULT_H3_DIR)[source]#

gedih3.gh3driver.gh3_list_files(gh3_root_dir=GH3_DEFAULT_H3_DIR)[source]#

gedih3.gh3driver.gh3_list_parts(gh3_root_dir=GH3_DEFAULT_H3_DIR)[source]#

gedih3.gh3driver.gh3_read_meta(var, gh3_root_dir=GH3_DEFAULT_H3_DIR)[source]#

gedih3.gh3driver.gh3_select_partitions(source, region=None)[source]#

Return the H3 partition cell IDs of a database that may hold shots for region.

This is the canonical, overhang-safe way to determine which partition files a region touches, and the function external consumers should use when reading the H3 database directly (i.e. not through gh3_load()). It applies the same ring-1 expansion gh3_load uses internally.

H3 partitions are not geometrically inclusive of the shots stored in them: a shot is filed under cell_to_parent(latlng_to_cell(lon, lat, index_res), partition_res), whose polygon a boundary shot can sit outside of by up to ~0.18 x the partition’s edge length (~11 km at partition level 3). Selecting partitions by an exact polygon intersection therefore silently drops boundary shots (~5% of a boundary partition’s shots observed in production). The ring-1 expansion applied here closes that gap; it is guaranteed sufficient because the overhang is far smaller than one cell width. See the “Parent/Child Nesting Caveat” in the H3 Indexing docs for the full explanation.

Parameters:

sourcestr: Path to an H3 database (the directory holding gedih3_build_log.json) or a simplified dataset (the directory holding gedih3_dataset.json — H3 or EGI). Datasets answer through the same safety-gated selection the loaders use (_select_dataset_files()): when the sidecar cannot prove the filenames bound their files, every partition is returned rather than risking an under-selection.
regionstr | list | GeoDataFrame | GeoSeries | shapely geometry, optional: ROI as a vector-file path / "W,S,E,N" string, [W, S, E, N] bbox list, geopandas object, or shapely geometry (EPSG:4326). When None (default), every partition is returned.

Returns:

list[str]: Sorted partition IDs: H3 cell IDs at the database’s partition level, or the dataset’s file-basename partition IDs (H3 cells / EGI hashes). Empty when no dataset partition intersects the region.

Raises:

GediDatabaseNotFoundError: If source has no readable build log / partition list.

Examples

>>> ids = gh3_select_partitions('/data/h3db', region='roi.shp')
>>> ids[:2]
['83804cfffffffff', '838041fffffffff']
>>> # Map ids -> files when reading outside gh3_load:
>>> import glob
>>> files = [f for i in ids
...          for f in glob.glob(f'/data/h3db/h3_03={i}/year=*/*.parquet')]

gedih3.gh3driver.gh3_write_meta(opath, **kwargs)[source]#

gedih3.gh3driver.gh3_write_dataset_meta(opath, index_type='h3', index_level=None, columns=None, source_database=None, query_filter=None, tool=None, file_format='parquet', **kwargs)[source]#

Write simplified metadata for extracted/aggregated datasets.

This creates a single metadata file for user-friendly outputs (not hive-partitioned), making it easy to understand and use the data outside of gedih3 tools.

Parameters:

opathstr: Output directory path
index_typestr: Type of spatial index (‘h3’ or ‘egi’)
index_levelint: Resolution level of the index
columnslist: List of data columns
source_databasestr: Path to source H3 database (if applicable)
query_filterstr: Query string used for filtering
toolstr: Name of the tool that created this dataset
file_formatstr: Output file format (e.g. ‘parquet’, ‘feather’, ‘gpkg’)
**kwargs: Additional metadata to include

gedih3.gh3driver.gh3_part_from_df(df)[source]#

gedih3.gh3driver.gh3_reindex(df)[source]#

gedih3.gh3driver.gh3_aggregate_func(df, res, agg='mean', cols=None, **kwargs)[source]#

gedih3.gh3driver.gh3_add_geometry(df)[source]#

gedih3.gh3driver.gh3_build_bbox_index(source=None)[source]#

Build the _bbox_index.parquet root sidecar for an H3 database.

One row per partition year-file with the true data envelope derived from existing parquet row-group statistics — footer-only reads, no data scan (~minutes for a continental database; seconds with a dask client). Query paths (gh3_load with region=, egi_load) then skip files whose envelope cannot intersect the query a-priori.

Uses the registered dask client when one exists (parallel_map); otherwise a local thread pool — footer reads are tiny and I/O bound. Local databases only: the index is written at the database root.

Returns the index path.

gedih3.gh3driver.refresh_bbox_index_after_build(gh3_dir, enabled=True, logger=None)[source]#

Best-effort bbox-index (re)build for a just-completed build.

MUST run after the final build-log save: _load_bbox_index treats any index older than the log as stale, so an index written before the COMPLETED save would be silently ignored. The index is a derived artifact — failure here never fails the build (absence only costs speed); the warning points at the manual gh3_bbox_index recovery. Returns the index path, or None when disabled or failed.

gedih3.gh3driver.invalidate_bbox_index(gh3_dir)[source]#

Delete the bbox-index sidecar when partition data may have changed.

The producer-side half of the index contract: a stale index silently under-selects (the one unacceptable failure), an absent one only costs speed. Called by _merge_and_finalize (before its manifest write — unlinking a root sidecar bumps the root dir mtime) and by gh3_doctor --fix after any applied remedy. Idempotent; returns True when a sidecar was actually removed.

gedih3.gh3driver.gh3_load_hex(d, _bbox_index=None, part_col=None, _storage_cfg=None, **kwargs)[source]#

gedih3.gh3driver.gh3_load(source=None, *, columns=None, region=None, query=None, from_map=True, lazy=True, filters=None)[source]#

Load H3-indexed GEDI data from any source.

Auto-detects whether the source is an H3 database, simplified dataset, or parquet directory and loads accordingly.

Parameters:

sourcestr, optional: Path to data source (H3 database, simplified dataset, or parquet dir). If None, falls back to default H3 directory. Self-hosted S3 sources may carry their endpoint in the URL (s3://host:port/bucket/...); an endpoint already configured via configure_storage wins.
columnslist, optional: Columns to load.
regionstr | list | GeoDataFrame | GeoSeries | shapely geometry, optional: Spatial filter.
querystr, optional: Pandas query string for filtering.
from_mapbool: Use from_map loading for H3 databases (default True). DEPRECATED — from_map=False selects the original dask_geopandas.read_parquet path, which is unmaintained and slower on every axis (reads _metadata, no region bbox pushdown, no _bbox_index file skipping). The argument and that branch are slated for removal; leave it at the default.
lazybool: If True (default), return Dask DataFrame. If False, return computed pandas DataFrame.
filterslist or pyarrow.compute.Expression, optional: PyArrow predicate pushdown filters (conjunctive list of (column, op, value) tuples, a DNF list-of-lists, or an Expression), applied as per-file row-group pushdown during the read. Works for H3 databases and simplified parquet datasets, and combines (AND) with region when both are given — the region bbox prefilter stays on, both predicates are pushed to the same row-group stats layer. Predicate columns do not have to be listed in columns.

Returns:

dask GeoDataFrame or GeoDataFrame: Loaded data (lazy by default, eager if lazy=False).

Raises:

GediValidationError: If the source is an EGI-indexed dataset (use egi_load() instead).
GediDatabaseNotFoundError: If no valid data source is found.

Examples

>>> import gedih3.gh3driver as gh3
>>> ddf = gh3.gh3_load(
...     source='/path/to/h3_database',
...     columns=['agbd_l4a', 'rh_098_l2a'],
...     region='region.shp',
... )
>>> ddf.compute().head()

gedih3.gh3driver.gh3_aggregate(gh3_df, target_res=5, agg='mean', columns=None, query=None, add_geometry=True, repartition=False, partition_level=None, **kwargs)[source]#

Aggregate H3-indexed GEDI data to a coarser H3 resolution.

Uses map_partitions for efficient processing when data is loaded with from_map=True (each partition corresponds to a single H3 partition cell).

Parameters:

gh3_dfdask GeoDataFrame: H3-indexed GEDI data loaded via gh3_load()
target_resint: Target H3 resolution level (0-15, lower = coarser)
aggstr, list, dict, or callable: Aggregation specification (same as pandas groupby.agg)
columnslist, optional: Columns to aggregate (if None, all numeric columns)
querystr, optional: Pandas query string for filtering before aggregation
add_geometrybool: If True, add H3 polygon geometries to output
repartitionbool: If True, repartition by H3 partition column for export
partition_levelint, optional: Explicit H3 partition level. Used as fallback when the DataFrame lacks h3_XX columns (e.g., loaded from a simplified dataset).
**kwargs: Additional arguments passed to aggregation function

Returns:

dask GeoDataFrame: H3-indexed aggregated data.

Raises:

H3ValidationError: If target_res is not a valid H3 resolution (0–15).
GediAggregationError: If spatial aggregation fails.

gedih3.gh3driver.gh3_export_part(df, odir, fmt='parquet', is_file_path=False, part_col=None, group_by_partition=False, naming_partition_level=None)[source]#

Export a single partition to file with a simple naming convention.

Creates user-friendly output files named by partition ID (e.g., ‘abc123.parquet’), not hive-style directories.

Parameters:

dfDataFrame or GeoDataFrame: Data partition to export
odirstr: Output directory or file path
fmtstr: Output format (‘parquet’, ‘gpkg’, ‘geojson’, ‘csv’, etc.)
is_file_pathbool: If True, odir is treated as a complete file path
part_colstr, optional: Partition column name to use for naming. If None, auto-detect.
group_by_partitionbool: If True and part_col is specified, group data by partition column and write separate files for each unique partition ID within this Dask partition. Use this after shuffling data by partition column (via set_index) to ensure each unique partition ID is in exactly one Dask partition, avoiding file collision issues.
naming_partition_levelint, optional: H3 resolution level for deriving file names via cell_to_parent. Used when no partition column is available (e.g., aggregated data).

Returns:

str: Output file path(s). Comma-separated if multiple files written.

gedih3.gh3driver.gh3_export(ddf, output, fmt='parquet', merge=False, show_progress=True, drop_internal=False, write_metadata=True, source_database=None, tool=None, h3_partition_level=None, cog=True, **metadata_kwargs)[source]#

Export a Dask DataFrame to simplified flat files with metadata.

This is the high-level export function that encapsulates the full export pipeline: persist, write partition files, and write dataset metadata. It replaces the boilerplate pattern of map_partitions + persist + progress + gh3_write_dataset_meta.

Parameters:

ddfdask DataFrame or GeoDataFrame: Data to export. Should already be persisted if it represents an expensive computation (e.g., aggregation result).
outputstr: Output directory path
fmtstr: Output format (‘parquet’, ‘feather’, ‘gpkg’, etc.)
mergebool: If True, compute and write a single merged file instead of per-partition files.
show_progressbool: If True and a Dask distributed client is available, show progress bar.
drop_internalbool: If True, drop internal columns (h3_XX, egiXX, _egi_x/y, shot_number*) before export. Default False — internal columns are kept so downstream tools can join on shot_number or spatial indexes.
write_metadatabool: If True, write dataset metadata file.
source_databasestr, optional: Path to source H3 database (recorded in metadata).
toolstr, optional: Name of the tool creating this dataset (recorded in metadata).
h3_partition_levelint, optional: H3 resolution level to use for naming output files. When provided, files are named by the parent cell at this level (via h3.cell_to_parent). Useful for aggregated data where the original partition column was lost. If None, auto-detected from source_database metadata when available.
cogbool: For raster formats, write Cloud Optimized GeoTIFFs (default). Ignored for every other format.
**metadata_kwargs: Additional key-value pairs to include in the dataset metadata. Common keys: query_filter, aggregation, egi_index_level, egi_partition_level, h3_partition_level, image_source, etc.

Returns:

list of str: Paths to output files created.

Examples

>>> import gedih3.gh3driver as gh3
>>> ddf = gh3.gh3_load(source='/db', columns=['agbd_l4a'], region='roi.shp')
>>> gh3.gh3_export(ddf, '/tmp/test_export/')
>>>
>>> # Merged export
>>> gh3.gh3_export(ddf, '/tmp/merged/', merge=True)
>>>
>>> # With metadata
>>> gh3.gh3_export(ddf, '/tmp/out/', source_database='/db', tool='my_script',
...               query_filter='quality_flag == 1')

gedih3.gh3driver.egi_load(source=None, *, columns=None, region=None, query=None, index_level=1, partition_level=12, lazy=True, filters=None)[source]#

Load EGI-indexed GEDI data from any source.

Auto-detects whether the source is an H3 database (direct EGI loading) or a simplified EGI dataset and loads accordingly.

Parameters:

sourcestr, optional: Path to data source (H3 database or EGI dataset). If None, falls back to default H3 directory. Self-hosted S3 sources may carry their endpoint in the URL (s3://host:port/bucket/...); an endpoint already configured via configure_storage wins.
columnslist, optional: Columns to load.
regionstr | list | GeoDataFrame | GeoSeries | shapely geometry, optional: Spatial filter.
querystr, optional: Pandas query string for filtering.
index_levelint: EGI resolution level for fine indexing (1-12, default=1 ~1m). Only used when loading from H3 database.
partition_levelint: EGI level for output partitioning (1-12, default=12 ~160km). Only used when loading from H3 database.
lazybool: If True (default), return Dask DataFrame. If False, return computed pandas DataFrame.
filterslist or pyarrow.compute.Expression, optional: PyArrow predicate pushdown filters (conjunctive list of (column, op, value) tuples, a DNF list-of-lists, or an Expression) — same contract as gh3_load(filters=...). Pushed straight into each per-year parquet read, so row groups that cannot match are never decompressed, and ANDed with the tile bbox predicate when both apply. Unlike query, which filters in pandas after the read, the rows never enter worker memory. Predicate columns do not have to be listed in columns. A file whose schema lacks a predicate column raises rather than silently returning unfiltered rows.

Returns:

dask GeoDataFrame or GeoDataFrame: EGI-indexed data (lazy by default, eager if lazy=False).

Raises:

GediValidationError: If source is an H3 dataset (use gh3_load() instead).
GediDatabaseNotFoundError: If no valid data source is found.
EGIValidationError: If index_level or partition_level is outside [1, 12].

Examples

>>> import gedih3.gh3driver as gh3
>>> ddf = gh3.egi_load(
...     source='/path/to/h3_database',
...     columns=['agbd_l4a'],
...     region='region.shp',
...     index_level=1,
...     partition_level=12,
... )
>>> agg = gh3.egi_aggregate(ddf, target_level=6, agg='mean')

>>> # Predicate pushdown: quality-flag rows are dropped at the parquet
>>> # row-group layer, before anything reaches worker memory.
>>> ddf = gh3.egi_load(
...     source='/path/to/h3_database',
...     columns=['agbd_l4a'],
...     region='region.shp',
...     filters=[('l2_quality_flag_l4a', '==', 1), ('agbd_l4a', '>', 0)],
... )

gedih3.gh3driver.egi_aggregate_func(df, level, agg='mean', cols=None, x_col='lon_lowestmode', y_col='lat_lowestmode', **kwargs)[source]#

Aggregate H3-indexed DataFrame to EGI (EASE Grid Index) pixels.

This function converts H3-indexed GEDI data to EGI square pixels, which are compatible with GEDI L4B products and standard raster formats.

Parameters:

dfDataFrame or GeoDataFrame: H3-indexed GEDI data (GeoDataFrame with Point geometry preferred)
levelint: Target EGI resolution level (1-12)
aggstr, list, dict, or callable: Aggregation specification (same as pandas groupby.agg)
colslist, optional: Columns to aggregate (numeric columns only)
x_colstr: Longitude column name (default: ‘lon_lowestmode’). Only used if df is not a GeoDataFrame with Point geometry.
y_colstr: Latitude column name (default: ‘lat_lowestmode’). Only used if df is not a GeoDataFrame with Point geometry.
**kwargs: Additional arguments passed to aggregation function

Returns:

DataFrame or GeoDataFrame: EGI-indexed aggregated data

gedih3.gh3driver.egi_add_geometry(df, polygons=True)[source]#

Add EGI pixel geometry to an EGI-indexed DataFrame.

Parameters:

dfDataFrame: EGI-indexed DataFrame
polygonsbool: If True, use polygon geometries; if False, use centroids

Returns:

GeoDataFrame: GeoDataFrame with geometry column

gedih3.gh3driver.egi_aggregate(gh3_df, target_level=6, agg='mean', columns=None, query=None, add_geometry=True, x_col='lon_lowestmode', y_col='lat_lowestmode', partition_level=12, repartition=False, **kwargs)[source]#

Aggregate GEDI data to EGI (EASE Grid Index) square pixels.

Supports two input types:

EGI-indexed (from egi_load()): Fast path — no shuffle needed, aggregation is purely local within each partition.
H3-indexed (from gh3_load()): Shuffle path — data is repartitioned by EGI tiles before local aggregation.

Parameters:

gh3_dfdask GeoDataFrame: GEDI data loaded via egi_load() (EGI-indexed) or gh3_load() (H3-indexed)
target_levelint: Target EGI resolution level (1-12): - Level 6 (~1km): GEDI baseline - Level 7 (~2km): GEDI threshold - Level 8 (~10km): GEDI wall-to-wall
aggstr, list, dict, or callable: Aggregation specification (same as pandas groupby.agg)
columnslist, optional: Columns to aggregate (if None, all numeric columns)
querystr, optional: Pandas query string for filtering before aggregation
add_geometrybool: If True, add pixel polygon geometries to output
x_colstr: Longitude column name for coordinate lookup (shuffle path only)
y_colstr: Latitude column name for coordinate lookup (shuffle path only)
partition_levelint: EGI level for output partitioning and data shuffling (1-12, default=12 ~160km). Higher levels = coarser tiles = fewer unique keys = more efficient shuffle. Use smaller values for regions with many variables to reduce file sizes.
repartitionbool: If True, add partition column for organized export
**kwargs: Additional arguments passed to aggregation function

Returns:

dask GeoDataFrame: EGI-indexed aggregated data

gedih3.gh3driver.egi_extract(gh3_df, index_level=1, partition_level=12, query=None, add_geometry=True, x_col='lon_lowestmode', y_col='lat_lowestmode')[source]#

Extract H3-indexed GEDI data with EGI spatial indexing.

This function converts H3-indexed GEDI shots to EGI-indexed data without aggregation. It repartitions data by EGI tiles for efficient H3->EGI conversion.

Parameters:

gh3_dfdask GeoDataFrame: H3-indexed GEDI data loaded via gh3_load()
index_levelint: EGI resolution level for fine indexing (1-12, default=1 ~1m)
partition_levelint: EGI level for output file partitioning and shuffling (1-12, default=12 ~160km). Higher levels = coarser tiles = fewer unique keys = more efficient shuffle.
querystr, optional: Pandas query string for filtering before extraction
add_geometrybool: If True, add Point geometries to output (in WGS84/EPSG:4326)
x_colstr: Longitude column name for coordinate lookup
y_colstr: Latitude column name for coordinate lookup

Returns:

dask GeoDataFrame: EGI-indexed data with all original columns plus EGI index columns

gedih3.gh3driver.egi_export_part(df, odir, fmt='parquet', is_file_path=False, partition_level=12)[source]#

Export a single EGI partition to file(s).

Splits the data by partition tile and writes one file per unique tile. File names are the EGI hash of the partition tile at the requested level.

Parameters:

dfDataFrame or GeoDataFrame: EGI-indexed data partition
odirstr: Output directory or file path
fmtstr: Output format (‘parquet’, ‘gpkg’, ‘geojson’, ‘tif’, etc.)
is_file_pathbool: If True, odir is treated as a complete file path (single output)
partition_levelint: EGI level used for output file naming (1-12, default=12). Used as a fallback when no egiXX column is present in the DataFrame.

Returns:

str: Output file path(s) - comma-separated if multiple files written

gedih3.gh3driver.is_egi_indexed(df)[source]#

Check if a DataFrame is EGI-indexed.

Parameters:

dfDataFrame or GeoDataFrame: DataFrame to check

Returns:

bool: True if EGI-indexed, False otherwise

gedih3.gh3driver.get_spatial_index_type(df)[source]#

Determine the spatial index type of a DataFrame.

Parameters:

dfDataFrame or GeoDataFrame: DataFrame to check

Returns:

str: ‘h3’, ‘egi’, or None

gedih3.gh3driver.gh3_to_raster(gdf, columns=None, output_path=None, compress='LZW')[source]#

Convert a single spatially-indexed GeoDataFrame to raster.

Dispatches on the frame’s spatial index: H3 frames go through raster.h3_to_raster, EGI frames through egi.geodf_to_raster.

This handles ONE in-memory frame. To rasterize every partition of a Dask DataFrame — or a dataset directory — in a single call, use gh3_rasterize().

Parameters:

gdfGeoDataFrame: H3- or EGI-indexed GeoDataFrame.
columnslist of str, optional: Columns to rasterize. If None, all numeric columns.
output_pathstr, optional: If provided, save raster to this path
compressstr: Compression method for GeoTIFF

Returns:

xr.Dataset: Raster dataset. The CRS follows the index type: EPSG:4326 for H3, EPSG:6933 (EASE-Grid 2.0) for EGI. EGI rasters are never reprojected — that alignment is the reason EGI exists.

Raises:

GediValidationError: If the spatial index type cannot be determined from the frame.
GediRasterizationError: Propagated from the rasterizer. Notably, an EGI frame spanning more than one level-12 outer tile is refused rather than silently reduced — use gh3_rasterize(..., merge=True) for that.