gedih3.gh3builder#

Attributes#

logger

Functions#

`download_soc`(product_vars[, spatial, temporal, ...])	Download GEDI HDF5 files in SOC (Science Operation Center) format.
`s3_etl_subset`(product_vars[, spatial, temporal, ...])	ETL-style S3 build: search for GEDI granules, then dispatch Dask workers
`h3_part_files`(df, dir_path[, res, part, lat_col, ...])	Write a DataFrame to H3-partitioned parquet files.
`h3_write_metadata`(h3_file[, stats])	Write a sidecar metadata JSON file for an H3 partition parquet file.
`h3_read_metadata`(h3_file)	Read the sidecar metadata JSON for an H3 partition parquet file.
`h3_merge_metadata`(h3_subdir)	Merge per-year metadata files into a single summary for an H3 partition.
`h3_skip_part`(h3_dir, h3_part, gedi_file[, cols])	Check whether an H3 partition already contains data from a GEDI granule.
`h3_add_skip_column`(df, h3_dir)	Add a `_skip` boolean column indicating partitions to skip.
`dh3_merge_metadata`(h3_subdir)
`h3_merge_files`(in_dir, out_dir[, rm_src, replace])	Merge multiple parquet files for an H3 partition into a single file.
`dh3_merge_files`(in_dir, out_dir[, rm_src, replace])
`preclean_merge_failures`(→ Dict[str, int])	L1 resume pre-clean: act on every recorded `_merge_failures` sentinel.
`apply_merge_failures_to_logger`(→ int)	Fold the merge-failed-granules sidecar into the build-log's
`build_h3db`(, h3_dir, skip_granules, status_callback, ...)	Build an H3-indexed GEDI database from local SOC files or S3 download.
`merge_build_logs`(→ dict)	Merge two build log files from separate databases.

Module Contents#

gedih3.gh3builder.logger#

gedih3.gh3builder.download_soc(product_vars: Dict, spatial=None, temporal=None, direct_access=False, update=False, version=None, odir=GH3_DEFAULT_SOC_DIR, n_jobs=5, on_granule_complete=None, ensure_l2a=True)[source]#

Download GEDI HDF5 files in SOC (Science Operation Center) format.

Expands variable specifications, ensures L2A essentials and shot_number are included, then delegates to gedi_download for retrieval.

Parameters:

product_varsdict: Mapping of GEDI product codes (e.g., ‘L2A’, ‘L4A’) to variable lists. Accepts ‘default’, ‘minimal’, or explicit variable names.
spatialGeoDataFrame, list, or str, optional: Spatial filter (vector file, bbox as [W,S,E,N], or ISO3 code).
temporaltuple of str, optional: Temporal range as (start_date, end_date) in ‘YYYY-MM-DD’ format.
direct_accessbool, default False: If True, use S3 streaming instead of downloading to disk.
updatebool, default False: If True, resume a previous download (skip already-downloaded files).
versionint or str, optional: GEDI data version (e.g., 2). If None, uses latest available.
odirstr: Output directory for downloaded HDF5 files.
n_jobsint, default 5: Number of parallel download workers.
on_granule_completecallable, optional: Callback (granule_info_dict, status_str) -> None for per-granule progress tracking. Passed through to gedi_download(); see its docstring for the granule_info_dict contract (includes a path key).
ensure_l2abool, default True: If True, automatically add L2A essentials when L2A is not in product_vars. Set to False for variable-only updates where L2A data already exists in the target database.

Returns:

list: List of downloaded SOC file paths or EarthAccessFile objects.

gedih3.gh3builder.s3_etl_subset(product_vars, spatial=None, temporal=None, version=None, odir=None, ensure_l2a=True)[source]#

ETL-style S3 build: search for GEDI granules, then dispatch Dask workers to open remote HDF5 files, extract only selected variables via range requests, and write compact local HDF5 files.

This transfers and stores significantly less data than full downloads (10-50x smaller depending on variable selection). Workers run as true separate processes via Dask (GIL-free), matching the parallelism of download mode.

Parameters:

product_varsdict: Mapping of GEDI product codes (e.g., ‘L2A’, ‘L4A’) to variable lists. Accepts ‘default’, ‘minimal’, or explicit variable names.
spatialGeoDataFrame, list, or str, optional: Spatial filter (vector file, bbox as [W,S,E,N], or ISO3 code).
temporaltuple of str, optional: Temporal range as (start_date, end_date) in ‘YYYY-MM-DD’ format.
versionint or str, optional: GEDI data version. If None, uses latest available.
odirstr: Output directory for compact local HDF5 files in SOC structure.
ensure_l2abool, default True: If True, automatically add L2A essentials when L2A is not in product_vars. Set to False for variable-only updates where L2A data already exists in the target database.

Returns:

str: Path to the output directory containing compact HDF5 files.

Raises:

GediFileError: If no GEDI files are found on S3 for the given parameters.

gedih3.gh3builder.h3_part_files(df, dir_path, res=12, part=3, lat_col='lat_lowestmode', lon_col='lon_lowestmode', roi_tiles=[])[source]#

Write a DataFrame to H3-partitioned parquet files.

Indexes the DataFrame by H3 cell, groups rows by partition cell, and writes each group as a separate parquet file. If a file already exists for a partition, new rows are appended.

Parameters:

dfpandas.DataFrame: Input DataFrame with coordinate columns and root_beam, root_file metadata columns.
dir_pathstr: Base directory for output partition subdirectories.
resint, default 12: H3 resolution for shot-level indexing.
partint, default 3: H3 resolution for file partitioning.
lat_colstr, default ‘lat_lowestmode’: Column name for latitude values.
lon_colstr, default ‘lon_lowestmode’: Column name for longitude values.
roi_tileslist of str, optional: If non-empty, only write partitions whose H3 cell ID is in this list.

Returns:

list of str or None: List of written parquet file paths, or None if the input is empty.

gedih3.gh3builder.h3_write_metadata(h3_file, stats=None)[source]#

Write a sidecar metadata JSON file for an H3 partition parquet file.

Parameters:

h3_filestr: Path to the H3 partition parquet file.
statsdict, optional: Pre-computed stats from parquet_merge_files’s streaming pass: {'shot_count', 'shot_min', 'shot_max', 'dt_min', 'dt_max', 'root_files'}. When provided, skips the ~1.5-2 GB pd.read_parquet re-read of the just-written merged file. When None or any field is None, falls back to reading the columns from disk (slower, more memory).

Returns:

str: Path to the written metadata JSON file (*PARTITION_META_FILENAME).

gedih3.gh3builder.h3_read_metadata(h3_file)[source]#

Read the sidecar metadata JSON for an H3 partition parquet file.

Parameters:

h3_filestr: Path to the H3 partition parquet file. The metadata file is expected at the same path with a PARTITION_META_FILENAME extension.

Returns:

dict or None: Parsed metadata dictionary, or None if the metadata file does not exist.

gedih3.gh3builder.h3_merge_metadata(h3_subdir)[source]#

Merge per-year metadata files into a single summary for an H3 partition.

Aggregates shot counts, expands shot and date ranges, and deduplicates granule identifiers across all year subdirectories within one H3 cell.

Parameters:

h3_subdirstr: Path to an H3 partition directory (e.g., h3_03=<cell_id>/) containing year subdirectories with parquet and metadata files.

Returns:

str or None: Path to the merged metadata JSON file, or None if no metadata files were found.

gedih3.gh3builder.h3_skip_part(h3_dir, h3_part, gedi_file, cols=None)[source]#

Check whether an H3 partition already contains data from a GEDI granule.

Reads the partition’s merged metadata to determine if the granule has already been indexed and, optionally, if the requested columns are already present.

Parameters:

h3_dirstr: Root directory of the H3 database.
h3_partstr: H3 cell ID of the partition to check.
gedi_filestr: Path or filename of the GEDI HDF5 file to test against.
colslist of str, optional: If provided, also verify that these columns already exist in the partition. The partition is only skipped when both the granule is present and all requested columns are available.

Returns:

bool: True if the partition should be skipped (granule already indexed and columns present), False otherwise.

gedih3.gh3builder.h3_add_skip_column(df, h3_dir)[source]#

Add a _skip boolean column indicating partitions to skip.

Designed for use with dask.dataframe.map_partitions. Checks each unique H3 partition cell individually, since a single beam can span multiple H3 cells. Delegates to h3_skip_part per cell.

Parameters:

dfpandas.DataFrame: A single Dask partition containing H3 partition column(s) (h3_XX) and root_file_l2a.
h3_dirstr: Root directory of the H3 database.

Returns:

pandas.DataFrame: Input DataFrame with an added _skip column (True if the cell’s granule data already exists in the database).

gedih3.gh3builder.dh3_merge_metadata(h3_subdir)#

gedih3.gh3builder.h3_merge_files(in_dir, out_dir, rm_src=True, replace=False)[source]#

Merge multiple parquet files for an H3 partition into a single file.

Reads all parquet files in in_dir, merges them (deduplicating shots when appending to an existing file), writes the result to out_dir, and generates a sidecar metadata JSON file.

Parameters:

in_dirstr: Input directory containing parquet fragment files for one H3 cell/year combination.
out_dirstr: Output directory where the merged file will be written, preserving the <h3_cell>/<year>/ structure.
rm_srcbool, default True: If True, remove the source directory after a successful merge.
replacebool, default False: If True, overwrite existing output files. If False, merge new data into any existing output file.

Returns:

str or None: Path to the merged parquet file, or None if in_dir contained no parquet files.

gedih3.gh3builder.dh3_merge_files(in_dir, out_dir, rm_src=True, replace=False)#

gedih3.gh3builder.preclean_merge_failures(tmp_dir: str) → Dict[str, int][source]#

L1 resume pre-clean: act on every recorded _merge_failures sentinel.

For each partition the prior run’s merge marked failed:

Stat each *.parquet in the partition — unlink if 0-byte (the canonical class B artifact). Truncated-but-nonzero parquets (class C) are unlinked only when pq.ParquetFile cannot open them — checked cheaply by opening the footer once. The cost is bounded by the failure list, never the full healthy tree.
Unlink any co-located *.tmp / *.merge.tmp siblings. These survive SIGKILL when AtomicFileWriter.__exit__ never runs and would otherwise pollute future merges.
Delete the failure sentinel itself so re-running the pre-clean is idempotent (next run only re-acts on freshly-failed merges).

Returns {'partitions_cleaned': N, 'parquets_removed': N, 'tmps_removed': N}. Companion to apply_merge_failures_to_logger — calling this without the granule flip-back would unlink fragments and leave their granules marked INDEXED, permanently dropping rows. The CLI runs both together.

gedih3.gh3builder.apply_merge_failures_to_logger(h3_logger, tmp_dir: str) → int[source]#

Fold the merge-failed-granules sidecar into the build-log’s granule status (INDEXED → MERGE_FAILED). Returns the count flipped.

Idempotent: re-applying after a successful resume is a no-op because re-extracted granules have already been flipped back to INDEXED by _reconcile_granules_from_disk. Truncates the sidecar after fold to keep it bounded across resumes.

Callable from the CLI right after _merge_and_finalize returns, regardless of which code path invoked merge. Does NOT call h3_logger.save_log — the caller controls save cadence.

gedih3.gh3builder.build_h3db(product_vars: Dict[str, List[str]], res: int = 12, part: int = 3, spatial=None, temporal=None, soc_source: str | List | None = None, version: int | None = None, tmp_dir: str = os.path.join(GH3_DEFAULT_TMP_DIR, 'gh3_build'), h3_dir: str = GH3_DEFAULT_H3_DIR, skip_granules: List[Dict] | None = None, status_callback: Callable[[str], None] | None = None, variable_only_update: bool = False, exclude: List[str] | None = None) → List[str] | None[source]#

Build an H3-indexed GEDI database from local SOC files or S3 download.

Parameters:

product_varsdict

Dictionary mapping GEDI product codes (e.g., ‘L2A’, ‘L4A’) to lists of variable names to extract. Use ‘default’, ‘minimal’, or explicit lists.

resint, default 12

H3 resolution level for indexing individual shots (0-15).

partint, default 3

H3 resolution level for file partitioning (0-15, must be <= res).

spatialGeoDataFrame, list, or str, optional

Spatial filter for the region of interest.

temporaltuple, optional

Temporal filter as (start_date, end_date) in ‘YYYY-MM-DD’ format.

soc_sourcestr, list, or None

None: download from NASA S3 to a temp directory (cleaned up after build)
str: path to local directory containing GEDI SOC HDF5 files
list: pre-acquired list of file paths or EarthAccessFile objects

versionint or None

GEDI data version. If None, uses latest available. Also used to filter local files by version when soc_source is a directory.

tmp_dirstr

Path to temporary directory for intermediate files.

h3_dirstr

Output path for the H3-indexed parquet database.

skip_granuleslist of dict, optional

List of granule identifiers to skip (from previous builds).

status_callbackcallable, optional

Called with status string at pipeline stage transitions.

variable_only_updatebool, default False

If True, only add new variable columns to existing partition files via shot_number join (_build_add_variables). Skips full pipeline. The caller (CLI) determines this from the build logger state.

Returns:

list of str or None: List of output parquet file paths, or None if no new data processed.

Raises:

H3ValidationError: If H3 resolution or partition parameters are invalid
GediFileError: If local source directory doesn’t exist or S3 returns no files

gedih3.gh3builder.merge_build_logs(log_file_1: str, log_file_2: str, output_log_file: str) → dict[source]#

Merge two build log files from separate databases.

Combines granules, columns, partition IDs, and date ranges while validating configuration consistency (gedi_version, h3_resolution_level, h3_partition_level).

Parameters:

log_file_1str: Path to first build log (base log)
log_file_2str: Path to second build log (log to merge in)
output_log_filestr: Path to write merged build log

Returns:

dict: Merged log dictionary

Raises:

FileNotFoundError: If either log file does not exist
ValueError: If log configurations are incompatible (different gedi_version, h3_resolution_level, or h3_partition_level)

Examples

>>> merged_log = merge_build_logs(
...     f'/path/to/database_world/{BUILD_LOG_FILENAME}',
...     f'/path/to/database_world_a10/{BUILD_LOG_FILENAME}',
...     f'/path/to/database_world_merged/{BUILD_LOG_FILENAME}'
... )