gedih3.gh3builder

Attributes

logger

Functions

download_soc(product_vars[, spatial, temporal, ...])	Download GEDI HDF5 files in SOC (Science Operation Center) format.
s3_etl_subset(product_vars[, spatial, temporal, ...])	ETL-style S3 build: search for GEDI granules, then dispatch Dask workers
h3_part_files(df, dir_path[, res, part, lat_col, ...])	Write a DataFrame to H3-partitioned parquet files.
h3_write_metadata(h3_file[, stats])	Write a sidecar metadata JSON file for an H3 partition parquet file.
h3_read_metadata(h3_file)	Read the sidecar metadata JSON for an H3 partition parquet file.
h3_merge_metadata(h3_subdir)	Merge per-year metadata files into a single summary for an H3 partition.
h3_skip_part(h3_dir, h3_part, gedi_file[, cols])	Check whether an H3 partition already contains data from a GEDI granule.
h3_add_skip_column(df, h3_dir)	Add a _skip boolean column indicating partitions to skip.
dh3_merge_metadata(h3_subdir)
h3_merge_files(in_dir, out_dir[, rm_src, replace, ...])	Merge multiple parquet files for an H3 partition into a single file.
dh3_merge_files(in_dir, out_dir[, rm_src, replace])
preclean_merge_failures(→ Dict[str, int])	L1 resume pre-clean: act on every recorded _merge_failures sentinel.
apply_merge_failures_to_logger(→ int)	Fold the merge-failed-granules sidecar into the build-log's granule
releases_from_rows(h3_dir, products, keys[, version])	The source release of granule products, read from the database's rows (#67).
replacement_abandon_hint(→ str)	How to abandon an unfinished replacement, and what that leaves.
blind_replacement_targets(→ Dict[Tuple[int, int, int], ...)	Granules a replacement cannot tell old rows from new ones for (#68).
blind_replacement_message(→ str)	Why the granules of blind_replacement_targets() are not replaced.
check_replacement_sources(→ None)	Refuse to replace a granule whose changed rows cannot be told from its old ones.
replacement_journal(→ str)	Path of the journal of a granule replacement under a build tmp dir.
read_replacement_journal(→ Optional[Dict[Tuple[int, ...)	The target set of an unfinished replacement, or None when there is none.
build_h3db(, h3_dir, skip_granules, status_callback, ...)	Build an H3-indexed GEDI database from local SOC files or S3 download.
merge_build_logs(→ dict)	Merge two build log files from separate databases.

Module Contents

gedih3.gh3builder.logger

gedih3.gh3builder.download_soc(product_vars: Dict, spatial=None, temporal=None, direct_access=False, update=False, version=None, odir=GH3_DEFAULT_SOC_DIR, n_jobs=5, on_granule_complete=None, ensure_l2a=True)

Download GEDI HDF5 files in SOC (Science Operation Center) format.

Expands variable specifications, ensures L2A essentials and shot_number are included, then delegates to gedi_download for retrieval.

Parameters:

product_varsdict

Mapping of GEDI product codes (e.g., ‘L2A’, ‘L4A’) to variable lists. Accepts ‘default’, ‘minimal’, or explicit variable names.

spatialGeoDataFrame, list, or str, optional

Spatial filter (vector file, bbox as [W,S,E,N], or ISO3 code).

temporaltuple of str, optional

Temporal range as (start_date, end_date) in ‘YYYY-MM-DD’ format.

direct_accessbool, default False

If True, use S3 streaming instead of downloading to disk.

updatebool, default False

If True, resume a previous download (skip already-downloaded files).

versionint or str, optional

GEDI data version (e.g., 3). If None, uses the package default (GEDI_DEFAULT_VERSION).

odirstr

Output directory for downloaded HDF5 files.

n_jobsint, default 5

Number of parallel download workers.

on_granule_completecallable, optional

Callback (granule_info_dict, status_str) -> None for per-granule progress tracking. Passed through to gedi_download(); see its docstring for the granule_info_dict contract (includes a path key).

ensure_l2abool, default True

If True, automatically add L2A essentials when L2A is not in product_vars. Set to False for variable-only updates where L2A data already exists in the target database.

Returns:

list

List of downloaded SOC file paths or EarthAccessFile objects.

gedih3.gh3builder.s3_etl_subset(product_vars, spatial=None, temporal=None, version=None, odir=None, ensure_l2a=True)

ETL-style S3 build: search for GEDI granules, then dispatch Dask workers to open remote HDF5 files, extract only selected variables via range requests, and write compact local HDF5 files.

This transfers and stores significantly less data than full downloads (10-50x smaller depending on variable selection). Workers run as true separate processes via Dask (GIL-free), matching the parallelism of download mode.

Parameters:

product_varsdict

Mapping of GEDI product codes (e.g., ‘L2A’, ‘L4A’) to variable lists. Accepts ‘default’, ‘minimal’, or explicit variable names.

spatialGeoDataFrame, list, or str, optional

Spatial filter (vector file, bbox as [W,S,E,N], or ISO3 code).

temporaltuple of str, optional

Temporal range as (start_date, end_date) in ‘YYYY-MM-DD’ format.

versionint or str, optional

GEDI data version. If None, uses the package default (GEDI_DEFAULT_VERSION).

odirstr

Output directory for compact local HDF5 files in SOC structure.

ensure_l2abool, default True

If True, automatically add L2A essentials when L2A is not in product_vars. Set to False for variable-only updates where L2A data already exists in the target database.

Returns:

str

Path to the output directory containing compact HDF5 files.

Raises:

GediFileError

If no GEDI files are found on S3 for the given parameters.

gedih3.gh3builder.h3_part_files(df, dir_path, res=12, part=3, lat_col='lat_lowestmode', lon_col='lon_lowestmode', roi_tiles=[])

Write a DataFrame to H3-partitioned parquet files.

Indexes the DataFrame by H3 cell, groups rows by partition cell, and writes each group as a separate parquet file. If a file already exists for a partition, new rows are appended.

Parameters:

dfpandas.DataFrame

Input DataFrame with coordinate columns and root_beam, root_file metadata columns.

dir_pathstr

Base directory for output partition subdirectories.

resint, default 12

H3 resolution for shot-level indexing.

partint, default 3

H3 resolution for file partitioning.

lat_colstr, default ‘lat_lowestmode’

Column name for latitude values.

lon_colstr, default ‘lon_lowestmode’

Column name for longitude values.

roi_tileslist of str, optional

If non-empty, only write partitions whose H3 cell ID is in this list.

Returns:

list of str or None

List of written parquet file paths, or None if the input is empty.

gedih3.gh3builder.h3_write_metadata(h3_file, stats=None)

Write a sidecar metadata JSON file for an H3 partition parquet file.

Parameters:

h3_filestr

Path to the H3 partition parquet file.

statsdict, optional

Pre-computed stats from parquet_merge_files’s streaming pass: {'shot_count', 'shot_min', 'shot_max', 'dt_min', 'dt_max', 'root_files'}. When provided, skips the ~1.5-2 GB pd.read_parquet re-read of the just-written merged file. When None or any field is None, falls back to reading the columns from disk (slower, more memory).

Returns:

str

Path to the written metadata JSON file (*PARTITION_META_FILENAME).

gedih3.gh3builder.h3_read_metadata(h3_file)

Read the sidecar metadata JSON for an H3 partition parquet file.

Parameters:

h3_filestr

Path to the H3 partition parquet file. The metadata file is expected at the same path with a PARTITION_META_FILENAME extension.

Returns:

dict or None

Parsed metadata dictionary, or None if the metadata file does not exist.

gedih3.gh3builder.h3_merge_metadata(h3_subdir)

Merge per-year metadata files into a single summary for an H3 partition.

Aggregates shot counts, expands shot and date ranges, and deduplicates granule identifiers across all year subdirectories within one H3 cell.

Parameters:

h3_subdirstr

Path to an H3 partition directory (e.g., h3_03=<cell_id>/) containing year subdirectories with parquet and metadata files.

Returns:

str or None

Path to the merged metadata JSON file, or None if no metadata files were found. With no year parquet left, a summary from an earlier merge is removed, and so is the partition directory when nothing else is left in it.

gedih3.gh3builder.h3_skip_part(h3_dir, h3_part, gedi_file, cols=None)

Check whether an H3 partition already contains data from a GEDI granule.

Reads the partition’s merged metadata to determine if the granule has already been indexed and, optionally, if the requested columns are already present.

Parameters:

h3_dirstr

Root directory of the H3 database.

h3_partstr

H3 cell ID of the partition to check.

gedi_filestr

Path or filename of the GEDI HDF5 file to test against.

colslist of str, optional

If provided, also verify that these columns already exist in the partition. The partition is only skipped when both the granule is present and all requested columns are available.

Returns:

bool

True if the partition should be skipped (granule already indexed and columns present), False otherwise.

gedih3.gh3builder.h3_add_skip_column(df, h3_dir)

Add a _skip boolean column indicating partitions to skip.

Designed for use with dask.dataframe.map_partitions. Checks each unique H3 partition cell individually, since a single beam can span multiple H3 cells. Delegates to h3_skip_part per cell.

Parameters:

dfpandas.DataFrame

A single Dask partition containing H3 partition column(s) (h3_XX) and root_file_l2a.

h3_dirstr

Root directory of the H3 database.

Returns:

pandas.DataFrame

Input DataFrame with an added _skip column (True if the cell’s granule data already exists in the database).

gedih3.gh3builder.dh3_merge_metadata(h3_subdir)

gedih3.gh3builder.h3_merge_files(in_dir, out_dir, rm_src=True, replace=False, replace_granules=None)

Merge multiple parquet files for an H3 partition into a single file.

Reads all parquet files in in_dir, merges them (deduplicating shots when appending to an existing file), writes the result to out_dir, and generates a sidecar metadata JSON file.

Parameters:

in_dirstr

Input directory containing parquet fragment files for one H3 cell/year combination.

out_dirstr

Output directory where the merged file will be written, preserving the <h3_cell>/<year>/ structure.

rm_srcbool, default True

If True, remove the source directory after a successful merge.

replacebool, default False

If True, overwrite existing output files. If False, merge new data into any existing output file.

replace_granulesdict, optional

{(orbit, granule, track): {product: basename}}: re-index these granules (#47). The existing file’s rows from them are dropped unless they already come from the given files, and the fragments’ rows take their place (parquet_merge_files). The existing file is rewritten even when in_dir holds no fragment (the granules’ shots left this cell-year), the “already merged” skip never applies, and a file left with no row is removed with its metadata (and its year directory, when empty). Re-running gives the same result. An unreadable existing file is handled as on any merge: the fragments alone are written (without fragments, it is left as it is). Not with replace=True (GediValidationError).

Returns:

str or None

Path to the merged parquet file, or None if in_dir contained no parquet files and there was no orphaned merge result to promote (see _promote_orphan_merge_result), or the file was left empty by replace_granules and removed.

gedih3.gh3builder.dh3_merge_files(in_dir, out_dir, rm_src=True, replace=False)

gedih3.gh3builder.preclean_merge_failures(tmp_dir: str) → Dict[str, int]

L1 resume pre-clean: act on every recorded _merge_failures sentinel.

For each partition the prior run’s merge marked failed:

1. Stat each *.parquet in the partition — unlink if 0-byte (the canonical class B artifact). Truncated-but-nonzero parquets (class C) are unlinked only when pq.ParquetFile cannot open them — checked cheaply by opening the footer once. The cost is bounded by the failure list, never the full healthy tree. Each unlinked fragment’s _complete/ sentinel goes too, so the next Stage 1 re-runs that (granule × beam) instead of skipping it as done.

2. Unlink any co-located *.tmp / *.merge.tmp siblings. These survive SIGKILL when AtomicFileWriter.__exit__ never runs and would otherwise pollute future merges.

3. Delete the failure sentinel itself so re-running the pre-clean is idempotent (next run only re-acts on freshly-failed merges).

Step 1 probes each fragment in O(1) (_parquet_tail_ok: the magic bytes at both ends and a plausible footer length), which catches 0-byte, truncated and never-finished files. The full footer parse is reserved for partitions whose recorded error is a footer-level artifact (thrift) or is missing: parsing a 1.4k-column footer costs seconds per fragment on GPFS, and a serial parse of every fragment behind a 14k-sentinel backlog (e.g. a batch of schema failures, whose fragments are healthy) was ~58 h on the driver. Partitions fan out over the registered dask Client.

Returns {'partitions_cleaned': N, 'parquets_removed': N, 'tmps_removed': N}. Companion to apply_merge_failures_to_logger — calling this without the granule flip-back would unlink fragments and leave their granules marked INDEXED, permanently dropping rows. build_h3db runs it before Stage 1 (so the lost tasks are re-extracted in the same run) and _merge_and_finalize again at merge entry.

gedih3.gh3builder.apply_merge_failures_to_logger(h3_logger, tmp_dir: str) → int

Fold the merge-failed-granules sidecar into the build-log’s granule status: every recorded granule becomes MERGE_FAILED. Returns the number of entries changed.

Any prior status flips, not only INDEXED: a granule extracted by the failing run is still PENDING here, and set_post_build_info would otherwise mark it INDEXED from its rows in other partitions. A recorded granule the log does not track yet (S3 mode registers none) is added. set_post_build_info, the reconcile and log_state all keep MERGE_FAILED; only _release_merge_failed(), after a Stage 1 run that re-extracted the granule, clears it.

Idempotent. Removes the sidecar after the fold. The CLI calls it at startup (a run killed before its fold leaves the sidecar behind) and right after every merge. Does NOT call h3_logger.save_log — the caller controls save cadence.

gedih3.gh3builder.releases_from_rows(h3_dir: str, products, keys, version: int | None = None)

The source release of granule products, read from the database’s rows (#67).

A database built before gedih3 0.20.10 records no release per granule, so re-released granules cannot be found from the build log. Every row names its source file per product (root_file_<prod>), and the file name holds the granule key and the release (PPDS, PGE, generation, suffix): one worker task per (cell, year) file reads its metadata and, if it lists one of keys, those columns only (_year_file_source_releases()). The files come from the database’s _manifest.txt (smart_glob()).

Parameters:

h3_dirstr

Database root.

productsiterable of str

Products to read, e.g. ['L2A', 'L4A']; one without a source column is skipped.

keysiterable of tuple

(orbit, granule, track) of the granules to read.

versionint, optional

The database’s GEDI version: names of another one are ignored.

Returns:

tuple

(releases, mixed, unread, failed): {key: {product: tag}} for products whose rows name one release; {key: {product: [tags]}} for those naming several (a replacement left half done: never guessed); the keys a file that could not be read may hold, left out of both (their rows were not all read: all of keys when that cannot be told from the file’s metadata); and the files that could not be read (skipped and logged; None for a batch of them). Requires a registered dask Client.

gedih3.gh3builder.replacement_abandon_hint(tmp_dir: str) → str

How to abandon an unfinished replacement, and what that leaves.

gedih3.gh3builder.blind_replacement_targets(h3_dir: str, targets, unsourced_by_key: Dict[Tuple[int, int, int], set] | None = None) → Dict[Tuple[int, int, int], List[str]]

Granules a replacement cannot tell old rows from new ones for (#68).

A replacement merge keeps a row when every root_file_<prod> column names the given file. A product added by a variable update of an older gedih3 has no such column, so when it is the only product of a granule with a new release, the merge would keep the old rows while the build log records the new release. A granule with another product re-released is fine: its rows are replaced whole. One with no recorded release (indexed before gedih3 recorded them) cannot be shown to be fine, so it counts. Answered from the build log (h3_columns, the union of every file’s columns, and the recorded releases), and from unsourced_by_key for files that differ.

Parameters:

h3_dirstr

Database root.

targetsdict

{(orbit, granule, track): {product: file}}, as passed to _replace_granules_build().

unsourced_by_keydict, optional

{(orbit, granule, track): products} lacking a source column in a file holding that granule (a database expanded after a product was added without one has it in some files only), from the files’ footers (_year_file_replace_targets()).

Returns:

dict

{(orbit, granule, track): [product, ...]}, the unsourced products of each such granule.

gedih3.gh3builder.blind_replacement_message(blind, tmp_dir: str | None = None) → str

Why the granules of blind_replacement_targets() are not replaced.

gedih3.gh3builder.check_replacement_sources(h3_dir: str, targets, tmp_dir: str | None = None, unsourced_by_key: Dict[Tuple[int, int, int], set] | None = None) → None

Refuse to replace a granule whose changed rows cannot be told from its old ones.

See blind_replacement_targets(); the parameters are the same, and tmp_dir (the build tmp dir) points the message at an unfinished replacement’s journal.

Raises:

GediValidationError

If such a granule is among targets. Nothing was changed.

gedih3.gh3builder.replacement_journal(tmp_dir: str) → str

Path of the journal of a granule replacement under a build tmp dir.

Its presence means a replacement started there and has not finished: gh3_build resumes it before anything else.

gedih3.gh3builder.read_replacement_journal(tmp_dir: str) → Dict[Tuple[int, int, int], Dict[str, str]] | None

The target set of an unfinished replacement, or None when there is none.

{(orbit, granule, track): {product: basename}}, as passed to _replace_granules_build().

gedih3.gh3builder.build_h3db(product_vars: Dict[str, List[str]], res: int = 12, part: int = 3, spatial=None, temporal=None, soc_source: str | List | None = None, version: int | None = None, tmp_dir: str = os.path.join(GH3_DEFAULT_TMP_DIR, 'gh3_build'), h3_dir: str = GH3_DEFAULT_H3_DIR, skip_granules: List[Dict] | None = None, status_callback: Callable[[str], None] | None = None, variable_only_update: bool = False, exclude: List[str] | None = None, allow_missing_products: bool = False, granule_products_callback: Callable[[Dict[Tuple[int, int, int], List[str]]], None] | None = None, stage1_outcome_callback: Callable[[set, set], None] | None = None, only_granules: List[Dict] | None = None, releases_callback: Callable[[Dict[Tuple[int, int, int], Dict[str, str]]], None] | None = None) → List[str] | None

Build an H3-indexed GEDI database from local SOC files or S3 download.

Parameters:

product_varsdict

Dictionary mapping GEDI product codes (e.g., ‘L2A’, ‘L4A’) to lists of variable names to extract. Use ‘default’, ‘minimal’, or explicit lists.

resint, default 12

H3 resolution level for indexing individual shots (0-15).

partint, default 3

H3 resolution level for file partitioning (0-15, must be <= res).

spatialGeoDataFrame, list, or str, optional

Spatial filter for the region of interest.

temporaltuple, optional

Temporal filter as (start_date, end_date) in ‘YYYY-MM-DD’ format.

soc_sourcestr, list, or None

• None: download from NASA S3 to a temp directory (cleaned up after build)

• str: path to local directory containing GEDI SOC HDF5 files

• list: pre-acquired list of file paths or EarthAccessFile objects

versionint or None

GEDI data release for every product. None resolves via _resolve_build_version() (build log > SOC tree > package default). Every SOC listing is filtered to this release, so the Stage 1 schema is derived from files of exactly that release.

tmp_dirstr

Path to temporary directory for intermediate files.

h3_dirstr

Output path for the H3-indexed parquet database.

skip_granuleslist of dict, optional

List of granule identifiers to skip (from previous builds).

status_callbackcallable, optional

Called with status string at pipeline stage transitions.

variable_only_updatebool, default False

If True, only add new variable columns to existing partition files via shot_number join (_build_add_variables). Skips full pipeline. The caller (CLI) determines this from the build logger state.

excludelist of str, optional

fnmatch patterns of source file basenames to ignore (e.g. ['*_SGS.h5']), in every listing. The patterns kept in an existing database’s build log (gh3_build --exclude) are always added.

allow_missing_productsbool, default False

Admit granules that have L2A but lack later products (L2B, L4A… published after L2A). Their rows store those products’ columns as null, in the database’s types, until _build_fill_products() backfills them. Requires the streaming writer. Default: a granule must have every requested product.

granule_products_callbackcallable, optional

Called once, before any fragment is written, with {(orbit, granule, track): [missing products]} for every granule this Stage 1 processes (an empty list when it has every product). The CLI records MISSING_SOURCE from it, so the backfill knows its targets without scanning the database, even after a crash mid-write.

stage1_outcome_callbackcallable, optional

Called as (incomplete_keys, empty_keys) once Stage 1 drained every task (see _write_partitioned_streaming()), before the merge. Never called by the legacy writer (GH3_WRITE_STREAMING off, deprecated), by an interrupted drain, or when no granule is left to process.

only_granuleslist of dict, optional

If given, Stage 1 reads only these granules and leaves every other one for a later build (H3BuildLogger.stage1_only_granules: a spatial-only expansion reads only the added region).

releases_callbackcallable, optional

variable_only_update only: called before the first write with the source releases ({(orbit, granule, track): {product: tag}}) read for each product new to the database, which gets source columns too (see _build_add_variables()), so the caller can record them (H3BuildLogger.record_releases()).

Returns:

list of str or None

List of output parquet file paths, or None if no new data processed.

Raises:

H3ValidationError

If H3 resolution or partition parameters are invalid

GediFileError

If local source directory doesn’t exist or S3 returns no files

gedih3.gh3builder.merge_build_logs(log_file_1: str, log_file_2: str, output_log_file: str) → dict

Merge two build log files from separate databases.

Combines granules, columns, partition IDs, and date ranges while validating configuration consistency (gedi_version, h3_resolution_level, h3_partition_level).

Parameters:

log_file_1str

Path to first build log (base log)

log_file_2str

Path to second build log (log to merge in)

output_log_filestr

Path to write merged build log

Returns:

dict

Merged log dictionary

Raises:

FileNotFoundError

If either log file does not exist

ValueError

If log configurations are incompatible (different gedi_version, h3_resolution_level, or h3_partition_level)

Examples

>>> merged_log = merge_build_logs(
...     f'/path/to/database_world/{BUILD_LOG_FILENAME}',
...     f'/path/to/database_world_a10/{BUILD_LOG_FILENAME}',
...     f'/path/to/database_world_merged/{BUILD_LOG_FILENAME}'
... )