gedih3.gedidriver#

GEDI data file operations and HDF5 I/O.

This module provides classes and functions for working with GEDI HDF5 files, including parsing filenames, extracting data, and building Dask DataFrames.

Attributes#

logger

Classes#

`GEDIFile`	Parser for GEDI HDF5 filenames.
`GEDIShot`	Decoder for GEDI shot numbers.

Functions#

`soc_prod_from_file`(→ str)	Extract GEDI product code from a file path.
`soc_from_file`(→ str)	Get the SOC root directory from a GEDI file path.
`write_soc_manifest`(→ int)	(Re)generate the SOC manifest sentinel by walking soc_dir once.
`gedi_release_tag`(→ Optional[str])	The release fields of a GEDI file name, as a compact tag.
`gedi_release_rank`(→ tuple)	Rank one file of a granule product against the granule's other files.
`soc_file_tree`(→ Union[Dict[str, Dict[str, str]], ...)	Build a structured tree of GEDI SOC files grouped by orbit/track.
`gedi_subset`(source_file, dest_file, variables[, ...])
`gedi_file_glob`([orbit, orbit_granule, track, product, ...])
`preset_extra_names`(spec)	Variable names given alongside a preset keyword in a product spec.
`gedi_vars_expand`(product_vars[, version])	Expand per-product variable specifications to variable lists, in place.
`expand_var_wildcards`(var_specs, available_vars)	Expand fnmatch-style wildcard patterns in variable specifications.
`gedi_vars_from_h5`(gedi_file)
`gedi_vars_static`(product[, version])	Return the canonical variable list for a GEDI `(product, version)`
`check_soc_file_vars`(soc_file, available_products)
`validate_soc_files`(product_vars[, soc_dir, version, ...])	Validate `product_vars` against the shipped static manifests.
`wfm_pad`(wave[, n_bins, cval])
`process_waveforms`(starts, ends, noises, wfs[, n_bins])
`wfm_extract`(h5_file, beam, idx[, tx])
`load_h5`(→ pandas.DataFrame)	Load data from a GEDI HDF5 file into a pandas DataFrame.
`load_h5_merged`(→ Optional[pandas.DataFrame])	Load and merge data from multiple GEDI product files.
`dask_h5_merged`(→ dask.dataframe.DataFrame)	Create a Dask DataFrame from multiple GEDI product file sets.
`add_special_columns`(df[, lon_col, lat_col, dat_col])

Module Contents#

gedih3.gedidriver.logger#

gedih3.gedidriver.soc_prod_from_file(file_path: str | pathlib.Path) → str[source]#

Extract GEDI product code from a file path.

Parameters:

file_pathstr or Path: Path to a GEDI HDF5 file

Returns:

str: Product code (e.g., ‘L2A’, ‘L4A’)

gedih3.gedidriver.soc_from_file(file_path: str | pathlib.Path) → str[source]#

Get the SOC root directory from a GEDI file path.

Parameters:

file_pathstr or Path: Path to a GEDI HDF5 file

Returns:

str: Path to the SOC root directory

gedih3.gedidriver.write_soc_manifest(soc_dir: str, files=None) → int[source]#

(Re)generate the SOC manifest sentinel by walking soc_dir once.

Producer-driven refresh (R2): every code path that mutates a SOC tree (gh3_download, gh3_build –download, s3_etl_subset, gh3_build -i exit, gh3_doctor –fix soc_health) calls this on exit. Cheaper than letting every consumer re-walk the tree.

Walks the SOC tree in parallel via gedih3.parallel.walk_soc_parallel() (year/doy fan-out on the registered dask Client). Returns the number of files written to the manifest (0 when soc_dir is missing / empty).

Parameters:

soc_dir: SOC root directory.
fileslist[str], optional: Pre-computed absolute file list to use instead of walking. When the caller has already enumerated the tree (e.g. cli/gh3_build.py after its existing-h5 listing), passing the list here avoids the redundant parallel walk.

gedih3.gedidriver.gedi_release_tag(file_name: str) → str | None[source]#

The release fields of a GEDI file name, as a compact tag.

With the granule key, the product and the GEDI version, the tag identifies the file: it is what the build log records per granule and product (releases), at a fraction of a basename’s size.

Parameters:

file_namestr: A GEDI file name, path or URL. Only the basename is read.

Returns:

str or None: 'PP_PGE_GG' (PPDS, PGE, generation), plus '_<suffix>' when the name has one; None for a name outside the GEDI convention. gedi_release_rank() ranks tags as it ranks names.

Examples

>>> gedi_release_tag('GEDI02_A_2019108002012_O01956_03_T03909_02_003_01_V002_SGS.h5')
'02_003_01_SGS'

gedih3.gedidriver.gedi_release_rank(file_name: str) → tuple[source]#

Rank one file of a granule product against the granule’s other files.

NASA re-releases granules within one GEDI version with a new PPDS (geolocation), PGE (software) or generation. The highest rank is the file to use: PPDS first, then PGE, then generation, then the suffix, where a suffixed file (e.g. _SGS) beats the plain file of the same release and suffixes compare alphabetically. For names that follow the GEDI convention, this is the order a sort by file name gives within one directory.

Parameters:

file_namestr: A GEDI file name, path or URL (only the basename is read), or a release tag from gedi_release_tag().

Returns:

tuple: (ppds, pge, generation, has_suffix, suffix), ordered so that a larger tuple is a later release. A name that does not follow the GEDI convention ranks below every one that does.

Examples

>>> old = 'GEDI02_A_2019108002012_O01956_03_T03909_02_003_01_V002.h5'
>>> new = 'GEDI02_A_2019108002012_O01956_03_T03909_02_004_01_V002.h5'
>>> gedi_release_rank(new) > gedi_release_rank(old)
True

gedih3.gedidriver.soc_file_tree(file_struct: str | List[str] | List[earthaccess.store.EarthAccessFile], to_list: bool = False, glob_kwargs: Dict | None = None, exclude: List[str] | None = None, require_all: bool = True) → Dict[str, Dict[str, str]] | List[Dict[str, str]][source]#

Build a structured tree of GEDI SOC files grouped by orbit/track.

Organizes GEDI HDF5 files into a nested structure for easy access to related files across different products (L2A, L2B, L4A, etc.).

Parameters:

file_structstr, list of str, or list of EarthAccessFile: Either a directory path containing GEDI files, or a list of file paths or EarthAccessFile objects.
to_listbool, default False: If True, return a list of dicts instead of a nested dict.
glob_kwargsdict, optional: Keyword arguments passed to gedi_file_glob() for filtering files (e.g., version, orbit). Pass version whenever the tree may hold more than one GEDI release: granules are keyed by orbit/track only, so two releases of the same granule would collide under one key.
excludelist of str, optional: fnmatch-style patterns matched against each file’s basename. Any file whose basename matches any pattern is dropped from the result. Useful to exclude internal/SGS variants (e.g. ['*_SGS.h5']) that share the SOC tree with public release files.
require_allbool, default True: Keep only granules that have a file for every product found in the listing. False keeps every granule with whatever products it has (its dict simply lacks the others) — for callers that judge completeness against the products they request, e.g. a build admitting granules whose later products are not published yet.

Returns:

dict or list: If to_list=False: dict keyed by orbit_track identifier, with values being dicts mapping product codes to file paths. A product with more than one file maps to its latest release (gedi_release_rank), and the duplicates are logged once per process. If to_list=True: list of product dicts.

Raises:

GediValidationError: When the listed files span more than one GEDI release (_V00N). One release per tree is the contract; a mixed listing would silently pair files of different releases under one granule key.

Examples

>>> tree = soc_file_tree('/path/to/soc')
>>> tree['O12345_01_T00001']
{'L2A': '/path/to/GEDI02_A_...h5', 'L4A': '/path/to/GEDI04_A_...h5'}
>>> # Exclude NASA SGS internal files from discovery
>>> tree = soc_file_tree('/path/to/soc', exclude=['*_SGS.h5'])

gedih3.gedidriver.gedi_subset(source_file, dest_file, variables, subset_beams=None)[source]#

gedih3.gedidriver.gedi_file_glob(orbit=None, orbit_granule=None, track=None, product: int = None, level: str = None, ppds: int = None, pge: int = None, generation: int = None, version: int = None)[source]#

gedih3.gedidriver.preset_extra_names(spec)[source]#

Variable names given alongside a preset keyword in a product spec.

['default', 'energy_total'] → ['energy_total']. Specs without a preset keyword, and the keywords themselves (all/* included), give []. The names are the user’s, not the preset’s: callers validate them like an explicit list.

gedih3.gedidriver.gedi_vars_expand(product_vars, version=None)[source]#

Expand per-product variable specifications to variable lists, in place.

Parameters:

product_varsdict: {product: spec}. A spec is None or [] (every variable), ['*'] / ['all'] (every variable), a single path to a file listing variables, a list naming a preset (minimal/min or default/def), or a plain list of variable names or wildcards. Names given alongside a preset are kept and added after the preset’s variables (preset_extra_names()): ['default', 'energy_total'] is the default list plus energy_total. minimal takes precedence over default when both are given, and all/* next to a preset add nothing.
versionint, optional: GEDI release whose preset lists to use.

Returns:

dict: product_vars, with each spec replaced by its variable list (None for every variable).

gedih3.gedidriver.expand_var_wildcards(var_specs, available_vars)[source]#

Expand fnmatch-style wildcard patterns in variable specifications.

Supports *, ?, [seq], [!seq]. Non-pattern specs pass through unchanged. Raises GediValidationError if a pattern matches nothing.

gedih3.gedidriver.gedi_vars_from_h5(gedi_file)[source]#

gedih3.gedidriver.gedi_vars_static(product, version=None)[source]#

Return the canonical variable list for a GEDI (product, version) from the static manifest shipped in gedih3.data — no I/O on any target file.

NASA release files of the same (product, version) always carry an identical dataset schema, so this is equivalent to gedi_vars_from_h5() for canonical release files but free.

Returns None when no static manifest exists for the requested combination; callers should fall back to gedi_vars_from_h5().

Note: the result reflects a NASA release file’s schema. Files that were previously subset (e.g. via gedi_subset() or S3 ETL mode) contain a subset of these variables — for those, use gedi_vars_from_h5() to introspect what is actually present.

gedih3.gedidriver.check_soc_file_vars(soc_file, available_products)[source]#

gedih3.gedidriver.validate_soc_files(product_vars: Dict, soc_dir: str = GH3_DEFAULT_SOC_DIR, version: int | None = None, exclude: List[str] | None = None)[source]#

Validate product_vars against the shipped static manifests.

Only authoritative for canonical NASA release files validated against the per-product manifest in src/gedih3/data/GEDI*_DATASETS_*.txt — that is, for the fresh-build default request (Regime A) or for a resume where the user explicitly re-requests default for a product (Regime C). On all other resume paths the build log is the contract, and gh3_build.py bypasses this check entirely. Callers are responsible for filtering product_vars to only the products they actually want to validate against the static manifest — passing the full union of existing + new vars on a resume will produce false negatives whenever the shipped manifest has drifted relative to the list saved in the build log.

class gedih3.gedidriver.GEDIFile(file_path: str | pathlib.Path | earthaccess.store.EarthAccessFile)[source]#

Parser for GEDI HDF5 filenames.

Extracts metadata from GEDI filename convention including orbit, granule, track, version, and date information.

Attributes:

file_pathstr: Full path to the GEDI file
full_namestr: Basename of the file
productstr: Product identifier (e.g., ‘GEDI02’)
levelstr: Processing level (e.g., ‘A’, ‘B’)
datedatetime: Acquisition date and time
product_codestr: Combined product code (e.g., ‘L2A’, ‘L4A’)
orbitint: Orbit number
orbit_granuleint: Granule number within orbit
trackint: Ground track number
versionint: Data version number
yearstr: Year string (e.g., ‘2019’)
doystr: Day-of-year string (e.g., ‘338’)
mission_weekint: Mission week number since GEDI launch (2018-12-13)
suffixstr or None: Extra filename parts after version field, if any

Examples

>>> gf = GEDIFile('/path/to/GEDI02_A_2020123120000_O12345_01_T00001_02_003_01_V002.h5')
>>> gf.orbit
12345
>>> gf.version
2

__str__() → str[source]#

parse_file(f: str) → None[source]#

get_glob_pattern(product: int = 1, level: str = 'B', ppds: int = 2)[source]#

classmethod from_filename(file_path)[source]#

search_file(soc_dir: str = None, product: int = 1, level: str = 'B', other_files=None)[source]#

class gedih3.gedidriver.GEDIShot(shot: int | numpy.ndarray, build_glob: bool = False)[source]#

Bases: GEDIFile

Decoder for GEDI shot numbers.

Extracts the orbit, beam, granule and shot index encoded in GEDI shot numbers. A shot number has 18 digits, OOOOOBBRRGNNNNNNNN: orbit (5), beam (2), reserved (2), granule within the orbit (1), shot index (8). The ground track is not encoded.

Attributes:

shotint or array: The shot number(s)
orbitint or array: Orbit number
orbit_granuleint or array: Granule number within orbit
trackNone: Not encoded in the shot number, so file globs match any track (orbit and granule identify the file).
beamint or array: Beam number (0-7)
beam_strstr or list: Beam identifier string (e.g., ‘BEAM0101’)
powerbool or array: True if power beam (beam > 3)
shot_indexint or array: Shot index within the granule

Examples

>>> shot = GEDIShot(19560500300012345)
>>> shot.orbit, shot.orbit_granule, shot.beam_str, shot.shot_index
(1956, 3, 'BEAM0101', 12345)

parse_shot(shot: int | numpy.ndarray) → None[source]#

get_glob_pattern(product: int = 1, level: str = 'B', ppds: int = 2, pge: int = None, generation: int = None, version: int = None)[source]#

Glob pattern(s) of the GEDI file(s) holding these shots (any track).

Returns:

str or list of str: One pattern for a scalar shot; for an array, one per distinct granule, sorted. Also stored as file_glob_pattern.

search_file(soc_dir: str)[source]#

Files under soc_dir matching file_glob_pattern (built if absent).

Returns:

list of str: Sorted paths; empty if none match.

gedih3.gedidriver.wfm_pad(wave, n_bins=1420, cval=0)[source]#

gedih3.gedidriver.process_waveforms(starts, ends, noises, wfs, n_bins=1420)[source]#

gedih3.gedidriver.wfm_extract(h5_file, beam, idx, tx=False)[source]#

gedih3.gedidriver.load_h5(fpath: str | pathlib.Path | earthaccess.store.EarthAccessFile, columns: List[str], which_beams: List[str] | None = None, shots: numpy.ndarray | None = None, include_source: bool = True, dropna: bool = True) → pandas.DataFrame[source]#

Load data from a GEDI HDF5 file into a pandas DataFrame.

Parameters:

fpathstr, Path, or EarthAccessFile: Path to the GEDI HDF5 file
columnslist of str: Variable names to extract from the file
which_beamslist of str, optional: Specific beams to load (e.g., [‘BEAM0101’, ‘BEAM1000’]). If None, loads all beams.
shotsarray, optional: Specific shot numbers to extract. If None, loads all shots.
include_sourcebool, default True: Add root_file and root_beam columns to output
dropnabool, default True: Drop rows with NaN values

Returns:

pd.DataFrame: DataFrame indexed by shot_number containing requested variables

gedih3.gedidriver.load_h5_merged(prod_files: Dict[str, str], product_vars: Dict[str, List[str]], which_beams: List[str] | None = None, shots: numpy.ndarray | None = None, dropna: bool = True, suffix_all: bool = False) → pandas.DataFrame | None[source]#

Load and merge data from multiple GEDI product files.

Combines data from different GEDI products (L2A, L2B, L4A, etc.) into a single DataFrame, joining on shot_number.

Parameters:

prod_filesdict: Mapping of product codes to file paths (e.g., {‘L2A’: ‘/path/to/l2a.h5’})
product_varsdict: Mapping of product codes to lists of variable names to extract
which_beamslist of str, optional: Specific beams to load
shotsarray, optional: Specific shot numbers to extract
dropnabool, default True: Drop rows with NaN values
suffix_allbool, default False: If True, suffix all columns with product code (e.g., ‘rh_l2a’). Required when variables have the same name across products.

Returns:

pd.DataFrame or None: Merged DataFrame indexed by shot_number, or None if no data

gedih3.gedidriver.dask_h5_merged(prod_files_list: List[Dict[str, str]], product_vars: Dict[str, List[str]], which_beams: List[str] | None = None, shots: numpy.ndarray | None = None, dropna: bool = True, suffix_all: bool = False, by_beam: bool = False) → dask.dataframe.DataFrame[source]#

Create a Dask DataFrame from multiple GEDI product file sets.

Lazily loads and merges GEDI data for distributed processing.

Parameters:

prod_files_listlist of dict: List of product file dictionaries from soc_file_tree()
product_varsdict: Mapping of product codes to variable lists
which_beamslist of str, optional: Specific beams to load
shotsarray, optional: Specific shot numbers to extract
dropnabool, default True: Drop rows with NaN values
suffix_allbool, default False: Suffix all columns with product code
by_beambool, default False: If True, create separate partitions for each beam (more parallelism)

Returns:

dask.dataframe.DataFrame: Lazy Dask DataFrame for distributed processing

gedih3.gedidriver.add_special_columns(df, lon_col: str = None, lat_col: str = None, dat_col: str = None)[source]#