gedih3.logger

Attributes

PRODUCT_STATUS_INDEXED
PRODUCT_STATUS_PARTIAL_NAN
PRODUCT_STATUS_MISSING_COLUMN
PRODUCT_STATUS_MISSING_SOURCE
PRODUCT_STATUS_FAILED
PRODUCT_STATUS_PENDING
GRANULE_STATUS_NO_DATA

Classes

SOCDownloadLogger
H3BuildLogger

Functions

load_log_data(file_path)
merge_temporal(existing, new)
merge_product_vars(existing_product_vars[, ...])
resolve_soc_version(soc_dir)	Return the GEDI data version a local SOC tree was populated with, or None.
stored_exclude(→ list)	The source exclusions a build log keeps (gh3_build --exclude).

Module Contents

gedih3.logger.PRODUCT_STATUS_INDEXED = 'INDEXED'

gedih3.logger.PRODUCT_STATUS_PARTIAL_NAN = 'PARTIAL_NAN'

gedih3.logger.PRODUCT_STATUS_MISSING_COLUMN = 'MISSING_COLUMN'

gedih3.logger.PRODUCT_STATUS_MISSING_SOURCE = 'MISSING_SOURCE'

gedih3.logger.PRODUCT_STATUS_FAILED = 'FAILED'

gedih3.logger.PRODUCT_STATUS_PENDING = 'PENDING'

gedih3.logger.GRANULE_STATUS_NO_DATA = 'NO_DATA'

gedih3.logger.load_log_data(file_path)

gedih3.logger.merge_temporal(existing, new)

gedih3.logger.merge_product_vars(existing_product_vars, new_product_vars=None)

gedih3.logger.resolve_soc_version(soc_dir)

Return the GEDI data version a local SOC tree was populated with, or None.

A-priori lookup — no HDF5 open, no recursive walk:

1. the download log’s gedi_version (gedih3_download_log.json at the SOC root), written by every gh3 producer;

2. else the _V00N field of the first GEDI*.h5 found by a bounded year/doy scandir (externally populated trees: manual rsync, NASA delivery).

Used by gh3_build / gh3_download to pin a fresh build or resumed download to the release already on disk when the user did not pass --gedi-version; the package default (GEDI_DEFAULT_VERSION) only applies when nothing on disk answers the question.

class gedih3.logger.SOCDownloadLogger(product_vars, spatial=None, temporal=None, version=None, dir=None)

gedi_version = None

s3_access = False

log_file

log_data

updating = False

new_spatial = None

new_temporal = None

new_product_vars = None

register_pending_granules(granule_list)

Register granules as PENDING. Called before download starts.

Parameters:

granule_listlist of dict

Each dict must have ‘orbit’, ‘granule’, ‘track’ keys.

update_granule_status(gran_key, status)

Update status of a single granule (does NOT auto-save).

Caller controls save frequency via explicit save_log() calls.

Parameters:

gran_keydict

Dict with ‘orbit’, ‘granule’, ‘track’ keys.

statusstr

‘DOWNLOADED’ or ‘FAILED’.

set_post_download_info()

Scan SOC directory and record downloaded granules.

Refreshes _soc_manifest.txt first so subsequent soc_file_tree calls (here and on the next resume) skip the recursive glob over the SOC tree. This is the same pattern the H3 database uses via MANIFEST_FILENAME.

get_finished_granules()

Return skip list when resuming with same filters.

Only returns successfully completed granules, with the status field stripped for backward compatibility with _filter_granules() dict comparison. Legacy logs without status fields are treated as successful.

get_temporal()

get_spatial()

get_product_vars()

to_dict(status)

save_log(status)

gedih3.logger.stored_exclude(log_data) → list

The source exclusions a build log keeps (gh3_build --exclude).

A single pattern written by hand as a string is one pattern, never its characters ('*' alone would exclude every file).

Raises:

GediValidationError

If exclude is neither a string nor a list of strings.

class gedih3.logger.H3BuildLogger(product_vars, spatial=None, temporal=None, res: int = 12, part: int = 3, version: int = None, dir=None, source_mode=None)

default_products

preset_products

preset_extra_vars

log_file

log_data

updating = False

update_ran = False

source_mode = None

build_start_time

previous_status

allow_missing_products

exclude

new_spatial = None

new_temporal = None

new_product_vars = None

requested_expansion

get_temporal()

get_spatial()

get_product_vars()

register_pending_granules(granule_list)

Register granules as PENDING. Called before build starts.

Parameters:

granule_listlist of dict

Each dict must have ‘orbit’, ‘granule’, ‘track’ keys, and may have releases: {product: tag} (gedi_release_tag()) of the source files the build lists for the granule. A newer release within the GEDI version is detected against the recorded tags (#47), so a tag is never replaced here: a granule not indexed yet gains tags only for products it has none for (rows a killed run left on disk may come from the file recorded first, and an older tag is reported later rather than a newer one hiding stale rows). An INDEXED or NO_DATA entry, and one registered before tags were recorded (no releases key, so rows on disk may come from any file), are left as they are. No tag means the source is unknown: an older log, a product added by a variable update, or an S3-sourced build (which registers no granules).

update_granule_status(gran_key, status)

Update status of a single granule (does NOT auto-save).

Caller controls save frequency via explicit save_log() calls.

Parameters:

gran_keydict

Dict with ‘orbit’, ‘granule’, ‘track’ keys.

statusstr

‘INDEXED’ or ‘PENDING’.

get_finished_granules()

Return skip list when resuming with same filters.

Only returns successfully completed granules, with the status and products fields stripped for backward compatibility with _filter_granules() dict comparison (which expects bare {'orbit', 'granule', 'track'} dicts). Legacy logs without status fields are treated as successful.

NO_DATA granules (read in full, zero rows) are skipped under exactly the conditions INDEXED ones are. They never stay skipped past a scope change that could make them non-empty: __init__ demotes every NO_DATA granule to PENDING as soon as the requested region or dates widen the stored ones, by any area, before this is consulted.

reads_added_area_only()

True when Stage 1 reads only the region this update adds (a spatial-only expansion).

stage1_only_granules()

Granules Stage 1 may read in this run, or None when it may read any.

A spatial-only expansion hands Stage 1 only the added region (get_spatial()). That is exactly what a granule finished under the stored region still needs. Any other granule (new to the database, PENDING, FAILED, a NO_DATA reopened by the widening) would have its rows outside the added region dropped, while its rows inside made it INDEXED and kept later builds from ever reading the rest (#42). Such granules are left as they are for the next build, which reads them over the whole region.

Returns:

list of dict or None

{'orbit', 'granule', 'track'} dicts, as for get_finished_granules().

is_up_to_date()

Check if database already matches requested parameters.

Returns True when all of these hold: - Log exists (updating=True) - No new products, spatial, or temporal changes detected - No pending variable update from a previous crash - All tracked granules are INDEXED (or NO_DATA: read, nothing to index)

set_post_build_info(verify_observed=False)

Fold partition metadata into the log: granule status, columns, dates.

Also records self._observed_granule_keys: the (orbit, granule, track) of every granule the partition metadata lists, or None when that listing cannot be trusted to be complete (no partitions, an unreadable scan). With verify_observed=True the scan additionally checks each partition’s sidecars against its parquet files and leaves None on any mismatch; mark_no_data() requires a set.

stage1_full_scope()

True when Stage 1 reads under the whole database scope.

get_spatial / get_temporal hand Stage 1 only the ADDED area (or dates) on a spatial-only (temporal-only) expansion, so a granule empty under that diff may hold rows in the old scope. Zero rows then proves nothing, and mark_no_data() refuses. Full scope: a fresh build, a plain resume or update (the log already holds the union), and a mixed update’s Phase 1. Mirrors the branches of the two getters.

mark_no_data(listed_keys, incomplete_keys, empty_keys)

Record granules Stage 1 read in full that hold no rows as NO_DATA.

Call after the final set_post_build_info() of a build whose Stage 1 drained fully; incomplete_keys is that run’s set of granules with any task not proven complete, empty_keys those whose every beam task is proven complete AND empty. Nothing is marked without a trustworthy observed set (set_post_build_info could not verify the partition metadata) or when Stage 1 did not run under the full database scope (stage1_full_scope()). In doubt a granule stays PENDING: that costs a re-read, a wrong NO_DATA costs rows.

• PENDING -> NO_DATA when listed, in empty_keys and absent from every partition’s metadata (the last guard matters for tasks emptied by the skip check, whose rows are already in the database);

• NO_DATA -> PENDING when listed and incomplete this run;

• MERGE_FAILED is never touched.

NO_DATA drops the products map and fill_attempts: there is nothing to fill. Does not save. Returns {'marked', 'reopened'}.

Parameters:

listed_keysiterable of tuple

Granules this run listed for Stage 1.

incomplete_keysiterable of tuple

Granules with any task not proven complete this run.

empty_keysiterable of tuple

Granules whose every beam task is proven complete and empty.

to_dict(status)

save_log(status)

get_granule_product_status(gran_key, product)

Return per-product status for a granule, or None if unknown.

Parameters:

gran_keydict or tuple

Either {'orbit', 'granule', 'track'} or (orbit, granule, track).

productstr

Product code (e.g. ‘L4A’).

get_product_gaps(active_products=None, gap_statuses=None)

List granules where any active product is not INDEXED.

Returns a list of (gran_key_dict, [missing_products]) pairs. Used by gh3_doctor as the fast-path audit (log-only, no parquet scan).

Parameters:

active_productsiterable of str, optional

Products to check. Defaults to self.product_vars.keys().

gap_statusesiterable of str, optional

Per-product statuses considered as gaps. Defaults to everything except INDEXED (i.e. PARTIAL_NAN, MISSING_COLUMN, MISSING_SOURCE, FAILED, PENDING).

mark_granule_product(gran_key, product, status)

Set per-product status for a granule (does not auto-save).

Caller is responsible for invoking save_log() to persist. Used by gh3_doctor after a fix or verification step.

set_product_statuses(updates)

Bulk per-product status update (does not auto-save).

updates maps (orbit, granule, track) to {product: status}. One pass over granule_info whatever the number of updates, unlike repeated mark_granule_product() calls. Returns the count applied.

superseded_releases(soc_entries)

Indexed granules whose listed source differs from the release read.

Compares each INDEXED or NO_DATA granule’s recorded releases with the file a listing would read now. Log-only, no I/O. A product whose rows name several releases (mixed_releases, see adopt_row_releases()) compares its oldest, so a listing of the newer one reports it until it is re-indexed whole. A product with no recorded tag is unknown and never reported. A log with no GEDI version reports nothing: tags carry no version, so a file of another release would read as a re-release.

Parameters:

soc_entriesiterable of dict

{product: path} dicts: per granule, as soc_file_tree(..., to_list=True) returns them, or one per file. Where a granule product is listed more than once, its latest release (gedi_release_rank) is the one compared, the file a build reads. Only granules with a recorded tag are ranked.

Returns:

dict

{'newer': {key: {product: (recorded, listed)}}, 'older': {...}} keyed by (orbit, granule, track). newer: a re-release within the GEDI version that the rows do not come from (#47). older: only an older release than the one read is listed; it is never applied, but variables added and products filled later would read it.

pending_product_fills(statuses=(PRODUCT_STATUS_MISSING_SOURCE,))

Indexed granules still waiting for products published after them.

{(orbit, granule, track): [product, ...]} for every INDEXED granule with a product in one of statuses. The default, MISSING_SOURCE, is what gh3_build backfills automatically; gh3_doctor also passes FAILED (fills given up on, see record_fill_results()) to retry them on demand. Log-only, no I/O.

record_releases(releases)

Record the source releases a write is about to read (does not save).

Parameters:

releasesdict

{(orbit, granule, track): {product: tag}} (tags from gedi_release_tag), e.g. from _build_fill_products’ releases_callback before it writes. As in register_pending_granules(), a product that has a tag keeps it (rows already written may come from that release), and an entry recorded before tags existed (no releases key) stays unknown: its other products’ rows may come from any file.

Returns:

int

Number of tags added.

static compared_releases(granule, newest=False)

{product: tag} a granule’s rows are compared by: its recorded releases, plus the oldest (newest=True: the newest) of the releases a product’s rows name when they name several (mixed_releases). Log-only.

row_release_targets()

Indexed granules whose releases were never recorded nor read from their rows (#67).

INDEXED granules with no releases key: a database built before gedih3 0.20.10 recorded none, and an entry registered by such a version before it was indexed keeps none (its rows may come from any file). Their rows name their source files (root_file_<prod>), so adopt_row_releases() takes them from there. Empty when the database has no source column. Log-only.

Returns:

set of tuple

(orbit, granule, track) keys.

adopt_row_releases(releases, keys, mixed=None)

Record the releases the database’s own rows were read from (does not save).

Parameters:

releasesdict

{(orbit, granule, track): {product: tag}} from the rows’ root_file_<prod> columns (gh3builder.releases_from_rows), for products whose rows name a single release. Unlike record_releases() (the files a write is about to read), these are what the rows hold.

keysiterable of tuple

The granules whose rows were all read (row_release_targets() less those an unreadable file may hold). Each gets a releases entry, empty when its rows named none, so it is not read again; a product not INDEXED for it stays out.

mixeddict, optional

{key: {product: [tags]}} for products whose rows name several releases (a replacement left half done). Kept apart as the granule’s mixed_releases, never as its release: reported by gh3_doctor --check superseded_granules, compared by its oldest (superseded_releases()), and dropped when it is re-indexed.

Returns:

int

Number of tags added, mixed ones included.

mixed_release_granules()

{(orbit, granule, track): {product: [tags]}}: granules whose rows name several releases of a product (mixed_releases). Log-only.

record_fill_results(result, max_attempts=3, count_failures=True)

Apply a _build_fill_products result to per-product statuses.

• filled → INDEXED (attempt count cleared);

• unlocated (the product file exists but no database file lists the granule, so it can never fill) → FAILED at once;

• failed (read or merge error, or no row matched) → retried by the next runs; after max_attempts consecutive failures → FAILED, so one bad file does not make every later build rescan. FAILED is not backfilled automatically; gh3_doctor --fix backfill retries it. count_failures=False leaves failures untouched.

Does not save the log. Returns {key: [products]} newly FAILED.