gedih3.doctor.inspect#

Read-only inspectors shared across diagnoses.

These helpers do not mutate the database. They consolidate filesystem and parquet I/O so individual diagnoses don’t re-walk the same directories.

Attributes#

GranuleKey

Functions#

`discover_partition_dirs`(→ List[str])	List `h3_*/` directories under the database root.
`partition_meta_file`(→ Optional[str])	Locate the merged partition-level metadata file.
`load_partition_meta`(→ Optional[dict])	Read the merged partition meta JSON (or None if missing).
`partition_parquet_files`(→ List[str])	List the per-year parquet files for one partition.
`per_granule_null_counts`(→ Dict[GranuleKey, Dict[str, int]])	Count nulls per (granule, product) pair within one parquet file.
`product_columns_in_schema`(→ Dict[str, List[str]])	Group columns by product suffix.

Module Contents#

gedih3.doctor.inspect.GranuleKey#

gedih3.doctor.inspect.discover_partition_dirs(h3_dir: str) → List[str][source]#: List h3_*/ directories under the database root.

gedih3.doctor.inspect.partition_meta_file(partition_dir: str) → str | None[source]#

Locate the merged partition-level metadata file.

Per h3_merge_metadata (gh3builder.py:475), each partition has a single <h3_part>_gedih3_partition_meta.json at its root. Returns None if absent.

gedih3.doctor.inspect.load_partition_meta(partition_dir: str) → dict | None[source]#: Read the merged partition meta JSON (or None if missing).

gedih3.doctor.inspect.partition_parquet_files(partition_dir: str) → List[str][source]#: List the per-year parquet files for one partition.

gedih3.doctor.inspect.per_granule_null_counts(parquet_file: str, product_columns: Dict[str, List[str]], granule_lookup: Dict[str, GranuleKey] | None = None) → Dict[GranuleKey, Dict[str, int]][source]#

Count nulls per (granule, product) pair within one parquet file.

Granule grouping uses the root_file_l2a column (the canonical source- granule pointer written by h3_write_metadata). This matches the granule identifiers stored in partition metadata exactly, avoiding any shot_number decoding ambiguity.

Memory pillar (v0.8.x lessons applied):

Streamed row-group at a time via pq.ParquetFile.iter_batches with pre_buffer=True and a small batch_size — not the full pd.read_parquet(columns=cols_to_read) that used to materialize every product column × millions of rows in one shot (multiple GB per file on continental partitions).
Each batch’s null counts are aggregated into a small accumulator dict (O(num_granules × num_products)) and the batch is dropped before the next is read — so worker RSS is bounded by one batch regardless of file size.
Native arrow null counting + groupby on a per-batch DataFrame; no full-file isnull().any(axis=1) allocation.
pa.default_memory_pool().release_unused() at end so the worker doesn’t drag transient buffers into the next task.

Parameters:

parquet_filestr: Path to a partition parquet file.
product_columnsdict: Map of product code → list of product-suffixed column names to scan.
granule_lookupdict, optional: Precomputed map {root_file_l2a_basename: (orbit, granule, track)}. If omitted, GEDIFile is invoked once per unique filename.

Returns:

dict: {(orbit, orbit_granule, track): {product_code: null_count}}. Only granule × product combos with at least one null are returned.

gedih3.doctor.inspect.product_columns_in_schema(columns: Iterable[str], products: Iterable[str]) → Dict[str, List[str]][source]#

Group columns by product suffix.

Returns {product_code: [columns...]} for each product whose suffix appears at least once. Products with no matching column are omitted.