gedih3.doctor.diagnoses.backfill#

backfill diagnosis — fill NaN gaps in product columns from source HDF5s.

Granules indexed before some of their products were published (gh3_build --allow-missing-products) are known from the build log alone: missing_source findings, no data scan. Their fix is the build’s own product backfill (gedih3.gh3builder._build_fill_products(): each product file read once, only the files holding those granules rewritten, only null cells written) — the same engine gh3_build runs automatically.

The detection below additionally runs row-level: for each partition × product, we count nulls per granule by grouping on root_file_l2a. Findings include the partition file, the product, and the per-granule null counts.

The fix:

  1. Group findings by source granule × product to compute the minimal set of HDF5 files to read.

  2. (S3 mode) run s3_etl_subset() to populate a temp dir with just those granules’ source files for the requested products. Cleanup in finally.

  3. For each affected partition file, read the relevant [shot_number, vars] from the source HDF5(s), suffix columns with _<prod>, write to a temporary patch parquet, and call parquet_fill_columns() to merge.

  4. Mark each healed granule × product as INDEXED in the build log.

Attributes#

GranuleKey

Functions#

backfill_check(→ gedih3.doctor.report.Report)

backfill_fix(→ gedih3.doctor.report.Report)

Module Contents#

gedih3.doctor.diagnoses.backfill.GranuleKey#
gedih3.doctor.diagnoses.backfill.backfill_check(ctx: gedih3.doctor.report.DoctorContext) → gedih3.doctor.report.Report[source]#
gedih3.doctor.diagnoses.backfill.backfill_fix(ctx: gedih3.doctor.report.DoctorContext, report: gedih3.doctor.report.Report) → gedih3.doctor.report.Report[source]#