gedih3.doctor.parquet_ops#

Streaming parquet operations specific to gh3_doctor.

Both functions process parquet files row-group by row-group to bound memory.

parquet_fill_columns() now lives in gedih3.utils (the build’s product backfill uses it too) and is re-exported here: the fill counterpart of gedih3.utils.parquet_join_columns() — null/NaN base cells take the patch value, existing values are never overwritten.
parquet_dedup_partition() rewrites a single parquet file dropping duplicate shot_number rows, keeping the first occurrence.

Functions#

parquet_dedup_partition(→ int)

Rewrite a parquet file dropping duplicate key_col rows.

Module Contents#

gedih3.doctor.parquet_ops.parquet_dedup_partition(pq_file: str, key_col: str = 'shot_number', keep: str = 'first', tmp_suffix: str = '.dedup.tmp') → int[source]#

Rewrite a parquet file dropping duplicate key_col rows.

Streams row-group by row-group: only one row group plus the cumulative set of seen keys is held in memory. shot_number is int64 so the seen-set cost is ~8 bytes/row.

Parameters:

pq_filestr: Parquet file to rewrite in-place (via temp + atomic rename).
key_colstr, default ‘shot_number’: Column to deduplicate on.
keep{‘first’, ‘last’}, default ‘first’: Which duplicate to keep. 'last' requires a second pass.
tmp_suffixstr: Suffix for the temp file used during atomic rewrite.

Returns:

int: Number of rows dropped.