torchcell.paper.tables.stream_gzip_signal

stream_gzip_signal(lmdb_dir: str | ~pathlib.Path, *, extract: ~collections.abc.Callable[[dict[str, ~typing.Any]], bytes] = <function default_phenotype_bytes>, level: int = 6, log_every: int = 1000000, label: str = '', log: ~collections.abc.Callable[[str], None] = <built-in function print>) → tuple[int, int]

Return (n_records, gzip_bytes) for an LMDB in a single streaming pass.

Byte-equivalent to gzip-compressing the concatenation of extract(record) over every record, but via zlib.compressobj so memory stays bounded – this is what lets 20M+ record LMDBs (tens of GB) get a real value instead of “pending”. extract is pluggable so the same machinery can measure other per-record payloads for future tables.