torchcell.data.Deduplicator

class Deduplicator(root: str)

Bases: ABC

Base class that deduplicates experiment records into an output LMDB store.

abstractmethod duplicate_check(data: Any) → dict[str, list[int]]

Return a mapping of hash key to the record indices sharing that key.

abstractmethod duplicate_key(record: dict[str, Any]) → str

Return the duplicate-group hash for ONE raw record dict (pre-pydantic).

Streaming counterpart of duplicate_check(): process calls this per record straight off the stored JSON, so grouping never requires the whole dataset in memory. Must produce the same key the pydantic-based duplicate_check would for that record.

abstractmethod create_deduplicate_entry(duplicate_experiments: list[dict[str, Any]]) → dict[str, Any]

Merge a group of duplicate experiments into a single representative entry.

close_lmdb() → None

Close the LMDB environment if it is open.

process(input_path: str, output_path: str, key_groups: dict[str, list[bytes]] | None = None) → None

Read records from input_path, deduplicate, and write to the output LMDB.

Two streaming passes, never the whole dataset in memory. The previous implementation materialized every record as pydantic objects in one list (plus a second full list of outputs) – at the 025 build’s 43.8M records that OOM-killed slurm job 1560 immediately after a completed conversion.

Pass 1 groups input KEYS by duplicate_key computed off the raw JSON. Pass 2 writes groups in first-occurrence order (dict insertion order, the same order the old code produced): singleton groups pass the stored bytes through untouched – they were written by the converter’s model_dump, so a reconstruct-and-redump would be an identity round trip – and only genuine duplicate groups are reconstructed for create_deduplicate_entry.

key_groups, when given, IS pass 1’s result and pass 1 is not run (same contract as Aggregator.process: groups in the order a cursor walk first meets them, keys in cursor order within each group), computed from duplicate_key while the raw stage writes the input.