Downloading datasets

Built torchcell datasets are served as packaged archives by tc-data, a public read-only HTTP endpoint with Swagger documentation. Each archive holds a dataset’s processed/ directory (the records LMDB and its interned sibling) and preprocess/ directory (build_manifest.json, gene_set.json, experiment_reference_index.json), exactly as the loader wrote them, so a download replaces the build step. The raw files each loader consumed are served from the same endpoint, hash-pinned by their manifest.json, and so are two more file tiers that graph records point at without storing their bytes: the genomes tier (reference and isolate assembly sets) and the objects tier (derived files such as embeddings, matrices and indexed FASTA).

The endpoint is http://torchcell-database.ncsa.illinois.edu:8724, on the database host, serving the archive store and the raw mirror from the Taiga project storage. No key, account or ssh is needed: every route is open for reading. The URL is built into the package as torchcell.endpoint_http.DEFAULT_URL, and TC_DATA_URL overrides it for a mirror or a development server. The store currently holds gene_essentiality_sgd, smf_costanzo2016, amino_acid_cooper2010, amino_acid_mulleder2016 and betaxanthin_cachera2023; GET /datasets is the authoritative list.

Swagger

The interactive documentation lives at <TC_DATA_URL>/docs and the OpenAPI schema at <TC_DATA_URL>/openapi.json. The routes:

Route	Returns
`GET /health`	liveness, artifact and raw-key counts
`GET /datasets`	the artifact index: one row per packaged archive
`GET /datasets/{slug}`	every packaged version of one dataset
`GET /datasets/{slug}/{archive}`	the archive; `X-Artifact-SHA256`, `Accept-Ranges: bytes`
`GET /raw`	citation keys in the raw mirror
`GET /raw/{citation_key}/manifest`	the key’s `manifest.json`
`GET /raw/{citation_key}/files`	the files that manifest lists, with sha256
`GET /raw/{citation_key}/artifact/{path}`	one raw file; `X-Artifact-SHA256`
`GET /genomes`	assembly sets in the genomes tier that carry a manifest
`GET /genomes/{assembly_set}/manifest`	the set’s genome manifest
`GET /genomes/{assembly_set}/files`	the files that manifest lists, with sha256
`GET /genomes/{assembly_set}/artifact/{path}`	one genomes file; `X-Artifact-SHA256`, `Accept-Ranges: bytes`
`GET /objects`	object keys in the objects tier that carry a manifest
`GET /objects/{object_key}/manifest`	the key’s `manifest.json`
`GET /objects/{object_key}/files`	the files that manifest lists, with sha256
`GET /objects/{object_key}/artifact/{path}`	one objects file; `X-Artifact-SHA256`, `Accept-Ranges: bytes`

An index row records the dataset slug, the loader dataset_class, the torchcell_version and git commit the LMDB was built with, the knowledge-graph release it was admitted to, content_sha256 (over the experiment ids, comparable with the release manifest), the archive name, archive_sha256, archive_bytes, and status (supported or deprecated). Pick the newest supported row whose torchcell_version shares your installed major.minor.

Raw, genomes and objects tiers

The three file tiers share one serving rule. Each key (a citation key, an assembly set, or an object key) is a directory with a manifest.json listing every file by path, role, size and sha256; a file is served only if its key’s manifest lists it, and the X-Artifact-SHA256 header of the response is the manifest’s hash, so a client checks the bytes it received against it. Every file route honors Range, which is how an interrupted download of a large assembly container resumes.

Raw (/raw): the source files each dataset loader consumed, keyed by citation key; the manifest is the literature Manifest.
Genomes (/genomes): sequence-level reference data, one assembly set per key (for example the SGD S288C R64-4-1 release, or the Peter et al. 2018 isolate assemblies); the manifest is a genome manifest naming the organism, strain or population, source and release.
Objects (/objects): derived files a record points at, keyed by a citation key or a named derived set; the manifest is the literature Manifest, and each file’s processing record names the script and inputs that produced it.

/genomes and /objects list only keys that carry a manifest, and an endpoint without one of those tiers answers its listing with an empty list.

Paper content is withheld. In the raw and objects tiers a manifest record whose role begins paper_ (paper_pdf, paper_text, paper_ocr, paper_fulltext) is left out of /manifest and /files, and a request for its bytes answers 404 paper content is not served. SI files and data files under the same key are served as usual.

curl

export TC_DATA_URL=http://torchcell-database.ncsa.illinois.edu:8724

curl "$TC_DATA_URL/datasets" | python -m json.tool
curl "$TC_DATA_URL/datasets/smf_costanzo2016"

# download an archive, resuming if interrupted, then verify it
ARCHIVE=smf_costanzo2016-1.2.1-d8a0f06c.tar.xz
curl -C - -o "$ARCHIVE" "$TC_DATA_URL/datasets/smf_costanzo2016/$ARCHIVE"
sha256sum "$ARCHIVE"   # must equal the row's archive_sha256

# unpack into a dataset root; processed/ and preprocess/ appear beside it
mkdir -p data/torchcell/smf_costanzo2016
tar -xf "$ARCHIVE" -C data/torchcell/smf_costanzo2016

# raw files
curl "$TC_DATA_URL/raw"
curl "$TC_DATA_URL/raw/<citation_key>/files"

# genomes and objects tiers: list, read the file rows, fetch one file, verify it
curl "$TC_DATA_URL/genomes"
curl "$TC_DATA_URL/genomes/<assembly_set>/files"
curl -C - -D headers.txt \
  -o "<file>" "$TC_DATA_URL/genomes/<assembly_set>/artifact/<path>"
grep -i x-artifact-sha256 headers.txt; sha256sum "<file>"   # the two must agree
curl "$TC_DATA_URL/objects"
curl "$TC_DATA_URL/objects/<object_key>/files"

Environment variables

Variable	Where	Meaning
`TC_DATA_URL`	client	endpoint base URL, overriding the built-in `DEFAULT_URL`; when set, loaders download instead of building
`TC_DATA_ROOT`	server	the artifact store (`index.json`, `SHA256SUMS`, `<slug>/<archive>`)
`TC_DATA_RAW_ROOT`	server	the raw mirror; defaults to `$DATA_ROOT/torchcell-raw`
`TC_DATA_GENOMES_ROOT`	server	the genomes tier; defaults to `$DATA_ROOT/torchcell-genomes`; may be absent
`TC_DATA_OBJECTS_ROOT`	server	the objects tier; defaults to `$DATA_ROOT/torchcell-objects`; may be absent
`TC_DATA_HOST`, `TC_DATA_PORT`	server	bind address and port (default `0.0.0.0:8724`)

From Python

With TC_DATA_URL set, any ExperimentDataset loader whose LMDB is absent fetches, verifies and unpacks its artifact instead of downloading the publisher files and running process(). If the endpoint has no artifact compatible with the installed version the loader raises and names the slug; it never falls back to the publisher path while the variable is set.

A loader prefers a download only when TC_DATA_URL is set explicitly; the built-in default does not switch loaders over. The variable is read from the process environment. Either export it in the shell, as in the curl section, or put the same line, without export, in a .env file in your project and load it before constructing a loader:

from dotenv import load_dotenv

load_dotenv()  # reads the .env beside this script or in a parent directory

from torchcell.datasets.scerevisiae.costanzo2016 import SmfCostanzo2016Dataset

dataset = SmfCostanzo2016Dataset(root="data/torchcell/smf_costanzo2016")

torchcell does not read .env on its own. A .env with no load_dotenv() call leaves TC_DATA_URL unset, and the loader then downloads the publisher files and builds, with no error. load_dotenv() never overrides a variable that is already exported.

The client needs no configuration: DatasetClient() with no arguments talks to the public endpoint, and DatasetClient.from_env() uses TC_DATA_URL when it is set and the public endpoint otherwise.

from torchcell.datasets.client import DatasetClient, unpack_artifact

client = DatasetClient.from_env()          # TC_DATA_URL if set, else the public endpoint
artifact = client.select("smf_costanzo2016")  # newest supported row on this major.minor
if artifact is None:
    raise SystemExit("nothing compatible with the installed torchcell")
archive = client.download(artifact, dest=f"artifacts/{artifact.archive}")  # verifies sha256
unpack_artifact(archive, "data/torchcell/smf_costanzo2016")

The raw mirror keys and files are one GET away; see the Swagger page for the exact shapes. All three file tiers have client methods:

from torchcell.datasets.client import DatasetClient

client = DatasetClient.from_env()
client.raw_manifest("<citation_key>")    # literature Manifest
client.raw_files("<citation_key>")       # path, role, bytes, sha256
client.download_raw_file("<citation_key>", "<path>", dest="raw/<path>")
client.genomes()                       # ["peter2018_1011_assemblies", ...]
client.genome_manifest("peter2018_1011_assemblies")   # GenomeManifest
client.genome_files("peter2018_1011_assemblies")      # path, role, bytes, sha256
client.download_genome_file(
    "peter2018_1011_assemblies", "<path>", dest="genomes/<path>"
)
client.objects()
client.object_manifest("<object_key>")  # literature Manifest
client.download_object_file("<object_key>", "<path>", dest="objects/<path>")

Each download looks up the file’s row in the key’s manifest first (a path the manifest does not list raises KeyError before any request), resumes a <dest>.part with a Range request, and requires both the server’s X-Artifact-SHA256 and the hash of the received bytes to equal the manifest row; on a mismatch the partial file is removed and ArtifactIntegrityError is raised.