energy_data_retrieval.storage#

Partition-sized Parquet outputs with verifiable, restart-safe provenance.

class energy_data_retrieval.storage.Artifact(filename: 'str', sha256: 'str', bytes: 'int', rows: 'int', request_fingerprint: 'str')[source]#

Bases: object

Parameters:
  • filename (str)

  • sha256 (str)

  • bytes (int)

  • rows (int)

  • request_fingerprint (str)

filename: str#
sha256: str#
bytes: int#
rows: int#
request_fingerprint: str#
energy_data_retrieval.storage.sha256_file(path)[source]#
Parameters:

path (Path)

Return type:

str

energy_data_retrieval.storage.fingerprint(metadata)[source]#

Fingerprint only non-secret request semantics and source/time conventions.

Parameters:

metadata (Mapping[str, JSONValue])

Return type:

str

energy_data_retrieval.storage.existing_artifact(path, request_fingerprint)[source]#
Parameters:
  • path (Path)

  • request_fingerprint (str)

Return type:

Artifact | None

energy_data_retrieval.storage.write_parquet(frame, path, *, request, provenance)[source]#

Publish a Parquet partition, then its manifest; refuse mismatched overwrites.

A crash between the two atomic publications is detected on restart. The user must inspect that incomplete pair; an unaudited file is never silently reused.

Parameters:
  • frame (DataFrame)

  • path (Path)

  • request (Mapping[str, JSONValue])

  • provenance (Mapping[str, JSONValue])

Return type:

Artifact

energy_data_retrieval.storage.bounded_map(function, items, *, workers=2)[source]#

Yield completion-order results with at most workers tasks in flight.

Parameters:
  • function (Callable[[Input], Output])

  • items (Iterable)

  • workers (int)

Return type:

Iterator