energy_data_retrieval.transport#

Bounded HTTP reads with source allowlists, retries and public-only caching.

exception energy_data_retrieval.transport.RetrievalError[source]#

Bases: RuntimeError

Sanitized retrieval failure; never contains a credential-bearing URL.

class energy_data_retrieval.transport.HTTPConfig(timeout_seconds=45.0, attempts=4, retry_base_seconds=1.0, maximum_retry_seconds=60.0, minimum_interval_seconds=0.25, maximum_response_bytes=67108864, cache_ttl_seconds=3600.0)[source]#

Bases: object

Finite resource and retry limits, shared by all sources.

Parameters:
  • timeout_seconds (float)

  • attempts (int)

  • retry_base_seconds (float)

  • maximum_retry_seconds (float)

  • minimum_interval_seconds (float)

  • maximum_response_bytes (int)

  • cache_ttl_seconds (float)

timeout_seconds: float#
attempts: int#
retry_base_seconds: float#
maximum_retry_seconds: float#
minimum_interval_seconds: float#
maximum_response_bytes: int#
cache_ttl_seconds: float#
class energy_data_retrieval.transport.RequestRecord(endpoint, parameters, retrieved_at_utc, sha256, size_bytes, cache_hit=False)[source]#

Bases: object

Non-secret provenance for the exact stored, sanitized response bytes.

Parameters:
  • endpoint (str)

  • parameters (dict[str, str])

  • retrieved_at_utc (str)

  • sha256 (str)

  • size_bytes (int)

  • cache_hit (bool)

endpoint: str#
parameters: dict[str, str]#
retrieved_at_utc: str#
sha256: str#
size_bytes: int#
cache_hit: bool#
class energy_data_retrieval.transport.JSONResponse(value: 'JSONValue', record: 'RequestRecord')[source]#

Bases: object

Parameters:
value: JSONValue#
record: RequestRecord#
class energy_data_retrieval.transport.TextResponse(value, record)[source]#

Bases: object

UTF-8 response with a hash of its stored bytes after credential redaction.

Parameters:
value: str#
record: RequestRecord#
class energy_data_retrieval.transport.BytesResponse(value, record, content_type, request_id)[source]#

Bases: object

Exact public response bytes, media type and provider request identifier.

Parameters:
  • value (bytes)

  • record (RequestRecord)

  • content_type (str)

  • request_id (str | None)

value: bytes#
record: RequestRecord#
content_type: str#
request_id: str | None#
energy_data_retrieval.transport.atomic_bytes(path, data)[source]#

Replace one file atomically; failed writes never publish a partial file.

Parameters:
  • path (Path)

  • data (bytes)

Return type:

None

energy_data_retrieval.transport.object_value(value)[source]#
Parameters:

value (JSONValue)

Return type:

dict[str, JSONValue]

energy_data_retrieval.transport.array_value(value)[source]#
Parameters:

value (JSONValue)

Return type:

list[JSONValue]

class energy_data_retrieval.transport.HTTPClient(allowed_hosts, *, config=None, cache_directory=None, transport=None, sleep=<built-in function sleep>)[source]#

Bases: object

Bounded HTTPS retrieval with a fixed host allowlist and no redirects.

Authentication must be passed separately through secrets or secret_headers. Authenticated responses never use the public cache. Supply an httpx transport for offline tests, rather than a preauthenticated client. This ensures authorization cannot silently bypass cache isolation.

Parameters:
  • allowed_hosts (frozenset[str])

  • config (HTTPConfig | None)

  • cache_directory (Path | None)

  • transport (httpx.BaseTransport | None)

  • sleep (Callable[[float], None])

close()[source]#
Return type:

None

request_bytes(endpoint, parameters=None, *, method='GET', form=None, files=None)[source]#

Retrieve public binary/text data or submit an idempotent calculation.

POST is for read-only provider calculations, never state changes. Public fields only; no authentication and no cache. List-valued query/form fields retain repeated keys. Uploads and responses obey the configured byte limit. Provenance includes method, form values and upload hashes. Redirects are refused; retries include 529 (PVGIS computation overload).

Parameters:
  • endpoint (str)

  • parameters (Mapping[str, str | list[str]] | None)

  • method (Literal['GET', 'POST'])

  • form (Mapping[str, str | list[str]] | None)

  • files (Mapping[str, tuple[str, bytes, str]] | None)

Return type:

BytesResponse

download_file(endpoint, path, *, expected_sha256=None, expected_size_bytes=None)[source]#

Stream one public export to disk with verified whole-file restart semantics.

An interrupted transfer is retried from byte zero. HTTP ranges are not assumed safe for changing archives. Reuse requires both an intact file and its matching manifest. Credential-bearing and redirected URLs fail. A supplied source byte count is checked before publishing or reusing a file.

Parameters:
  • endpoint (str)

  • path (Path)

  • expected_sha256 (str | None)

  • expected_size_bytes (int | None)

Return type:

RequestRecord

get_text(endpoint, parameters=None, *, secrets=None, secret_headers=None, use_cache=True)[source]#

Read bounded UTF-8 text, retaining tabs, line endings and physical units.

Credentials must use secrets or secret_headers. Authenticated responses bypass the public cache; logs, response text and provenance redact credentials. Public CSV/PSV/TSV bytes remain unchanged. Cache entries are accepted only after source, size and hash verification.

Parameters:
  • endpoint (str)

  • parameters (Mapping[str, str] | None)

  • secrets (Mapping[str, str] | None)

  • secret_headers (Mapping[str, str] | None)

  • use_cache (bool)

Return type:

TextResponse

get_json(endpoint, parameters=None, *, secrets=None, secret_headers=None, allow_legacy_charset=False, use_cache=True, public_pagination_token=False)[source]#

Fetch JSON, preserving zero/null values and non-secret request provenance.

public_pagination_token opts into a provider’s non-authentication token cursor, including next links on its first page. Other credential names and explicit secrets remain redacted; authentication tokens cannot use this option. The default treats all token parameters as secrets. Header credentials are never included in request provenance or cache keys. allow_legacy_charset permits only a declared Latin-1/Latin-9 fallback when Unicode JSON decoding fails; it never overrides valid Unicode JSON.

Parameters:
  • endpoint (str)

  • parameters (Mapping[str, str] | None)

  • secrets (Mapping[str, str] | None)

  • secret_headers (Mapping[str, str] | None)

  • allow_legacy_charset (bool)

  • use_cache (bool)

  • public_pagination_token (bool)

Return type:

JSONResponse