energy_data_retrieval.pv_catalogue.catalogue#

Deterministic JSONL source records and typed, compressed Parquet snapshots.

Network access is never implicit. Changes require field-level source references; conflicting populated values are returned for review instead of overwritten.

energy_data_retrieval.pv_catalogue.catalogue.canonical_json(value)[source]#

Canonical nested JSON, rejecting non-finite numbers.

Parameters:

value (Any)

Return type:

str

energy_data_retrieval.pv_catalogue.catalogue.stable_id(entity_type, *identity)[source]#

Stable identifier; keep all punctuation that may distinguish OEM models.

Identity is chosen once. Renames become aliases; IDs never include mutable specs. Project identity should use a registry ID, or country + locality + original name.

Parameters:
  • entity_type (str)

  • identity (str)

Return type:

str

energy_data_retrieval.pv_catalogue.catalogue.validate_record(row, sources)[source]#

Reject schema drift, unsupported values, invalid units/ranges and missing evidence.

Parameters:
  • row (dict[str, Any])

  • sources (dict[str, Any])

Return type:

None

energy_data_retrieval.pv_catalogue.catalogue.validate_catalogue(rows, sources)[source]#

Validate uniqueness, natural equipment keys and referenced equipment types.

Parameters:
  • rows (list[dict[str, Any]])

  • sources (dict[str, Any])

Return type:

None

energy_data_retrieval.pv_catalogue.catalogue.load_records(root)[source]#

Read the sharded, sorted JSONL working dataset.

Parameters:

root (Path)

Return type:

list[dict[str, Any]]

energy_data_retrieval.pv_catalogue.catalogue.write_records(root, rows)[source]#

Write stable hash shards (<2 MB at launch); validate before any mutation.

Single-writer operation: automation must use an isolated Git branch and a push lease. Git is the transaction/history boundary; do not share a working tree.

Parameters:
  • root (Path)

  • rows (list[dict[str, Any]])

Return type:

None

energy_data_retrieval.pv_catalogue.catalogue.upsert_records(existing, incoming, *, now)[source]#

Fill nulls, preserve creation and quarantine conflicting populated fields.

No deletion, no silent winner, no last-writer-wins. Identical observations do not change updated_at. Conflicts leave the entire existing record unchanged. Resolving one requires a reviewed explicit edit with evidence and an updated timestamp.

Parameters:
  • existing (list[dict[str, Any]])

  • incoming (list[dict[str, Any]])

  • now (str)

Return type:

tuple[list[dict[str, Any]], list[dict[str, Any]]]

energy_data_retrieval.pv_catalogue.catalogue.build_parquet(root, output)[source]#

Build an immutable snapshot, checksums and coverage; output must not exist.

All five tables use an explicit union schema, including empty/all-null columns. Nested sources remain lists. Timestamps are native UTC Arrow timestamps.

Parameters:
  • root (Path)

  • output (Path)

Return type:

dict[str, Any]