energy_data_retrieval.pv_catalogue.catalogue#
Deterministic JSONL source records and typed, compressed Parquet snapshots.
Network access is never implicit. Changes require field-level source references; conflicting populated values are returned for review instead of overwritten.
- energy_data_retrieval.pv_catalogue.catalogue.canonical_json(value)[source]#
Canonical nested JSON, rejecting non-finite numbers.
- Parameters:
value (Any)
- Return type:
str
- energy_data_retrieval.pv_catalogue.catalogue.stable_id(entity_type, *identity)[source]#
Stable identifier; keep all punctuation that may distinguish OEM models.
Identity is chosen once. Renames become aliases; IDs never include mutable specs. Project identity should use a registry ID, or country + locality + original name.
- Parameters:
entity_type (str)
identity (str)
- Return type:
str
- energy_data_retrieval.pv_catalogue.catalogue.validate_record(row, sources)[source]#
Reject schema drift, unsupported values, invalid units/ranges and missing evidence.
- Parameters:
row (dict[str, Any])
sources (dict[str, Any])
- Return type:
None
- energy_data_retrieval.pv_catalogue.catalogue.validate_catalogue(rows, sources)[source]#
Validate uniqueness, natural equipment keys and referenced equipment types.
- Parameters:
rows (list[dict[str, Any]])
sources (dict[str, Any])
- Return type:
None
- energy_data_retrieval.pv_catalogue.catalogue.load_records(root)[source]#
Read the sharded, sorted JSONL working dataset.
- Parameters:
root (Path)
- Return type:
list[dict[str, Any]]
- energy_data_retrieval.pv_catalogue.catalogue.write_records(root, rows)[source]#
Write stable hash shards (<2 MB at launch); validate before any mutation.
Single-writer operation: automation must use an isolated Git branch and a push lease. Git is the transaction/history boundary; do not share a working tree.
- Parameters:
root (Path)
rows (list[dict[str, Any]])
- Return type:
None
- energy_data_retrieval.pv_catalogue.catalogue.upsert_records(existing, incoming, *, now)[source]#
Fill nulls, preserve creation and quarantine conflicting populated fields.
No deletion, no silent winner, no last-writer-wins. Identical observations do not change updated_at. Conflicts leave the entire existing record unchanged. Resolving one requires a reviewed explicit edit with evidence and an updated timestamp.
- Parameters:
existing (list[dict[str, Any]])
incoming (list[dict[str, Any]])
now (str)
- Return type:
tuple[list[dict[str, Any]], list[dict[str, Any]]]
- energy_data_retrieval.pv_catalogue.catalogue.build_parquet(root, output)[source]#
Build an immutable snapshot, checksums and coverage; output must not exist.
All five tables use an explicit union schema, including empty/all-null columns. Nested sources remain lists. Timestamps are native UTC Arrow timestamps.
- Parameters:
root (Path)
output (Path)
- Return type:
dict[str, Any]