Architecture#
Three subprojects, one reproducible workspace#
The repository has three independently buildable Python distributions. They use lowercase directory names and conventional import names:
Directory |
Distribution |
Python import |
Responsibility |
|---|---|---|---|
|
|
|
Provider clients, retrieval, normalization and provenance |
|
|
|
Country-neutral scientific algorithms |
|
|
|
Configured country studies, beginning with Chile’s PV fleet |
Each member has its own pyproject.toml, src layout, documentation and tests.
The root uv.lock resolves them together. A change to a scientific method and
its country-level caller can therefore be reviewed and tested in one pull
request. These are package subprojects with a shared Git history.
Dependencies flow from analysis to methods and data-retrieval. The methods
package performs no provider retrieval. Source clients know their provider’s
payloads and clocks; they do not decide which sites underperform.
Location discovery#
energy_data_retrieval.discovery composes the existing station catalogues and
grid products for a latitude/longitude query. The pure, vectorized spatial
method remains inside the retrieval package, which stays independently
installable without the scientific-model dependencies. Station adapters retain
source identities, catalogue scope, native metadata and explicit failure states.
Distance ranking handles the antimeridian and poles and never silently treats
an unavailable catalogue as evidence that no stations exist.
Grid discovery separates documented geographic coverage, source-catalogue evidence and an optional small data probe. CMIP6 plans retain their exact model, experiment, member, grid, version and native files; geographic coverage is not inferred from their presence in a catalogue. Discovery JSON snapshots include source request evidence, reference IDs and a content checksum. They inform a subsequent explicit retrieval and analysis selection. See location discovery.
Data contracts#
Generation and hourly weather use a typed, explicit tabular interchange schema. Hourly records identify a site and an offset-aware UTC timestamp. Energy is MWh per declared interval; irradiance is W/m², temperature °C and wind m/s. Coordinate units are decimal degrees. Capacity values are MW with DC and AC kept distinct. See each package’s schema documentation for its exact required columns and validation rules.
Missing data are missing, not zero. A reported zero is retained. Negative energy
is excluded from performance comparisons and remains distinguishable from an
unreported value. Duplicate site/time keys require an explicit resolution.
Naive local timestamps must not be silently shifted through a daylight-saving
ambiguity. Chile reporting dates use America/Santiago; TDP uses its explicitly
declared analysis clock.
Large raw responses, normalized Parquet, fitted models and report outputs live outside the source tree or in ignored directories. Provenance records bind inputs and outputs using hashes and record the relevant configuration. API credentials are supplied locally, not stored in Git or output manifests.
Native station observation contracts#
energy_data_retrieval.sources.bsrn uses PANGAEA’s documented DOI export
interface for one explicitly selected BSRN station/month. Its separate discovery
command returns DOI candidates through the current search endpoint used by
PANGAEA’s official client. That internal search interface has no stable public
REST contract; DOI retrieval validates the actual dataset independently.
The supported radiation records retain their original UTC minute labels,
statistics, missing and negative values, raw columns and numeric qualifiers.
Source text, JSON-LD metadata, citations and release conditions are saved with
the output. Archive annotations do not establish completed BSRN scientific
quality control. See the BSRN source contract
and official archive guidance.
energy_data_retrieval.sources.noaa retrieves the current GHCNh catalogue and
explicit station/year files from NOAA NCEI over HTTPS. Country and geographic
filters select station metadata; an observation request names an exact GHCNh
station ID, preserving its case. Generated output directories are lowercase
and distinguish IDs that differ only in case. Invalid catalogue coordinate pairs
remain in the raw fields and are flagged unavailable in normalized coordinates;
no coordinate swap or repair is inferred. Whole native annual PSV files are
acquired under response and row limits, then filtered to the requested UTC dates.
Measurements retain their
native physical units, report timestamps and raw quality, source and measurement
codes. GHCNh is NOAA’s current replacement for frozen ISD; no automatic legacy
fallback is used. See the NOAA source contract
and official GHCNh documentation.
energy_data_retrieval.sources.aemet follows the current AEMET OpenData
two-stage API: an authenticated product request returns approved temporary URLs
for data and metadata. The key is sent as a private HTTP header, excluded from
provenance and caches, and redacted from responses and logs. Daily climate
records retain their date semantics; recent conventional observations have a
separate rolling-window snapshot interface. Response metadata and canonical
JSON payloads are preserved. Canonical JSON is not described as exact wire bytes.
energy_data_retrieval.sources.dwd retrieves the hourly solar catalogue and
complete station ZIP archives from DWD CDC. The archive, station history and
parameter descriptions remain available. Native UTC minute offsets and WOZ
(local true solar time) labels are preserved, as are hourly radiation totals in
J/cm² and source quality levels. The reader does not infer interval boundaries
or silently turn accumulated energy into irradiance.
energy_data_retrieval.sources.surfrad retrieves NOAA’s seven-station catalogue
and native daily text exports. Original UTC interval-end labels remain intact;
documented one-minute or historical three-minute intervals are recorded
explicitly. Instrument temperatures, radiation units and quality flags retain
their source meaning. Catalogue coordinates and source-file header coordinates
are separate evidence. SURFRAD’s participation in BSRN is identified in the
provenance; combining the feeds requires an explicit overlap decision.
All five clients use the shared bounded HTTP transport and existing dependencies. Only AEMET needs a credential for acquisition. Original source artifacts or explicitly canonical JSON snapshots and their hashes are saved alongside the normalized outputs; completed local snapshots verify before reuse. Native station observations preserve their own scientific contract. Mapping stations to plants, applying quality masks, filling gaps and producing the hourly fleet weather schema belong in an explicit subsequent analysis, retaining each station dataset’s attribution alongside the thesis.
CMIP6 native data contract#
energy_data_retrieval.sources.cmip6 accesses the current public ESGF STAC API.
It uses the existing HTTP transport and introduces no additional dependencies.
Discovery selects one model, experiment, member, CMOR table, variable and grid,
then saves a bounded retrieval plan with resolved versions and the original
catalogue records. East and west indices are selected explicitly; there is no
automatic fallback or substitution of model, member or grid.
File transfer consumes the saved plan, streams native NetCDF files under explicit size limits, and checks their published byte size and SHA-256 multihash. Local folders are lowercase and derived from dataset hashes; manifests retain the mapping to official identifiers, original filenames and source/citation URLs. Verified completed downloads can be reused; failed transfers restart from zero and conflicting outputs are not silently overwritten. See the provider commands and limits.
CMIP6 retains its own scientific data contract: native calendars, time bounds, cell methods, coordinates, units and model/experiment identity. Year filters select overlapping whole files; they do not crop data or certify temporal completeness. Spatial subsetting, regridding, bias adjustment and ensemble statistics belong in an explicit subsequent analysis. Climate projections cannot be inserted into the normalized hourly weather contract without that work. The CMIP6 design and data-citation guidance are recorded in the shared reference catalogue; each acquisition also retains the actual dataset citation and license metadata.
Scientific boundaries#
The reusable implementation comes from the preceding ASVA Chile analysis. It does not claim to be a byte-identical replacement for every historical script or fitted model. The saved dashboard release and its production data remain separate records. New fits record their own configuration and evaluation.
Geometry, timing, acceptance-band bounds and model parameters must be fitted on training observations. Held-out observations remain available for honest evaluation, including low or zero production; they must not be filtered using their own target values to improve the score.
The constrained physical response and LightGBM are distinct algorithms. A monotonic LightGBM model does not automatically guarantee a concave response. Neither algorithm guarantees a 75% prediction score for every plant.
An hourly operational screen describes evidence. It does not confirm the cause of curtailment, outage, environmental restriction or equipment failure. Public energy and modeled weather alone cannot establish continuous hourly availability.
Adding another country#
Add or reuse a provider folder in
data-retrieval. Translate its units, timestamp convention and identity fields into the documented schema.Add a country package under
energy_analysis, with the regional clock, metadata mapping and study configuration.Reuse
energy_methods; put country-specific assumptions in the country workflow, not in the shared algorithms. Put report generation in that country’sreports/subpackage and generated artifacts under the corresponding run’sreports/directory.Add fixtures for provider irregularities, calendar transitions, coverage and chronological evaluation. Keep live API calls out of ordinary CI.
Reports and research attribution#
Chile rendering lives in energy_analysis.chile_fleet.reports. It consumes the
completed run or frozen-reference comparison contract. The report manifest
records the inputs and output hashes separately from the original analysis;
rendering does not fit a model or change predictions. Individual site pages keep
the fleet landing page small, and publication drafts use the same qualified
totals shown in the dashboard.
Julián Andrés Ascencio Vásquez’s 2021 PhD thesis inspired the project and its
early studies. Later components extend beyond that work and document their own
scientific sources. CITATION.cff, references.bib and docs/references.md expose citation
metadata for readers and tools. The energy_methods.references catalogue is
also available to analysis reports. Method-specific citations credit the
papers actually used and distinguish background context from implemented work.