Solar and wind forecasting: four models, one common test#

Compare LightGBM, XGBoost, Mistral Timestral and physical surrogates with a previous-day baseline on real ONS plant observations. Each plant has 16 daily 24-hour forecasts after an initial 14-day training period. The default run fits the tree/physical models locally and replays checksum-verified real Timestral inference; it requires neither network access nor PyTorch.

Research inspiration: Ascencio-Vásquez, J., Bevc, J., Reba, K., Brecl, K., Jankovec, M. and Topič, M. (2020), Advanced PV Performance Modelling Based on Different Levels of Irradiance Data Accuracy, Energies 13(9), 2166. doi:10.3390/en13092166. Its attention to irradiance quality and empirical/ML comparisons motivates this example. This is a forecasting extension to different data, models and wind generation, not a reproduction of the paper’s experiment, accuracy tiers or numerical results.

Download the runnable example and data, scores and input provenance. See the benchmark guide for model equations, reproduction commands and methodological differences from the paper.

Scope: one solar plant and one wind farm in June 2024. Future weather is constructed by persistence, not archived NWP. Historical ONS data include post-operation revisions. Installed equipment and irradiance-plane metadata are incomplete; the physical models are explicit approximations. No universal model ranking or operational forecast claim follows from this small study.

1. Run the same chronological experiment for every model#

A forecast sees only the preceding 672 half-hour observations. Tree models share resource, clock sine/cosine and previous-day power features. Timestral receives the same covariates plus the full generation context. Physical surrogates use the resource forecast. All are evaluated on the same 768 future labels per plant, including nights and restricted-generation intervals.

import json
import os
from pathlib import Path

import matplotlib.dates as mdates
import matplotlib.pyplot as plt

from energy_analysis.forecast_benchmark import (
    MODELS,
    TIMESTRAL_MANIFEST,
    benchmark_scores,
    run_forecast_benchmark,
)
from energy_methods.timestral import TimestralConnector

directory = (
    Path(globals().get("__file__", Path.cwd() / "plot_forecast_model_benchmark.py"))
    .resolve()
    .parent
    / "data/forecast"
)
manifest = json.loads((directory / TIMESTRAL_MANIFEST).read_text())
connector = None
if os.environ.get("ENERGY_FORECAST_TIMESTRAL_LIVE") == "1":
    connector = TimestralConnector(
        manifest["model_id"],
        revision=manifest["model_revision"],
        device=os.environ.get("ENERGY_TIMESTRAL_DEVICE", "cpu"),
        local_files_only=os.environ.get("ENERGY_TIMESTRAL_LOCAL_ONLY") == "1",
    )
print("Timestral mode:", "live inference" if connector else "saved real inference, offline replay")
print(f"Checkpoint: {manifest['model_id']} @ {manifest['model_revision']}")
predictions = run_forecast_benchmark(directory, connector=connector)
scores = benchmark_scores(predictions)
lead_scores = benchmark_scores(predictions, by_lead=True)
print(scores.round(3).to_string(index=False))
if output := os.environ.get("ENERGY_FORECAST_OUTPUT"):
    destination = Path(output)
    destination.mkdir(parents=True, exist_ok=True)
    predictions.to_csv(destination / "predictions.csv", index=False)
    scores.to_csv(destination / "scores.csv", index=False)
    lead_scores.to_csv(destination / "lead_scores.csv", index=False)
Timestral mode: saved real inference, offline replay
Checkpoint: mistralai/Timestral-5M @ bdd20cfc15a3fea0e4cb5ccef37f8d3ca7aa335f
technology        model  samples  origins  mae_mw  rmse_mw  bias_mw  wape_pct  nmae_training_q95_pct  p10_p90_coverage  mae_skill_vs_previous_day
     solar     LightGBM      768       16   2.200    4.593   -0.010    38.131                 10.497               NaN                      0.063
     solar     Physical      768       16   2.133    4.190   -0.083    36.965                 10.163               NaN                      0.091
     solar Previous day      768       16   2.348    5.128    0.043    40.687                 11.216               NaN                      0.000
     solar    Timestral      768       16   2.152    4.271   -0.002    37.296                 10.261             0.637                      0.083
     solar      XGBoost      768       16   2.228    4.614    0.013    38.608                 10.629               NaN                      0.051
      wind     LightGBM      768       16   4.998    6.673   -0.019    71.348                 29.850               NaN                      0.049
      wind     Physical      768       16   4.683    6.120   -1.469    66.850                 27.975               NaN                      0.109
      wind Previous day      768       16   5.254    7.051    0.121    74.997                 31.233               NaN                      0.000
      wind    Timestral      768       16   5.663    7.413    1.030    80.836                 33.848             0.260                     -0.078
      wind      XGBoost      768       16   5.012    6.730    0.036    71.548                 29.941               NaN                      0.046

2. Compare errors without excluding difficult test observations#

MAE is in mean MW over a half-hour. RMSE, bias, WAPE, skill against the daily baseline and training-Q95-normalized MAE are printed above. The normalization uses historical power, not nameplate capacity, so it differs from the paper’s capacity-normalized RMSE. We do not apply its acceptance-band filtering or remove test outliers. Historical ONS resource flags mask model inputs only.

colors = dict(zip(MODELS, ("#c27619", "#287758", "#9a4d8b", "#2976b5", "#8b929b"), strict=True))
labels = {"solar": "Coremas IV · solar", "wind": "Delta 3 I · wind"}
fig, axes = plt.subplots(1, 2, figsize=(11, 4.4), layout="constrained")
for ax, technology in zip(axes, ("solar", "wind"), strict=True):
    table = scores[scores["technology"] == technology].set_index("model").loc[list(MODELS)]
    bars = ax.barh(table.index, table["mae_mw"], color=[colors[m] for m in table.index])
    ax.bar_label(bars, fmt="%.2f", padding=4)
    ax.invert_yaxis()
    ax.set(
        title=labels[technology],
        xlabel="Mean absolute error (MW)",
        xlim=(0, table["mae_mw"].max() * 1.2),
    )
    ax.grid(axis="x", alpha=0.2)
fig.suptitle("24-hour forecasts · 16 daily origins · lower error is better")
plt.show()
24-hour forecasts · 16 daily origins · lower error is better, Coremas IV · solar, Delta 3 I · wind

3. Inspect the final three days, chosen by date#

This fixed date slice is not selected for favorable errors. Daily model fits are concatenated here; forecasts are never updated within their 24-hour window.

fig, axes = plt.subplots(2, 1, figsize=(12, 7), layout="constrained", sharex=True)
for ax, technology in zip(axes, ("solar", "wind"), strict=True):
    case = predictions[predictions["technology"] == technology]
    last_issues = sorted(case["issue_time"].unique())[-3:]
    case = case[case["issue_time"].isin(last_issues)]
    actual = case[case["model"] == "Previous day"].sort_values("timestamp")
    ax.plot(
        actual["timestamp"], actual["actual_mw"], color="#202b35", linewidth=2, label="Measured"
    )
    for model in MODELS:
        rows = case[case["model"] == model].sort_values("timestamp")
        ax.plot(
            rows["timestamp"],
            rows["prediction_mw"],
            color=colors[model],
            label=model,
            linewidth=1.2,
            linestyle="--" if model == "Previous day" else "-",
        )
    for issue in last_issues:
        ax.axvline(issue, color="#abb2b9", alpha=0.5, linestyle=":")
    ax.set(title=labels[technology], ylabel="Mean power (MW)")
    ax.grid(alpha=0.2)
axes[0].legend(ncols=3, fontsize=9)
axes[-1].xaxis.set_major_formatter(mdates.DateFormatter("%d %b\n%H:%M"))
axes[-1].set_xlabel("Source reference label (UTC); dotted lines mark forecast issues")
fig.suptitle("Observed and forecast generation · final three holdout days")
plt.show()
Observed and forecast generation · final three holdout days, Coremas IV · solar, Delta 3 I · wind

4. Inspect error by forecast lead#

Each lead has 16 verification points per plant. Solar lead and clock time are coupled because every forecast is issued at the same daily clock time. These curves therefore reflect both horizon length and the daily resource cycle. Timestral P10-P90 coverage in the printed table is descriptive, not calibrated uncertainty. Pretraining overlap with this historical period is unknown.

fig, axes = plt.subplots(1, 2, figsize=(11, 4), layout="constrained")
for ax, technology in zip(axes, ("solar", "wind"), strict=True):
    for model in MODELS:
        rows = lead_scores[
            (lead_scores["technology"] == technology) & (lead_scores["model"] == model)
        ]
        ax.plot(rows["lead_hours"], rows["mae_mw"], color=colors[model], label=model)
    ax.set(title=labels[technology], xlabel="Hours after issue", ylabel="MAE (MW)", xlim=(0.5, 24))
    ax.grid(alpha=0.2)
axes[0].legend(fontsize=8)
fig.suptitle("Error by lead · common observations and scoring rules")
plt.show()
Error by lead · common observations and scoring rules, Coremas IV · solar, Delta 3 I · wind

Total running time of the script: (0 minutes 2.291 seconds)

Gallery generated by Sphinx-Gallery