Forecasts not responding to changed dampening factors

[Nilogax]

In previous versions any change to dampening factors was very quickly reflected in all solcast forecast data, but latest versions don't seem to behave in the same way.

To Reproduce
Steps to reproduce the behavior:

1. Note current solcast PV forecast totals for today, day 2 etc and current dampening settings
2. In Developer Tools call solcast_solar.set_dampening with 24 zeroes, or reset all dampening factors to zero via the integration UI
3. Expected behaviour is that all forecast data should be set to zero, but for me they remain unchanged (I have also try varying a single hourly dampening factor rather than the full zero option, but again there's no effect)
4. Call set dampening or use the UI again to restore prior dampening settings

Logs don't seem to help much as it just records the service call with no extra info.

Noticed in 4.0.31 and 4.0.30, tried 4.0.27 which seemed to behave as expected

[Nilogax]

Had a bit of time to explore this today and think I may have identified the source of the issue, but I'm not sure if my simple fix is valid.

In the update from 4.0.27 I found this commit:

181ea1c

Specifically I edited my local copy of init.py and undid the change at 292:293/291:300 so I'm assuming the code to detect if an item has been changed wasn't detecting changes to the dampening settings. Not sure if this the actual cause or how to correct the new code, but for now it seems the forecasts (and calls to query_forecast_data) are responding to dampening changes again.

Could someone who actually knows what they're doing take a look please?

[autoSteve]

That was part of the change made to incorporate the "lost" Oziee v4.0.23 changes. I'm beginning to wish they had remained lost.

Maybe Oziee didn't think through the broader impact.

I already have one merge and a draft release prepared covering another of these lost v4.0.23 changes that broke long term statistics for forecast history, which will likely be released unless a reason can be found not to. (Have you lost forecast LTS, @Nilogax?)

So I throw the floor open to opinion.

Should this modification to only reload on config change also be reverted? At the very least the potantial for impact needs to be further investigated by someone.

[autoSteve]

Converted to a discussion.

Anybody willing to take a look at this?

[autoSteve]

Okay. Might I throw a possible change out there? Essentially remove DAMP_FACTOR from the equation. The others make sense in this circumstance, given they are likely only ever updated in a manual config change. I cannot imagine that dampening factor ever would get updated manually given how the earth tilts, making fixed values meaningless.

Can you try this as a replacement, @Nilogax?

In __init__.py

async def async_update_options(hass: HomeAssistant, entry: ConfigEntry):
    """Handle options update. Only reload if any item was changed"""
    if any(
        entry.data.get(attrib) != entry.options.get(attrib)
        for attrib in (HARD_LIMIT, KEY_ESTIMATE, CUSTOM_HOUR_SENSOR, CONF_API_KEY)
    ):
        # update entry replacing data with new options
        hass.config_entries.async_update_entry(
            entry, data={**entry.data, **entry.options}
        )
        await hass.config_entries.async_reload(entry.entry_id)

[autoSteve]

No. That won't work. By adjusting dampening in the way you are it obviously hits this __init__ routine, so specifically excluding it will just make things worse.

[autoSteve]

I think going back to the pre-Oziee 4.0.23 way may make the most sense. He was possibly trying to improve efficiency, I guess, for something that almost never gets changed: The config. So @Nilogax, I think you have proposed the fix. Revert.

[Nilogax]

Thanks for taking a look @autoSteve. From a purists perspective I can appreciate the intention of the new check but given the likely infrequenct changes (probably by a minority of users) there seems to be little practical benefit.

[autoSteve]

Kind of agree. Purists would want their config changes to stick, wherever they came from, not just the UI. And your changes to dampening are clearly hitting this change, or the reversion would not have been required.

This code should check for differences in config, which you are making. It can't be determining that correctly, hence my call for others to take a look. I've looked at it and can't see an obvious problem, but if reverting does the trick then this code has issue, and probably should be rolled back for now.

[autoSteve]

I'm eager to know, @Nilogax. Have you lost long-term history for forecast by updating? Have a look at the monthly view in the energy dashboard.

[Nilogax]

@autoSteve Seems my forecast history matches my HA purge setting rather than showing any long term statistics.

[autoSteve]

Great to know. The LTS for forecast is kind of useless, but kind of an eye tick. No one wants to know what the high temperature was for Dec 3 2021, because we already know what it was, just like solar forecasting... Yet folks like an historical dotted line just because.

I even miss it, but then give myself a good hard talking to.

[autoSteve]

What say you, @Nilogax? Has reverting Oziee's change to the config reload done the trick for dampening? I'm fixing to roll that back in a commit.

[autoSteve]

It. Is. Done.

This will be coming to a future release. The upcoming 4.0.32, which is already in pre-release will not incorporate it, so I would encourage you to manually edit the change again if applying that once it goes GA.

This is probably targetted for 4.0.34, which should come after a rather tasty 4.0.33 that addresses some serious performance issues (again, manually edit __init__.py if applying that one). 4.0.33 will be pre-release for a bit, given the extent of optimisation. But I love those optimisations, and am running that code today.

[Nilogax]

Thanks - I'm already running the 0.32 beta so I'll shout if I spot anything unusual after tweaking the next release or two.

[autoSteve]

Excellent. Do shout out both positive and negative experiences in the beta discussion. We're a small, but growing community at BJReplay since Oziee deserted, and there are far too few data points at this stage. So do add your voice.

[autoSteve]

Revised to suggest target 4.0.33 for this, in conjunction with the performance improvements, which will go beta for a time. PR #55 set to review.