Continuous updates (experimental feature)

[jeroenterheerdt]

Let's discuss the continuous updates feature here!

[jeroenterheerdt]

#363 and #436 let's move discussion about how to handle various sensors / aggregates to this conversation please!

[jeroenterheerdt]

Two things come to mind:

1. How about precipitation? Shouldn't that be the actual rain rate and not the last/max of daily rain rate when continuous updates are on?
2. Should we be using reimann integral for solrad sensor? If so, how?

[jata1]

@jeroenterheerdt - thanks for introducing this feature. I have it switched on on prod and test environments so happy to help investigate and test.

solar radiation is tricky and I am not getting bucket calculations like I would expect but it is winter here in Oz so that is a factor.

I think there is something not right with how I setup my reimann integral sensor - see below.

raw data from the solar rad sensor on my weather station:

daily solar rad using a reimann integral (curve shape is good but daily total is out maybe by a factor of 100 or 1000):

[jata1]

So as a next step, I will create a new riemann sensor x100 and one x1000 to see if this is closer.

For reference - before I used a riemann I had a senor that simply added up the sol rad sensor and it looks like this - but even this was underestimating watering so this approach did not work either (for me).

[jata1]

Tried changing the riemann to kilo but it only changes the unit so not helpful (for me)

[jata1]

I think I might have found my issue with solar rad being under reported. I found this on the interweb...

The daily irradiation in Wh/m2 will be obtained as the sum of all hourly values in W/m2. For instance, if the irradiance is constant at 100 W/m2 during 10 hours, the daily total irradiation is 1000 Wh/m2=1 kWh/m2.

The same principle applies to 10-min values, but then the total needs to be divided by 6. So if the irradiance is constant at 100 W/m2 during 10 hours or 60 10-min periods, the daily irradiation is 100x60/6=1 kWh/m2 again, as expected.

So I think you need to setup the riemann integral as an hourly calculation (rather than daily) - this means that my original setup was under reporting by a factor of 24. So

I will see what happens now

[pumabaer]

Hi,
Can you share your configuration for your Riemann sum integral sensor?
Did you solve it using a helper or in the configuration.yaml?

[posixx]

Just to understand continous updates; if using sensors only including ET, continuous updates are not possible? What is the advantage of continuous updates?

[jeroenterheerdt]

If you use sensors only you should be able to use continuous updates. If you rely on a weather service we block you right now. Continuous updates update as soon as your weatherstation gives new values so should be highly accurate to your weather as it's up to the minute or second updates depending on how quickly your weatherstation operates..

[posixx]

Ok so this also works using passthrough module with ET sensor available?

[jeroenterheerdt]

it should.

[posixx]

Nice. It would be nice you can fix #430 (comment) so my log isn’t flooded between calculation time and 1st ET sensor update

[pumabaer]

Hello,

I have enabled continuous calculation.
Now, when it calculates in the evening at 11:50 PM, the time increases again. However, the time then looks much higher than during the continuous calculation.
Example:
Over the day, the bucket empties to -0.5. In the evening, it then jumps to -2.5.

Does the evening calculation need to be turned off?

[jeroenterheerdt]

It should not be necessary to turn it off but there is a bug right now that seems to cause this

[pumabaer]

Ah okay.
Let me know if you need any more information.

[jeroenterheerdt]

I think I solved the bug in the latest update (v2024.7.10) so you should be able to keep both continuous updates and automatic updates on.

[pumabaer]

[jeroenterheerdt]

please open an issue for this.

[pumabaer]

Hi,
How should the "solar radiation" sensor be configured?
Is "average" the right option?

[jata1]

Hi, How should the "solar radiation" sensor be configured? Is "average" the right option?

I have made good progress on calculated daily solar radiation. Here is my setup but you will need to adjust for whatever your raw/source solar rad sensor is outputting.

I am using 2 helpers to get this working. One calculates the riemann integral for solar radiation hourly as my sensor in in watts (watts hrs). The other is a utility meter that resets the the riemann sum data every day so you have a good daily calculation of total solar radiation.

In smart irrigation setup you use the utility meter and set average as the calculation method.

sol rad sensor output units is W/M2:

Riemann helper:

Utility meter:

Here is what the solar output daily looks like from my system based on the above. Just plug this sensor into smart irrigation.

[jeroenterheerdt]

@jata1 how easy was this to set up? should we even try to implement this all in the integration itself?

[pumabaer]

Thanks for sharing!
I will test it as well.
If it works as expected, it would be good to incorporate it directly into the integration.

[jata1]

@jata1 how easy was this to set up? should we even try to implement this all in the integration itself?

The root issue in my case is related to the weather station integration/data as it provides real time solar radiation in W/M2. This data needs to be manipulated to be used in the ETO calcs in smart irrigation.

Now I know what I am doing it is easy but took a bit of working through (as you know).

I will contact the guys that maintain the integration for my weather station to see if I can get them to add an additional sensor for daily accumulated solar radiation. solar radiation needs to be handled the same as rain/precipitation.

[pumabaer]

Hi, How should the "solar radiation" sensor be configured? Is "average" the right option?

I have made good progress on calculated daily solar radiation. Here is my setup but you will need to adjust for whatever your raw/source solar rad sensor is outputting.

I am using 2 helpers to get this working. One calculates the riemann integral for solar radiation hourly as my sensor in in watts (watts hrs). The other is a utility meter that resets the the riemann sum data every day so you have a good daily calculation of total solar radiation.

In smart irrigation setup you use the utility meter and set average as the calculation method.

sol rad sensor output units is W/M2:

Riemann helper:

Utility meter:

Here is what the solar output daily looks like from my system based on the above. Just plug this sensor into smart irrigation.

Are you sure that "average" is the right setting for solar radiation?

If I have a counter, the average value is always below the actual (last counted) measured value, right?
Value 1 + value 2, etc. / 2

Just a thought, I might be wrong about this.

[jata1]

@jeroenterheerdt - the good news with all of this work is that I finally have smart irrigation working properly as I am handling solar radiation correctly so the daily ETO calcs are working. The sensor for total daily solar radiation looks correct when comparing to the solar radiation raw data sensor.

[jeroenterheerdt]

Love it. Would you be willing to write up a step by step process of what you did to configure your sensor group and input sensors for continuous updates? I am planning to rewrite the readme (move to wiki) and I think it would be good to have what you learned in there!

[jata1]

Happy to do a write up but not quite there yet.

My testing over the last week has not been using continuous updates as I wanted to test one thing at a time. I now have daily calcs working correctly with my solar radiation sensors. Now I am switching to continuous and I will see what I find and update this thread.

[jata1]

After switching to continuous updates, I have some new errors in the log and a couple of observations.

My setup has two zones (front and back) each with its own sensor group. The sensor groups are identical except one has precipitation set as a static value of 0 (as this zone is under cover). However, the amount of data collected for the two zones are very different - see image below from the last hour since I switched to continuous.

Diagnostic file attached...
config_entry-smart_irrigation-01J3NXJFTGQR2QDP3BMYX2SF83.json

[jeroenterheerdt]

Wait, that's weird! The sensor groups are the same except for the one static? I am wondering if somehow the static zone gets skipped?

[jata1]

So I have been playing with continuous updates and there is something strange going on. I have a feeling it is related to having two sensor groups that use the mostly the same sensors.

I had it working but only one zone was actually updating continuously. So I deleted the sensor group that I assumed was not working and set it up new.

Now this one works but the other zone is not updating. It's very confusing.

My next test is to use the same sensor group on both zones. I will report back

[jeroenterheerdt]

Is there anything in the logs that can help?

[jata1]

Nothing obvious in the logs. I don't understand why there is much more data being collected for one sensor group compared to the other. I think this is a clue to an underlying issue.

Right now I am running a different test. I have set both zones to use the same (newly created) sensor group so I will see what happens with this and report back.

[jata1]

Just an update from my experience over the last few days...

I can't get good results with continuous enabled. Not sure how to investigate further but happy to do so if helpful.

For now, I have switched to a daily calcs and this seems to be woking nicely now that I have fixed solar radiation by using the riemann sum helper/sensor setup.

[ChrisHomeAssistant]

Yes, definitely. Also transforming the daily "Wh" value to MJ/d makes sense. E.g. taking a value of 2000Wh you showed equals (*3.6/1000) 7.2MJ per day which is a plausible value.

For the ET calculation I would assume the input should be the last value in the sensor. Or does the formula in the calculation expect a different input?

[jata1]

Yes. I am using last or max in the sensor group config. I think either is fine.

[ChrisHomeAssistant]

I'm not completely sure about this. The input is converted with a factor of 0.0864, which also accounts for a "hour to day" conversion (0.0036*24).
Nevertheless, if it is already an aggregated figure using the Riemann sum it should only be converted by 0.0036 (from Wh to MJ), right?

[jata1]

I'm not sure to be honest. You might be right. If correct then this could be a bug.

At the moment, my calculated watering is about right and I spent a long time getting sol rad to work properly. I hope it does not have a calculation error x 24.

Suggest creating a new bug / issue on this topic so we can bottom it out with @jeroenterheerdt

[ChrisHomeAssistant]

Maybe it is related to the bug #461 (comment)

and the general question how the input is expected.

The calculation of the solrad itself looks absolutely fine for me.

[jata1]

see below. I had a powercut for 2hrs and you can see that riemann is flat during this time (as there is no data).

left chart is my source solar radiation sensor. right is riemann sum of the data...