Incorrect values with cost sensors and IOG

[ragg987]

Describe the bug

When the IOG import rate transitions from low to high rate, the accumulative cost sensor incorrectly attributes a portion of the low-rate consumption to a high rate. In effect the cost sensor shows an inflated value caused by the transition. It only seems to happen when the rate transitions from low to high - I don't see a similar effect when it goes from high to low. I have noticed this behaviour for a few months and a few version of the integration. When I was on Octopus Agile, the transition used to work correctly.

Reproduction steps

Allow the integration / sensor to run through the rate transition. Approx 10-20 mins after the cost change the accumulative cost sensor will update and show a jump in values. I attach a screenshot of the sensor history, you can see there were 2 cost jumps at 10:18 and 11:20. These correspond with rate changes from low to peak at 10:00 and 11:00.

Expected behaviour

Accum cost sensor to reflect the correct rate.

Tariff Code

22l4295702_1012455614354

Integration Version

13.5.4

Home Assistant Version

HAOS 2025.1.3

Fresh Install?

Not specified

Home Assistant Logs

I have removed personal info from the log - hopefully it gives you enough to trace this.

debug.zip

Confirmation

• I confirm that I cannot find my solution within the documentation
• I confirm that I cannot find my solution within the FAQ

[ragg987]

Sorry should have mentioned the sensor is
sensor.octopus_energy_electricity_XXX_YYY_current_accumulative_cost

[BottlecapDave]

Hello and sorry you've encountered this issue. This can sometimes happen when a pending dispatch doesn't turn into a completed dispatch from OE. I use pending and completed dispatches to determine if the price should be peak or off peak as the OE rate APIs do not account for this. What the values of the completed dispatches in your dispatching sensor? I assume this sensor also turned off around the time of the jump?

[ragg987]

Hi, the dispatch history for these 3 events all show they happened at 29s past the planned event time, which seems to be consistent with the way HA records these events i.e. within the minute tick. This problem is not intermittent - it is with every event where it switches from low rate to high.

the attributes of the sensor seem to be missing these events time under completed_dispatches, which is odd. Snapshot taken 28/01/2025 @ 18:00.
``
completed_dispatches:`

• start: '2025-01-26T00:30:00+00:00'
  end: '2025-01-26T01:00:00+00:00'
  charge_in_kwh: -1.12
  source: null
  location: AT_HOME
• start: '2025-01-26T00:00:00+00:00'
  end: '2025-01-26T00:30:00+00:00'
  charge_in_kwh: -1.18
  source: null
  location: AT_HOME
• start: '2025-01-26T01:30:00+00:00'
  end: '2025-01-26T02:00:00+00:00'
  charge_in_kwh: -1.14
  source: null
  location: AT_HOME
• start: '2025-01-26T01:00:00+00:00'
  end: '2025-01-26T01:30:00+00:00'
  charge_in_kwh: -1.14
  source: null
  location: AT_HOME
• start: '2025-01-26T02:00:00+00:00'
  end: '2025-01-26T02:30:00+00:00'
  charge_in_kwh: -0.44
  source: null
  location: AT_HOME
• start: '2025-01-26T23:00:00+00:00'
  end: '2025-01-26T23:30:00+00:00'
  charge_in_kwh: -1.78
  source: null
  location: AT_HOME
• start: '2025-01-26T22:30:00+00:00'
  end: '2025-01-26T23:00:00+00:00'
  charge_in_kwh: -0.57
  source: null
  location: AT_HOME
• start: '2025-01-27T00:00:00+00:00'
  end: '2025-01-27T00:30:00+00:00'
  charge_in_kwh: -1.77
  source: null
  location: AT_HOME
• start: '2025-01-26T23:30:00+00:00'
  end: '2025-01-27T00:00:00+00:00'
  charge_in_kwh: -1.86
  source: null
  location: AT_HOME
• start: '2025-01-27T01:30:00+00:00'
  end: '2025-01-27T02:00:00+00:00'
  charge_in_kwh: -1.8
  source: null
  location: AT_HOME
• start: '2025-01-27T01:00:00+00:00'
  end: '2025-01-27T01:30:00+00:00'
  charge_in_kwh: -1.75
  source: null
  location: AT_HOME
• start: '2025-01-27T00:30:00+00:00'
  end: '2025-01-27T01:00:00+00:00'
  charge_in_kwh: -1.81
  source: null
  location: AT_HOME
• start: '2025-01-27T02:30:00+00:00'
  end: '2025-01-27T03:00:00+00:00'
  charge_in_kwh: -1.74
  source: null
  location: AT_HOME
• start: '2025-01-27T02:00:00+00:00'
  end: '2025-01-27T02:30:00+00:00'
  charge_in_kwh: -1.84
  source: null
  location: AT_HOME
• start: '2025-01-27T03:30:00+00:00'
  end: '2025-01-27T04:00:00+00:00'
  charge_in_kwh: -1.79
  source: null
  location: AT_HOME
• start: '2025-01-27T03:00:00+00:00'
  end: '2025-01-27T03:30:00+00:00'
  charge_in_kwh: -1.75
  source: null
  location: AT_HOME
• start: '2025-01-27T04:30:00+00:00'
  end: '2025-01-27T05:00:00+00:00'
  charge_in_kwh: -1.78
  source: null
  location: AT_HOME
• start: '2025-01-27T04:00:00+00:00'
  end: '2025-01-27T04:30:00+00:00'
  charge_in_kwh: -1.73
  source: null
  location: AT_HOME
• start: '2025-01-27T05:30:00+00:00'
  end: '2025-01-27T06:00:00+00:00'
  charge_in_kwh: -1.23
  source: null
  location: AT_HOME
• start: '2025-01-27T05:00:00+00:00'
  end: '2025-01-27T05:30:00+00:00'
  charge_in_kwh: -1.75
  source: null
  location: AT_HOME
• start: '2025-01-27T06:30:00+00:00'
  end: '2025-01-27T07:00:00+00:00'
  charge_in_kwh: -1.28
  source: null
  location: AT_HOME
• start: '2025-01-27T07:30:00+00:00'
  end: '2025-01-27T08:00:00+00:00'
  charge_in_kwh: -1.8
  source: null
  location: AT_HOME
• start: '2025-01-27T07:00:00+00:00'
  end: '2025-01-27T07:30:00+00:00'
  charge_in_kwh: -1.79
  source: null
  location: AT_HOME
• start: '2025-01-27T08:30:00+00:00'
  end: '2025-01-27T09:00:00+00:00'
  charge_in_kwh: -1.77
  source: null
  location: AT_HOME
• start: '2025-01-27T08:00:00+00:00'
  end: '2025-01-27T08:30:00+00:00'
  charge_in_kwh: -1.75
  source: null
  location: AT_HOME
• start: '2025-01-27T09:00:00+00:00'
  end: '2025-01-27T09:30:00+00:00'
  charge_in_kwh: -0.22
  source: null
  location: AT_HOME
• start: '2025-01-27T15:30:00+00:00'
  end: '2025-01-27T16:00:00+00:00'
  charge_in_kwh: -3.25
  source: null
  location: AT_HOME
• start: '2025-01-27T16:30:00+00:00'
  end: '2025-01-27T17:00:00+00:00'
  charge_in_kwh: -3.31
  source: null
  location: AT_HOME
• start: '2025-01-27T16:00:00+00:00'
  end: '2025-01-27T16:30:00+00:00'
  charge_in_kwh: -3.35
  source: null
  location: AT_HOME
• start: '2025-01-27T17:30:00+00:00'
  end: '2025-01-27T18:00:00+00:00'
  charge_in_kwh: -3.27
  source: null
  location: AT_HOME
• start: '2025-01-27T17:00:00+00:00'
  end: '2025-01-27T17:30:00+00:00'
  charge_in_kwh: -3.32
  source: null
  location: AT_HOME
• start: '2025-01-27T18:30:00+00:00'
  end: '2025-01-27T19:00:00+00:00'
  charge_in_kwh: -3.35
  source: null
  location: AT_HOME
• start: '2025-01-27T18:00:00+00:00'
  end: '2025-01-27T18:30:00+00:00'
  charge_in_kwh: -3.27
  source: null
  location: AT_HOME
• start: '2025-01-27T23:00:00+00:00'
  end: '2025-01-27T23:30:00+00:00'
  charge_in_kwh: -3.31
  source: null
  location: AT_HOME
• start: '2025-01-28T00:00:00+00:00'
  end: '2025-01-28T00:30:00+00:00'
  charge_in_kwh: -3.29
  source: null
  location: AT_HOME
• start: '2025-01-27T23:30:00+00:00'
  end: '2025-01-28T00:00:00+00:00'
  charge_in_kwh: -3.47
  source: null
  location: AT_HOME
• start: '2025-01-28T01:00:00+00:00'
  end: '2025-01-28T01:30:00+00:00'
  charge_in_kwh: -3.44
  source: null
  location: AT_HOME
• start: '2025-01-28T00:30:00+00:00'
  end: '2025-01-28T01:00:00+00:00'
  charge_in_kwh: -3.35
  source: null
  location: AT_HOME
• start: '2025-01-28T02:00:00+00:00'
  end: '2025-01-28T02:30:00+00:00'
  charge_in_kwh: -2.97
  source: null
  location: AT_HOME
• start: '2025-01-28T01:30:00+00:00'
  end: '2025-01-28T02:00:00+00:00'
  charge_in_kwh: -3.38
  source: null
  location: AT_HOME
  ``

[BottlecapDave]

If they're missing from the completed dispatches collection, then this will be the reason why. This implies it was a pending dispatch, but was not transitioned to a completed dispatch. This information is taken from the API and saved locally for longer persistence (the API only keeps completed dispatches for a maximum of 24 hours). Pending dispatches don't guarantee charging will occur, but is the only information we have to go on from the api. I assume your charger also confirms that charging occurred at these pending times?

[ragg987]

OK so if I understand that then the API did not send a completed dispatch message? Would that account for the 10-20 minute delay before the cost is rolled up, but at a higher rate? I must admit I don't follow, however I am not particulary knowledgable at this level so take this with a pinch of salt.

I do not directly rely on the dispatching sensor for my automations in HA - I use target sensors which I believe rely on the published rates only. Hence my automations seem to be recognising the rate transition and react accordingly. I am aware of the health-warning you have placed around this, but so far it appears to work reliably.

[BottlecapDave]

Yeah the API did not report a completed dispatch for that time slot. Rate information is reviewed every time either the rate of dispatches are updated. It might have taken 10-20 minutes for the pending dispatch to be removed from the list, as I use either pending or completed dispatches to to alter the rates. I'm not on intelligent, so I'm not too sure of the frequency/process.

There will be history of the pending/completed dispatches in the HA database if you're comfortable performing SQL queries? Unfortunately HA doesn't expose this information for some reason.

This sounds like a similar issue to #1094

[cyberkryten]

Yep, looks identical to that other issue, anything that recalculates based on previous dispatches is prone to these cost jumps. Octopus spec is to keep dispatches for 12 hours (not 24 as quoted) so you cannot rely on it for 'today' at all.

The 'fix' would be for Octopus to keep all dispatches for at least 24 hours, but I doubt they'll change it as I assume there's a reason they only keep for 12 hours.

I guess this integration could cache dispatches and keep them for a day but that really goes against it's purpose which is to relay the accurate Octopus integration to HA.

The way to work around it not use previous history to calculate costs and create a sensor that accumulates current cost through the day based on actual, current price.

[ragg987]

The way to work around it not use previous history to calculate costs and create a sensor that accumulates current cost through the day based on actual, current price.

Yes makes sense, creating a helper sensor should be straightforward, however I cannot figure out a way to add the daily standing charge. An automation running at midnight?

@BottlecapDave apologies for raising a duplicate. I had done a search prior but I think I only checked open issues.

[BottlecapDave]

No problem on the duplicate. The integration caches the dispatches for 3 days, but only complete ones. If OE don't move the pending dispatch to a completed dispatch there's not a lot I can do :(. I thought about potentially moving pending dispatches to completed ones in certain scenarios, but trying to work out how to do this without it being captured when it's valid (e.g. when the car isn't charged during the pending dispatch)

[RobinXe]

I've always seen this issue, though in my case it has generally tended to correct itself, so I had always assumed it was the Octopus API belatedly reporting things.

As you can see, at some period after the jump up, it jumps back down again. The only real issue occurs when it's up, but not back down, over midnight, when I aggregate the day's net cost (this sensor - export*0.15) into the monthly cost.

@BottlecapDave is there something, that the API serves, that could do with being fixed/optimised, in order to correct this behaviour? Happy to make suggestions/lend weight over on the OE API forum.

[BottlecapDave]

At the moment, we have to manually adjust the rates based on the intelligent schedule, as the standard endpoint only reports the basic schedule. This is identified by the is_intelligent_adjusted attribute.

The rate changing is optimistic in that it assumes all pending dispatches will turn into off peak rates (this is required to calculate target rate sensor schedules, but others use the rate sensors to power automations). Once a pending dispatch is removed, if it's not replaced with a completed dispatch then the rate will revert to its original value (which is what causes the jumps). I've had various reports of the completed dispatches not being returned sometimes (but the user still being charged at off peak rates).

The intelligent API will also only return completed dispatches for a small amount of time (between 12 and 24 hours), which the integration stores locally for longer to ensure rates are adjusted more long term.

Unfortunately the API doesn't return a state to indicate if the car is charging (only that it's plugged in and had a schedule). It might be good to compare what this state was during these periods to see if there's additional information that can be used.

I heard rumblings that a new version of the rate endpoint might be provided that performs this calculation, but I can't find where I read it.

[giuliodali]

Hi, I'm not sure if what I see is caused by the same logic, but I keep getting random spikes in the Accumulative Cost sensor like this, where it jumped from ~£3.40 to £4.40 instantly (red arrow). I'm on Intelligent octopus and as you can see from the other charts the rate changed much earlier and at the time of the jump, the rate was £0.27, so I'd need to have used ~3.7kwH, yet all the other charts show the consumption was either 0 or 0.01 kWh.

[BottlecapDave]

Because the cost takes into account pending and completed dispatches, the jump could be due to a pending dispatch disappearing and not being replaced with a completed dispatch. The only real way of knowing what's happened is by looking at the attributes of the effected sensors before and after the jump, which unfortunately requires a little bit of SQL as HA doesn't expose the information historically.

The following query will get us the attributes at the various time periods for your cost sensor. This will detail the breakdown of how the cost was calculated, including the associated rates before and after the jump. You'll need to replace sensor.octopus_energy_electricity_xxx_xxx_current_accumulative_cost with your cost sensor

SELECT  "states_meta"."entity_id", "state_attributes"."shared_attrs", DATETIME(last_updated_ts, 'unixepoch') as last_updated
FROM "states"
JOIN "states_meta" ON "states_meta"."metadata_id" = "states"."metadata_id"
JOIN "state_attributes" ON "state_attributes"."attributes_id" = "states"."attributes_id"
WHERE "states_meta"."entity_id" = "sensor.octopus_energy_electricity_xxx_xxx_current_accumulative_cost"
AND DATETIME(last_updated_ts, 'unixepoch') BETWEEN '2025-03-01 17:00:00' AND '2025-03-01 18:00:00'

The following query will get us the attributes at the various time periods for your dispatching sensor. This will detail the pending and completed dispatches before and after the jump. You'll need to replace binary_sensor.octopus_energy_xxx_intelligent_dispatching with your dispatching sensor

SELECT  "states_meta"."entity_id", "state_attributes"."shared_attrs", DATETIME(last_updated_ts, 'unixepoch') as last_updated
FROM "states"
JOIN "states_meta" ON "states_meta"."metadata_id" = "states"."metadata_id"
JOIN "state_attributes" ON "state_attributes"."attributes_id" = "states"."attributes_id"
WHERE "states_meta"."entity_id" = "binary_sensor.octopus_energy_xxx_intelligent_dispatching"
AND DATETIME(last_updated_ts, 'unixepoch') BETWEEN '2025-03-01 15:00:00' AND '2025-03-01 18:00:00'

[giuliodali]

Hi - Thanks for getting back to me on this. Attached are the query outputs requested.

Accumulative Cost.csv
Intelligent Dispatching.csv

[giuliodali]

Happened again today, jumped from £3.61 to £6.13.

Intelligent Dispatching - 2025-03-03.csv
Accumulative.Cost - 2025-03-03.csv

[BottlecapDave]

@giuliodali Thanks for supplying the information. As suspected there are pending dispatches, in the past, which are not being replaced with completed dispatches. This might be due to your car not being plugged in or charging during the pending dispatches and so is legitimately not turning into completed dispatches. In the first batch of files, at 16:00, you had the following pending dispatches

• 2025-03-01T06:58:00+00:00 -> 2025-03-01T12:00:00+00:00
• 2025-03-01T13:00:00+00:00 -> 2025-03-01T13:30:00+00:00
• 2025-03-01T13:30:00+00:00 -> 2025-03-01T16:00:00+00:00

so these would change the rates during these periods to be off peak

However at 17:36, all pending dispatches were removed and 08:30-09:00 was converted into a completed dispatch that would keep the rate at off peak. This would cause the jump. It would be interesting to know what the current state was during the pending dispatch periods to see if there's a state that I can look for to ignore pending dispatch periods. However I have a feeling this jump might just be something we have to live with with the current state of the APIs.

[giuliodali]

Hi, The current state did indeed flip to SMART_CONTROL_NOT_AVAILABLE but the current rate sensor stayed flat (correctly) at the peak rate price.

In fact, the main energy dashboard simply uses the current rate sensor and is 100% correct, I'm not seeing incorrect rates there. Can the accumulative cost sensor depend on that rate sensor in the same way?

Or alternatively, is there a way of pulling the home assistant native costs into an entity?

[BottlecapDave]

The current cost sensor re-evaluates the entire day everytime new consumption and rate information is downloaded. The reason is two fold. Firstly I've seen the consumption data go down as well as up. I believe this is when either the data is reconciled or it's the mini correcting itself. Secondly the rate information can change, as is happening here, because there are legitimate scenarios where a pending dispatch doesn't doesn't turn into to a completed dispatch (e.g. the car isn't plugged in, doesn't charge during the period, the car is away from the designated house or some intelligent providers provide more periods than it's planning to use). For example, in your first set of data you have a pending dispatch between 6:58 and 12, but for an hour you have completed dispatches with a location of away, which from my understanding is not charged at off peak rates when this occurs.

The current rate sensor acts in an optimistic way, where it assumes all pending dispatches will be converted into completed dispatches. The energy dashboard captures the cost at the point the consumption changes and doesn't re-evaluate old data if you use the current rate sensor in current rate entity mode. This is why the value is higher when compared to the current cost sensor.

The current cost sensor could be updated to act more like the energy dashboard, but it'll then be wrong and too low compared to bills in the scenarios mentioned above. This scenario very much feels like a lose lose scenario with the data available by the APIs.

I wish the current state data was more granular so I knew when your car was charging as that could be used to cross compare with the pending dispatches.

When the cost sensor went up in your picture, it's not because the current rate has changed but because a rate in the past was previously treated at off peak, but is now peak due to lack of pending/completed dispatches during that period.

[cyberkryten]

Many of the ad-hoc "Smart Charge" dispatches are not shown as completed dispatches for the entire day and are removed quickly from the completed dispatches list, even when they did occur and the car was plugged in.

Given that the Octopus specs only allow for them to remain up to 12 hours, recalculating the whole day's costs throughout the day is a 'bad plan' and asking for errors to occur because any morning dispatch, as per the spec, probably won't be there at 23:59 and so you'll end up with innacurate data.

I raised this on Predbat with info/examples as it has the same issue of recalculating costs throughout the day and shows identical jumps in cost with no associated increase in consumption:

springfall2008/batpred#1663 (comment)

I have got around this myself by creating peak and off-peak sensors which are updated only at the time of power usage and reset at midnight back to zero - these are then totally correct because the IOG status from this integration is always correct at the time the power is used, but Octopus do not guarantee to persist that data for more than 12 hours, therefore doing it later cannot be accurate.

[BottlecapDave]

Given that the Octopus specs only allow for them to remain up to 12 hours, recalculating the whole day's costs throughout the day is a 'bad plan' and asking for errors to occur because any morning dispatch, as per the spec, probably won't be there at 23:59 and so you'll end up with innacurate data.

This is correct, except this integration should be storing completed dispatches for 3 days locally in order to keep the cost calculations accurate. This should be reflected in the completed dispatches attribute.

[BottlecapDave]

The main issue seen here is pending dispatches that are present, stick around while they're "active" and then once in the past are removed but don't get replaced with completed dispatches.

I could try and keep track of pending dispatches if they become active and make them sudo completed dispatches incase they don't turn into real completed dispatches, but I feel that could become messy and could also lead to errors if the car isn't charging while they're active and still present (as the API doesn't provide this information).

I could also update the cost sensors to only take completed dispatches into consideration, but this would result in the opposite where costs drop suddenly as completed dispatches come in.

[BottlecapDave]

As previously mentioned, this might all be resolved once OE provide a new API which takes these dispatches into consideration - https://forum.octopus.energy/t/pending-and-completed-octopus-intelligent-dispatches/8510/108, however it is said that it might not be adopted by OE. I'm also not sure if it handles upcoming dispatches, which need to be taken into consideration of target rate sensors.

[giuliodali]

What I don't follow is that the current cost sensor seems to be right near enough 100% of the time and I look at the main home assistant dashboard which shows data that matches octopus within a very tight margin of difference. So, why is the main accumulative cost sensor not just using the same current price sensor and hence going off that?

[cyberkryten]

The main issue seen here is pending dispatches that are present, stick around while they're "active" and then once in the past are removed but don't get replaced with completed dispatches.

Yes, this definitely happens. I just caught a live issue which I've added to the Octopus Energy forum post - dispatch started at 12:01-12:30 and was updated at 12:26 to extend it to 3pm.

However, the 12:01-12:26 period didn't get included in the completed dispatches so as far as anything is concerned, the charging for those 25 minutes was peak rate. I'll check later if it ever makes it to completed, but that period seems to have been lost.

Definitely seems like a bug in the Octopus system when an active dispatch gets updated/extended and other than switch to calculation of live prices only, I can't see there's really a way to fix it.