Multiple identified data gaps including an 8-day outage encountered in Dec-2019, have had production data successfully restored via a backfill operation process.
The immediate cause was determined to be related to the time when a meter last provided a set of readings versus the time the readings were registered for inclusion, with a contributing cause associated to the time of the "last sync" between system databases, which the vendor identified and as previously reported has provided root cause based corrective action.
With the root cause analysis and backfill operation procedure both completed, this incident is considered resolved.
Posted Nov 17, 2020 - 14:49 PST
Monitoring
Iterative changes with vendor participation have improved reliability of daily meter reading ingestion, which remains closely scrutinized to preclude data gap recurrence. Processes and procedures are underway to apply retroactive reading inclusion across the fleet on a a rolling 60 day basis, with operation set to commence within the next 5 business days, after a small subset (~2k meters) has been completely confirmed w.r.t. daily data integrity, which is presently in review.
Updates will be posted as we have new information, particularly an ETA on completion.
Posted Oct 15, 2020 - 16:10 PDT
Update
We are experiencing delayed or missing data from our CATM meters as a result of issues communicating with meters and the data ingestion workflow to LocusNOC. We are working closely with our vendor to resolve this ASAP. We will provide updates as they become available.
Posted Sep 30, 2020 - 10:41 PDT
Update
We are continuing to work on a fix for this issue. At this time, some data has been ingested into DAS. Updates will be posted as we have new information.
Posted Sep 10, 2020 - 23:00 PDT
Identified
The vendor has been contacted regarding investigation findings and additional testing underway to fully isolate the matter to resolution. Readings are approximately 36 hours behind, yet are present in the upstream system database. More information to be provided as it becomes available.
Posted Sep 10, 2020 - 20:32 PDT
Investigating
We are continuing to investigate the issue. Updates will be posted as we have new information.
Posted Sep 10, 2020 - 16:04 PDT
This incident affected: LocusNOC (API, Data Acquisition).