Resolved -
All processing are back to normal over the last 48 hours.
Umbrella team is working on long term solutions that will improve our resiliency from such infrastructure services malfunctioning and improve our recovery time.
May 6, 09:57 UTC
Update -
Processing is back to normal with the exception of processing GCP K8S which is still suffering delays.
May 3, 06:56 UTC
Update -
We are continuing to monitor for any further issues.
May 3, 06:54 UTC
Update -
While 90% of invoices are now running in latency < 12hr there are still some accounts which takes longer, we are working to reduce the load on our data infrastructure and in contact with our cloud providers to remove some of the bottlenecks.
May 2, 18:38 UTC
Update -
Most invoices were updated with latest data but there are still some lagging and slow in processing. We expect full recovery by end of the day.
Apr 29, 09:33 UTC
Monitoring -
Data infrastructure issue has been handled, we're monitoring the progress of re-running invoices, we'll notify once we're back to normal processing.
Apr 28, 12:29 UTC
Update -
Root cause is in one of the data services we consume from AWS, we are actively working with them on solving the issue.
Apr 28, 08:20 UTC
Identified -
Please expect a delay in the current invoice data
Apr 28, 08:11 UTC