Summary of Impact: Between 07:05 UTC and 09:45 UTC on 25 January 2023, customers experienced issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in Public Azure regions, as well as other Microsoft services including M365 and PowerBI.
Preliminary Root Cause: We determined that a change made to the Microsoft Wide Area Network (WAN) impacted connectivity between clients on the internet to Azure, connectivity between services within regions, as well as ExpressRoute connections.
Mitigation: We identified a recent change to WAN as the underlying cause and have rolled back this change. Networking telemetry shows recovery from 09:00 UTC onwards across all regions and services, with the final networking equipment recovering at 09:35 UTC. Most impacted Microsoft services automatically recovered once network connectivity was restored, and we worked to recover the remaining impacted services.
We are sorry for the inconvenience caused.
Posted Jan 25, 2023 - 15:08 CET
Monitoring
Please find update from Microsoft Azure:
Most customers should now see full recovery as WAN networking has recovered fully. We are working to monitor and ensure full recovery for services that were impacted. The next update will be in 30 minutes or as soon as we have further information.
Posted Jan 25, 2023 - 11:35 CET
Update
Please find update from Microsoft Azure:
We've determined the network connectivity issue is occurring with devices across the Microsoft Wide Area Network (WAN). This impacts connectivity between clients on the internet to Azure, as well as connectivity between services in datacenters, as well as ExpressRoute connections. The issue is causing impact in waves, peaking approximately every 30 minutes.
We have identified a recent WAN update as the likely underlying cause, and have taken steps to roll back this update. Our latest telemetry shows signs of recovery across multiple regions and services, and we are continuing to actively monitor the situation.
Posted Jan 25, 2023 - 10:53 CET
Identified
Please find the update from Microsoft Azure:
Azure Networking - Multiple regions - Investigating Starting at 07:05 UTC on 25 January 2023, customers may experience issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in multiple regions, as well as other Microsoft services. We are actively investigating and will share updates as soon as more is known.
Posted Jan 25, 2023 - 10:14 CET
Investigating
We are currently experiencing issues with all Cumulocity platforms that are hosted on Microsoft Azure that are having it's root cause based on issues in the Microsoft infrastructure.
We will inform you about any changes in that matter in due course.
Posted Jan 25, 2023 - 10:13 CET
This incident affected: emea.cumulocity.com (emea.cumulocity.com REST API, emea.cumulocity.com MQTT API).