Users may be unable to access or use some Microsoft 365 services and features
Issue ID: MO941162
Affected services: Exchange Online, Microsoft 365 suite, Microsoft Power Automate in Microsoft 365, Microsoft Purview, Microsoft Teams, SharePoint Online
Status: Service degradation
Issue type: Incident
Start time: Nov 24, 2024, 6:54 PM PST
More info
The affected scenarios are as follows:
Users may be unable to access Exchange Online using the following impacted connection methods:
- Outlook on the web
- Outlook desktop client
- Representational State Transfer (REST)
- Exchange ActiveSync (EAS)
Users may be unable to use the following features within Microsoft Teams:
- Users are unable to create or update Virtual Events, including webinars and Town Halls.
- Users may be unable to access or modify their calendar in Microsoft Teams. This would include loading calendar, viewing meetings, creating/updating meetings and joining meetings.
- Users are unable to create chat, add users and create or edited meetings.
- Users are unable to create or modify new teams and channels.
- Users may be unable to update presence.
- Users may be unable to use the search function.
- Users may not see updated list of files and links failing to load within the Chat shared tab.
Users may experience the following issues with Microsoft Purview:
- Users may be unable to access the Purview Portal, or Purview Solutions.
- Users may experience delays in policy stamping and with Adaptive Scope Evaluations.
Users may be unable to export content or set and view labels within Microsoft Fabric.
Some Microsoft Fabric users with Purview Information Protection Policies with sensitivity labels enabled, may be unable to use interactive operations on Power BI Desktop format files and reports, including export operations on Fabric artifacts with Sensitivity labels applied.
Users may be unable to use the search feature within SharePoint Online.
Users may be unable to perform the following actions within Microsoft Defender for Office365.
- Users may be unable to create simulations, simulation payloads or end user notifications.
- Users may experience issues with delivery for end user notifications and simulation messages
- Some users may experience failures in manual or AIR approved Remediation Actions submitted through ThreatExplorer, Advanced Hunting or the Action Center.
- Users may experiences issues with viewing simulation reports, and content.
Users may also be unable to print via Universal Print.
Users may experience errors running flows that utilize cloud connectors in Power Automate for Desktop.
Users may be unable to access their bookings within Microsoft Bookings.
Users may be unable to use some Microsoft Copilot features including:
- Users are unable to use the personal Copilot panel in meetings and post meetings.
- Users are unable to see historic Copilot conversation history in meetings and post meetings.
Scope of impact
Any user routed through affected infrastructure and attempting to use the functionalities outlined in the More info section of this communication may be affected by this event.
Preliminary root cause
A recent change has resulted in a portion of infrastructure not operating as expected.
Current status
Nov 25, 2024, 8:44 AM PST
History of updates
Nov 25, 2024, 8:15 AM PST
The fix which is addressing impact is continuing to deploy and has now reached 98 percent of the affected environment. Regarding recovery, we’ve encountered delays in our targeted restarts; however, we’re identifying alternative options to expedite recovery of the affected infrastructure.
Nov 25, 2024, 8:02 AM PST
Nov 25, 2024, 6:27 AM PST
Nov 25, 2024, 5:58 AM PST
We’ve started to deploy the fix which is currently progressing through the affected environment. While this progresses, we’re beginning manual restarts on a subset of machines that are in an unhealthy state.
Nov 25, 2024, 5:30 AM PST
Nov 25, 2024, 3:50 AM PST
We’ve identified a recent change which we believe has resulted in impact. We’ve begun to revert the change and are investigating what additional actions are required to mitigate the issue.
Nov 25, 2024, 2:35 AM PST
We’ve updated the More Info section to include a list of impacted services and scenarios. We’re continuing to investigate recent changes, service telemetry and the failure paths to determine the root cause of impact.
Nov 25, 2024, 12:54 AM PST
While we’re continuing the restarts to improve service availability, we’re thoroughly analyzing recent changes and the specific failure path to determine the next steps required to resolve the issue.
Nov 25, 2024, 12:18 AM PST
Nov 24, 2024, 11:21 PM PST
Our ongoing service monitoring has determined that the issue is not limited to users in Asia Pacific, and instead that all users worldwide could be intermittently impacted. As such, we’re expanding our restart efforts to ensure the issue is resolved for all users.
Nov 24, 2024, 9:46 PM PST
We’re continuing to work to restart the small subset of machines which have previously failed to restart. We’re also continuing to add additional capacity to the affected environment to mitigate impact as quickly as possible. At this time, we expect the majority of users will experience relief from impact as we work to address the remaining issues.
Nov 24, 2024, 8:56 PM PST
We’ve identified a small subset of machines which are failing to restart, resulting in persisting impact. We’re further investigating what is causing the restarts to fail to determine our next steps.
Nov 24, 2024, 8:18 PM PST
Nov 24, 2024, 7:50 PM PST
We’ve determined via our internal telemetry that the impact to Microsoft Teams calendars has self-recovered. To fully resolve the issue, we initiated restarts on affected infrastructure and are monitoring further to determine whether additional steps are required. We’re also continuing our investigation into the root cause of this issue.
Nov 24, 2024, 7:29 PM PST
Nov 24, 2024, 7:00 PM PST
Our automated systems alerted us to an issue in which some users may be intermittently unable to access their mailboxes in Exchange Online using any connection method or their calendar functionality in Microsoft Teams. We’re analyzing a portion of Exchange Online infrastructure responsible for accessing mailboxes to understand the root cause of the issue and determine our next troubleshooting steps.
Latest update from Microsoft:
We’re continuing to reroute traffic to alternate infrastructure and have reinitiated targeted server restarts to ensure the fix takes effect as expected. We’re monitoring to confirm the restarts proceed successfully. We don’t yet have an estimated time to resolution; however, we’ll provide one as soon as it becomes available.
Newest update:
We identified a change in the environment that resulted in an influx in request retries routed through affected servers. Our optimizations, which enhanced the infrastructure’s processing capabilities, continue to provide incremental relief. We’re monitoring the service and continuing our work to perform any follow-up actions or opening additional workstreams needed to fully resolve the problem.
We understand the significant impact of this event to your organization, we’re treating this issue with the highest priority, and we’re working to provide relief as soon as possible.
Microsoft has resolved the issue.