LSI22013 - Výpadek databázových operací / Database operations failing
Incident Report for TALXIS
Resolved
EN:

Cosmos DB issue has been resolved

CZ:

Cosmos DB problém je vyřešený.
Posted Sep 08, 2022 - 15:10 CEST
Update
EN:

Services and features impacted include, but are not limited to:

- Delays for calling and message notifications and being unable to manage/create channels, call queues and auto attendant within Microsoft Teams. Teams Meeting Rooms are also experiencing impact.

- Power App operations and app management is not performing as expected.

- Failures when attempting to run approvals, universal search scenarios, and/or connection wizard within Power Automate.

- Intermittently unable to create Roadmaps within Project for the web service.

- Unable to access Usage Reports and Adoption scores within the Admin Center, Graph API and Power BI services.

Some Microsoft 365 services, which have dependencies on Cosmos DB, are performing failovers to remediate impact. In parallel, we’re analyzing network diagnostics from the impacted Cosmos DB infrastructure to understand the underlying issue.

In parallel to the failover and migration actions to provide relief, we’re continuing to review telemetry and logs from affected components within the impacted Azure Cosmos DB cluster. This includes the analysis of networking diagnostics and the request logic flow within the service fabric layers.

- OneNote Section Groups and Lists experience failures to sync approximately 20 percent of hierarchy change operations, such as creating, moving, or modifying a section's order, color, or title. Opening a Notebook can also fail in certain circumstances. This affects Universal Windows Platform, iOS, Mac, and Android platform OneNote apps.

We continue our analysis of specific impacted nodes and containers within the impacted Cosmos DB, and we’ve identified a specific operation process that was not completing as expected. We’re investigating this operation further to determine if it is contributing to impact.
Posted Sep 07, 2022 - 19:55 CEST
Update
EN:

Microsoft is reporting the issue with creating records be resolved the following is the description of the problem and the root-cause:

September 7, 2022 3:34 PM
Title: Sandbox plug-in failures

User Impact: Users may experience failures when utilizing sandbox plug-ins

Final Status: We performed a failover of the affected database infrastructure to redundant service infrastructure, and confirmed through telemetry analysis that this restored sandbox plug-in functionality.

Incident Start Time: Wednesday, September 7, 2022, 11:55 AM (9:55 AM UTC)

Incident End Time: Wednesday, September 7, 2022, 3:25 PM (1:25 PM UTC)

Preliminary Root Cause: A portion of the database infrastructure responsible for plugin operation has entered an unhealthy state.

Next Steps: We're reviewing our monitoring services to reduce detection time and more quickly restore service.

A Post Incident Report will be published within five business days.
---

Issue with Cosmos DB is persisting. Users may encounter issues importing solutions.

CZ:

Microsoft vyřešil problém týkající se vytvoření záznamů. Níže zasláno jejich stanovisko:

September 7, 2022 3:34 PM
Title: Sandbox plug-in failures

User Impact: Users may experience failures when utilizing sandbox plug-ins

Final Status: We performed a failover of the affected database infrastructure to redundant service infrastructure, and confirmed through telemetry analysis that this restored sandbox plug-in functionality.

Incident Start Time: Wednesday, September 7, 2022, 11:55 AM (9:55 AM UTC)

Incident End Time: Wednesday, September 7, 2022, 3:25 PM (1:25 PM UTC)

Preliminary Root Cause: A portion of the database infrastructure responsible for plugin operation has entered an unhealthy state.

Next Steps: We're reviewing our monitoring services to reduce detection time and more quickly restore service.

A Post Incident Report will be published within five business days.
---

Problém s Cosmos DB stále přetrvává jsou ovlivněny hlavně importy solutions.
Posted Sep 07, 2022 - 16:06 CEST
Identified
Update from Microsoft:

September 7, 2022 3:01 PM
Title: Sandbox plug-in failures

User Impact: Users may experience failures when utilizing sandbox plug-ins

More Info: Users may receive the error message, “The plug-in execution failed because the Sandbox Worker process crashed".

Current Status: After our investigation, we identified that a portion of the database infrastructure responsible for plugin operation has entered an unhealthy state. We are currently gathering and reviewing additional telemetry in preparation of performing a failover to redundant infrastructure.

Incident Start Time: Wednesday, September 7, 2022, 1:22 PM (11:22 AM UTC)
Next Update: Wednesday, September 7, 2022, 5:00 PM (3:00 PM UTC)
Posted Sep 07, 2022 - 15:32 CEST
Update
We are continuing to investigate this issue.
Posted Sep 07, 2022 - 15:00 CEST
Update
EN:
We are checking Microsoft Service Health and it seems like the issue is connected with the platform.

We will have an update in 30 minutes.

CZ:

Kontrolujeme Microsoft Service Health a vypadá to, že problém je spojený s výpadkem platformy.

Další informace budeme mít za 30 minut.
Posted Sep 07, 2022 - 14:58 CEST
Investigating
EN:
Some operations in Dataverse may be unavailable due to issue in Microsoft services. We are investigating the root cause.

CZ:
Některé operace v Dataversu mohou být dočasně nedostupné kvůli problému na straně Microsoftu. Zjišťujeme příčinu problému.

https://status.azure.com/en-us/status
https://admin.microsoft.com/Adminportal/Home#/servicehealth/:/alerts/CR427450
Posted Sep 07, 2022 - 14:55 CEST
This incident affected: Microsoft (Power Apps, Azure) and TALXIS (Apps, Integration, Platform, Portal).