Summary:

Between 09/08/22 at 05:15 PM and 09/08/22 at 06:10 PM, we experienced a source connectivity issue in all the pipelines running against OAuth-based sources. The issue was experienced in all regions and resulted in all OAuth-based pipelines being inoperable. The bug has been fixed in production. 

Date of the first occurrence:

Aug 9, 2022 05:21 PM IST

Impact of the incident on the customer:

Impact: High

Several running pipelines got failed.

Timeline (IST):

• Aug 9 2022, 05:15 PM IST - A ticket was raised when the customer experienced a connectivity issue in one of the sources
• Aug 9 2022, 05:35 PM IST - The issue got identified. As part of the fix, our security service needed to be rolled back to the previous version.
• Aug 9 2022, 05:48 PM IST - Security service deployment got triggered.
• Aug 9 2022, 06:01 PM IST - Deployment got over.  Changes were verified by running a few pipelines in different regions.
• Aug 9 2022, 06:10 PM IST - The cluster was fully operational.
• Aug 10 2022, 03:15 PM IST - Proper fix was deployed, and changes were tested to make sure pipelines are operational.

Root cause:

This happened due to a bug identified for the OAuth service at the backend.

Solution:

The bug was identified and resolved, and the pipelines were restored.