Goldsky - Subgraph indexing experiencing issues – Incident details

Subgraph indexing experiencing issues

Resolved
Operational
Started 22 days agoLasted about 6 hours

Affected

Subgraphs

Major outage from 2:38 PM to 5:38 PM, Operational from 5:38 PM to 8:51 PM

API

Major outage from 2:38 PM to 5:38 PM, Operational from 5:38 PM to 8:51 PM

Chain Ingestion

Major outage from 2:38 PM to 5:38 PM, Operational from 5:38 PM to 8:51 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved. We have a clear root cause and know how to prevent this going into the future. We'll continue reviewing our processes and will make a post-mortem available to enterprise-plan customers.

  • Monitoring
    Monitoring

    Two of our database machines became inaccessible. After working with AWS, we were able to access it. During the incident, we have moved VIP workloads to use other databases, and after monitoring and ensuring it won't happen again, we'll move back to our existing distributed setup.

  • Identified
    Identified

    We identified issues indexing new blocks and are continuing to work on a fix for this incident.