Goldsky - Notice history

We implemented a fix and are currently monitoring the result.

We had multiple providers and types of nodes we would do consensus checking on to ensure data quality during subgraph indexing. Two agreed upon results were required for consensus to pass.

Unfortunately the majority of the nodes had issues. We manually checked which one was going to be more accurate and shifted to it for now, and will recover with consensus checking once the EU team comes on.

Solana data is now streaming at edge.

The root cause was an EC2 nodepool scaling failure for a region.

Our auto-cross-AZ redundancy networking didn’t recover the pipelines correctly in this scenario due to some edge cases. We have fixed this with a workaround and the auto-cross-az recovery will be fixed with an update in the coming weeks.

Base mainnet is operational again.

This incident has been resolved. We have a clear root cause and know how to prevent this going into the future. We'll continue reviewing our processes and will make a post-mortem available to enterprise-plan customers.

Two of our database machines became inaccessible. After working with AWS, we were able to access it. During the incident, we have moved VIP workloads to use other databases, and after monitoring and ensuring it won't happen again, we'll move back to our existing distributed setup.