Goldsky - Notice history

Incedent has been resolved:
Subgraph support for HyperEVM was temporarily disrupted for ~1 hour due to some of the nodes on our load balancer not returning correct blocks data. This issue took a bit longer than expected to get resolved as the new node configuration seemingly fixed the problem but it turned out to be a false signal; we identified it and applied a proper configuration to get it resolved.
As a countermeasure, we've also added new network-wide monitors to proactively notify us if this specific problem shall occur in the future.

We implemented a fix and are currently monitoring the result.

We had multiple providers and types of nodes we would do consensus checking on to ensure data quality during subgraph indexing. Two agreed upon results were required for consensus to pass.

Unfortunately the majority of the nodes had issues. We manually checked which one was going to be more accurate and shifted to it for now, and will recover with consensus checking once the EU team comes on.

Solana data is now streaming at edge.

The root cause was an EC2 nodepool scaling failure for a region.

Our auto-cross-AZ redundancy networking didn’t recover the pipelines correctly in this scenario due to some edge cases. We have fixed this with a workaround and the auto-cross-az recovery will be fixed with an update in the coming weeks.

Base mainnet is operational again.

This incident has been resolved. We have a clear root cause and know how to prevent this going into the future. We'll continue reviewing our processes and will make a post-mortem available to enterprise-plan customers.

Two of our database machines became inaccessible. After working with AWS, we were able to access it. During the incident, we have moved VIP workloads to use other databases, and after monitoring and ensuring it won't happen again, we'll move back to our existing distributed setup.