Goldsky - Notice history

We implemented a fix and are currently monitoring the result. The problem is improving slowly. We are looking for how to accelerate the rate of recovery.

Incedent has been resolved:
Subgraph support for HyperEVM was temporarily disrupted for ~1 hour due to some of the nodes on our load balancer not returning correct blocks data. This issue took a bit longer than expected to get resolved as the new node configuration seemingly fixed the problem but it turned out to be a false signal; we identified it and applied a proper configuration to get it resolved.
As a countermeasure, we've also added new network-wide monitors to proactively notify us if this specific problem shall occur in the future.

We implemented a fix and are currently monitoring the result.

We had multiple providers and types of nodes we would do consensus checking on to ensure data quality during subgraph indexing. Two agreed upon results were required for consensus to pass.

Unfortunately the majority of the nodes had issues. We manually checked which one was going to be more accurate and shifted to it for now, and will recover with consensus checking once the EU team comes on.

Solana data is now streaming at edge.

The root cause was an EC2 nodepool scaling failure for a region.

Our auto-cross-AZ redundancy networking didn’t recover the pipelines correctly in this scenario due to some edge cases. We have fixed this with a workaround and the auto-cross-az recovery will be fixed with an update in the coming weeks.

Base mainnet is operational again.