Goldsky - Solana ingestion is degraded because of upstream RPC issues. – Incident details

Solana ingestion is degraded because of upstream RPC issues.

Resolved
Degraded performance
Started 5 days agoLasted about 1 hour

Affected

Mirror

Degraded performance from 1:48 AM to 3:10 AM, Operational from 1:48 AM to 3:10 AM

Mirror Replication

Operational from 1:48 AM to 3:10 AM

Chain Ingestion

Degraded performance from 1:48 AM to 3:10 AM

Subgraph Sources

Operational from 1:48 AM to 3:10 AM

Updates
  • Resolved
    Resolved

    We have identified the root cause of this outage. The issue originated with an upstream RPC provider that began missing specific blocks. This was a failure mode we had not previously encountered. As a result, Solana Turbo pipelines stalled as parallel workers entered a retry loop waiting on unavailable blocks.

    Detection was difficult initially, as all existing health indicators remained green. We have since forced a reset of affected pipelines to exit the retry loop, and processing has resumed. We have also deployed a dedicated monitor for this failure mode and are working to deploy automatic recovery for future incidents.

  • Identified
    Identified

    We are continuing to work on a fix for this incident, we're working on load balancing with additional RPC providers.