Goldsky - Notice history

Mirror Replication experiencing partial outage

API - Operational

100% - uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024
Apr 2024
May 2024

Dashboard - Operational

100% - uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024
Apr 2024
May 2024
100% - uptime

Mirror Replication - Partial outage

100% - uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024
Apr 2024
May 2024

Chain Ingestion - Operational

100% - uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024
Apr 2024
May 2024

Subgraph Sources - Operational

100% - uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024
Apr 2024
May 2024
100% - uptime

API - Operational

Chain Ingestion - Degraded performance

100% - uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024
Apr 2024
May 2024

Arweave Search Gateway - Operational

100% - uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024
Apr 2024
May 2024

Notice history

May 2024

App and query endpoints returning 500 errors
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified
    We are continuing to work on a fix for this incident.
Latency spike in subgraph API endpoint queries
  • Resolved
    Resolved

    We observed increased latency on subgraph API endpoints as a result of a database failover between 9:45 and 9:58 AM Pacific time.

    The systems are now operating normally.

Some subgraph pipelines may have lag for recent data
  • Resolved
    Resolved

    This incident has been resolved. We experienced a spike in data received which caused a backlog in processing subgraph events for mirror pipelines. We've cleared the backlog and are putting mitigations in place to prevent further issues.

  • Identified
    Identified

    Not all customers are affected, and no data has been lost, we've identified the issue and are working to resolve it.

Queries for some subgraphs are timing out
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    This issue is resolved and the team is monitoring.

  • Identified
    Identified

    A portion of subgraphs have their queries timing out. Our team is mitigating the issue currently.

Apr 2024

Mirror pipeline maintance
  • Completed
    April 23, 2024 at 9:20 PM
    Completed
    April 23, 2024 at 9:20 PM

    Upgrading underlying infrastructure, there may be slight delays in mirror pipelines

Subgraph indexing is lagging for some subgraphs
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.
Mirror pipelines with subgraph sources are experiencing lag
  • Resolved
    Resolved

    Subgraph pipeline ingestion is back online

Outage with upstream provider affecting pipelines
  • Resolved
    Resolved

    We are currently investigating an outage with an upstream provider. This will affect pipeline creation and operation of some pipelines. There may be lag in data arriving, but no data has been lost.

Some subgraph APIs are experiencing high latency
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.
  • Resolved
    Resolved
    This incident has been resolved.

Mar 2024

Subgraph indexer maintenance
  • Completed
    March 26, 2024 at 1:45 PM
    Completed
    March 26, 2024 at 1:45 PM

    We are planning for a scheduled maintenance during this time which may prevent deploys of new subgraphs and lead to indexing lag until maintenance completes.

Subgraph indexing is lagging for some subgraphs
  • Resolved
    Resolved

    This incident was resolved.

    The root cause was a runaway DB instance that caused higher commit latencies. Our engineers have a plan on mitigating this going forward

Ethereum Mainnet Trace data is not real-time
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We're currently updating Ethereum Mainnet Traces and Decoded Traces and there may be pausing or lag for that data source. Other data sources are not affected.

Arweave Search Gateway: Stale data and Degraded Query Performance
  • Resolved
    Resolved

    The search gateway's query latencies and indexing rate has now stabilized. We're doing a set of final maintenance

    Rate limits have increased to 50 per 10s and will be raised further on Monday

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result. Query speeds are still affected as we complete additional infrastructure upgrades.

  • Identified
    Identified

    A database upgrade for the Arweave Search Gateway API has failed, and is recovering. Query latency may be slower, and data will be stale during some parts of the recovery.

    Estimate recovery time may vary, from 2-10 hours.

Mar 2024 to May 2024

Next