Goldsky - Notice history

All systems operational

API - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Dashboard - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024
100% - uptime

Mirror Replication - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Chain Ingestion - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 99.98%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Subgraph Sources - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024
100% - uptime

API - Operational

Chain Ingestion - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 99.98%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Arweave Search Gateway - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 99.98%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Notice history

Jul 2024

Subgraphs erroneously marked as failed
  • Resolved
    Resolved

    During a routine database tuning operation at 8:02am Pacific Time, subgraph failure emails were sent out in error, and users looking at the dashboard might have seen an error status on the subgraph.

    The subgraphs marked with errors were in fact healthy, and the error status would have disappeared after 5 minutes.

    Subgraph indexing and querying was not degraded during that time.

    Our error notification system will be tuned further to prevent this from happening again, sorry for the inconvenience!

Increased database load is causing high latency for API server requests
  • Resolved
    Resolved

    This incident has been resolved.

    A database read replica was locked due to an automated maintenance system. We have implemented an immediate fix and are investigating the underlying causes to prevent future issues like this.

  • Investigating
    Update

    We've opened a support ticket with AWS. One of our database shards is under high load and our proxies are showing a high number of pending transactions.

  • Investigating
    Investigating
    We are currently investigating this incident.
Indexing for Arweave has been temporarily interrupted
  • Resolved
    Resolved

    This incident has been resolved. Arweave search and indexing is back to normal, no data has been lost.

  • Identified
    Identified
    This incident has been resolved.
BSC indexing stalled
  • Resolved
    Resolved

    BSC indexing has resolved

  • Monitoring
    Monitoring

    We have found a new RPC provider that is not having issues. Status seems resolved. We will be monitoring for further issues.

  • Investigating
    Investigating

    Our RPC providers are having issues with the BSC nodes.

Mirror pipes using Subgraph Sources may have higher latency
  • Resolved
    Resolved

    DB load is back to normal. Mirror pipelines using subgraph sources should no longer be lagging.

    Subgraph indexing was not affected, but replication into mirror pipelines may have seen lag over this period of time.

  • Investigating
    Investigating

    One of our database shards is experiencing high write volume, which is resulting in lag for mirror pipelines using Subgraph Sources for some subgraphs.

Jun 2024

Some subgraphs are experiencing query time outs.
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    One of our database shards is undergoing heavy load, and certain subgraphs with larger tables will receive timeouts.

API performance for a subset of subgraphs was degraded
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    Users may see elevated timeout rates for some subgraphs.

Database connectivity and performance issues
  • Resolved
    Resolved

    We have tuned our database backend and have recovered. All subgraph mirror pipelines should have the most up to date data at edge, and no data was lost.

  • Identified
    Identified

    We are currently investigating this incident. Some users are reporting that the GraphQL API for subgraphs has issues connecting to the database. Sometimes the dashboard is slow to load. Subgraph replication with mirror is lagging.

    We have opened a chat with AWS to help investigate our database.

    We have identified the root cause, measures are being applied but will take some time to complete.

New Subgraph deployments may time out
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    Subgraph deployments are degraded. We've found the issue (runaway wantlist growth for IPFS) and are working on a fix.

May 2024

App and query endpoints returning 500 errors
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified
    We are continuing to work on a fix for this incident.
Latency spike in subgraph API endpoint queries
  • Resolved
    Resolved

    We observed increased latency on subgraph API endpoints as a result of a database failover between 9:45 and 9:58 AM Pacific time.

    The systems are now operating normally.

Some subgraph pipelines may have lag for recent data
  • Resolved
    Resolved

    This incident has been resolved. We experienced a spike in data received which caused a backlog in processing subgraph events for mirror pipelines. We've cleared the backlog and are putting mitigations in place to prevent further issues.

  • Identified
    Identified

    Not all customers are affected, and no data has been lost, we've identified the issue and are working to resolve it.

Queries for some subgraphs are timing out
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    This issue is resolved and the team is monitoring.

  • Identified
    Identified

    A portion of subgraphs have their queries timing out. Our team is mitigating the issue currently.

May 2024 to Jul 2024

Next