Goldsky - Notice history

All systems operational

API - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Dashboard - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024
100% - uptime

Mirror Replication - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Chain Ingestion - Operational

100% - uptime
Jul 2024 · 99.98%Aug · 99.97%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Subgraph Sources - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 99.85%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024
100% - uptime

API - Operational

Chain Ingestion - Operational

100% - uptime
Jul 2024 · 99.98%Aug · 99.99%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Arweave Search Gateway - Operational

100% - uptime
Jul 2024 · 99.98%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Notice history

Sep 2024

No notices reported this month

Aug 2024

Mirror pipelines are lagging
  • Resolved
    Resolved

    This incident has been resolved. Data will automatically be backfilled.

    Only chain ingestion was lagging, and mirror pipelines using subgraph sources were not affected.

  • Investigating
    Investigating

    We are currently investigating this incident. New blockchain events may be slow to be indexed. There has been no data loss, and we will backfill all data.

Gateway timeouts for webapp
  • Resolved
    Resolved

    This incident has been resolved. There was an infinite loop eating up resources. A fix has been deployed.

  • Investigating
    Investigating
    We are currently investigating this incident.
We identified an issue with subgraph ingestion for mirror.
  • Resolved
    Resolved

    Our ingestion system has recovered and all subgraph sourced mirror data for the affected subgraphs have been backfilled. No data was lost.

  • Monitoring
    Monitoring

    We've recovered the failing system and are currently backfilling data.

  • Identified
    Identified

    We are continuing to work on a fix for this incident. This will lead to a temporary delay in data from some subgraphs arriving in mirror sinks. We will backfill the data, and nothing should be lost.

Rolling all node groups to update subnets
  • Completed
    August 13, 2024 at 1:19 AM
    Completed
    August 13, 2024 at 1:19 AM
    Maintenance has completed successfully.
  • In progress
    August 12, 2024 at 4:00 PM
    In progress
    August 12, 2024 at 4:00 PM

    We are rolling all of our workloads from old node groups to new node groups to expand our subnets. All workloads should automatically restart. You may get notifications about subgraphs or pipelines failing, this is transient.

Compute outage with AWS
  • Resolved
    Update
    This incident has been resolved.
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Recovery has started. For the next few minutes:

    • Consumers might experience loss of data freshness.

    • Pipelines might not start.

  • Identified
    Identified

    We are in the middle of a compute outage with AWS. We have opened a high urgency support ticket with them. We expect resolution soon. Degraded performance for Mirror.

Jul 2024

Subgraphs erroneously marked as failed
  • Resolved
    Resolved

    During a routine database tuning operation at 8:02am Pacific Time, subgraph failure emails were sent out in error, and users looking at the dashboard might have seen an error status on the subgraph.

    The subgraphs marked with errors were in fact healthy, and the error status would have disappeared after 5 minutes.

    Subgraph indexing and querying was not degraded during that time.

    Our error notification system will be tuned further to prevent this from happening again, sorry for the inconvenience!

Increased database load is causing high latency for API server requests
  • Resolved
    Resolved

    This incident has been resolved.

    A database read replica was locked due to an automated maintenance system. We have implemented an immediate fix and are investigating the underlying causes to prevent future issues like this.

  • Investigating
    Update

    We've opened a support ticket with AWS. One of our database shards is under high load and our proxies are showing a high number of pending transactions.

  • Investigating
    Investigating
    We are currently investigating this incident.
Indexing for Arweave has been temporarily interrupted
  • Resolved
    Resolved

    This incident has been resolved. Arweave search and indexing is back to normal, no data has been lost.

  • Identified
    Identified
    This incident has been resolved.
BSC indexing stalled
  • Resolved
    Resolved

    BSC indexing has resolved

  • Monitoring
    Monitoring

    We have found a new RPC provider that is not having issues. Status seems resolved. We will be monitoring for further issues.

  • Investigating
    Investigating

    Our RPC providers are having issues with the BSC nodes.

Mirror pipes using Subgraph Sources may have higher latency
  • Resolved
    Resolved

    DB load is back to normal. Mirror pipelines using subgraph sources should no longer be lagging.

    Subgraph indexing was not affected, but replication into mirror pipelines may have seen lag over this period of time.

  • Investigating
    Investigating

    One of our database shards is experiencing high write volume, which is resulting in lag for mirror pipelines using Subgraph Sources for some subgraphs.

Jul 2024 to Sep 2024

Next