Dev APIs / Realtime messaging / Ably

Ably

OPERATIONAL

Since Jan 2023 · 50 incidents · status.ably.com · checked just now

User reportsVisitors who said it's down for them, per hour over 24 hours. Not from the status page. Never counted in grades.
…
CompareAdd to stackStatus page
GradeALast 90 days
APRODA+SHIP

Overview

30d90d1y
Admitted uptimeCustomer time not hit, by the provider's own account. Regional and partial incidents count in part.
99.973%Grade A
IncidentsEverything posted on the status page except scheduled maintenance.
70 Sev 1
Fully downMost customers down, by the provider's own account.
0m
Median fixMedian time to resolve. Half of incidents were fixed faster.
7m
LongestLongest Sev 1–2 incident, start to resolved.
1h 24m
Since lastTime since the last counted incident was resolved.
4d 4h

By area

AProduction
6INCIDENTS
2h 49mAFFECTED
Your users are hit: apps, APIs, databases, inference
Worst, 90d
SEV 2 · SECURITYOct 1, 2026 · 27mElevated API Errors
SEV 2Aug 28, 2026 · 1h 24m · regionalElevated API Errors - LiveObjects
SEV 2Aug 20, 2026 · 55m · regionalElevated Errors in ap-southeast-1
Most hit
Ably Pub/Sub - Asia-Pacific (Singapore)2
Integrations1
Ably Platform1
Ably Pub/Sub - Europe (Frankfurt)1
Worse. 6 in the last 6 months, 4 in the 6 before.
A+Shipping
0INCIDENTS
0mAFFECTED
You can't deploy, build or manage: CI, CLI, dashboard, auth, billing
Worst, 90dNothing.
Better. 0 in the last 6 months, 2 in the 6 before.
CleanSev 3–4Sev 1–2No data

Trend, last 12 months

Worse. 7 in the last 6 months, 6 in the 6 before.

Severity mix

Sev 1 · Lights outProd site, API or database down, or data loss0
Sev 2 · BleedingProd degraded or partly down6
Sev 3 · Can't shipCan't deploy, build or manage it0
Sev 4 · PapercutDocs, status page, slow dashboard1
1 security incident

When it breaks

Start times, last 12 months, UTC.

What breaks

Ably Website3 incidents
Ably Pub/Sub - Asia-Pacific (Singapore)3 incidents
Integrations2 incidents
Ably Platform1 incident
Ably Pub/Sub - Europe (Frankfurt)1 incident
Ably Pub/Sub - US East (Ohio)1 incident

Where it breaks

Last 12 monthsPlaces named on the status page: cloud regions, cities, data centres. Incidents that name none stay off the map.
7 places · 7 with none named
Singapore: 4 incidents · AblyFrankfurt: 2 incidents · AblyOhio: 2 incidents · AblySydney (ap-southeast-2): 2 incidents · AblyCalifornia: 1 incident · AblyDublin (eu-west-1): 1 incident · AblyVirginia: 1 incident · Ably
Sev 1–2 time by place, 12 monthsTime with a Sev 1 or 2 open that names the place, any provider. Overlaps count once.
Frankfurt1h 57m
Ohio1h 57m
Singapore1h 43m
Sydney56m
California33m
Virginia33m

Communication

Updates eachStatus page updates per incident, on average.
3.6Last 12 months
Update gapMedian time between status page updates during an incident.
6mLast 12 months
PostmortemsSev 1–2 incidents with a published write-up.
2 of 11Sev 1–2

Incident history

Oct 20, 20260mSEV 2
issue with rewind=1 and persist-last in some situationsBetween about 14:00 UTC on October 20 and 17:00 UTC on October 21, attaches using rewind=1 on channels with persist-last enabled sometimes did not receive the rewound message. The issue affected only certain conditions: the latest message was over two minutes old but recent, and the channel had been continuously active in the region. Ably apologised; the update does not describe a fix.AI-graded · some customers
Why this grade
  1. RULES · SEV 2
    Sev 2 from the incident title (no components listed)
  2. CLAUDE · SEV 2 · SUBSET · MEDIUM CONFIDENCE
    Rewound messages were sometimes not delivered to attaching clients on persist-last channels for about 27 hours; other channels were not described as affected.
  3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
1 update
  1. 13:00 UTC · resolved
    Between about 14:00 UTC on the 20th and 17:00 on the 21st, on channels with the persist last message (https://ably.com/docs/channels#rules) channel rule enabled, attachments with `rewind=1` in situations where the most recent message was published over two minutes ago but still somewhat recently (and where the channel has been continuously active in a given region since the message was published) may not have had that rewound message correctly delivered to the attaching client. We apologise to anyone whose application was affected by this.
Source
Oct 16, 20260mSEV 2
MongoDB ingress rule issuesMongoDB integration rules may have failed to publish changestream events for about 1h45m; impact unclear and limited to affected customers.AI-graded · some customers · Integrations
Why this grade
  1. RULES · SEV 2
    Sev 2 because Integrations had degraded performance
  2. CLAUDE · SEV 2 · SUBSET · LOW CONFIDENCE
    MongoDB integration rules may have failed to publish changestream events for about 1h45m; impact unclear and limited to affected customers.
  3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
1 update
  1. 13:00 UTC · resolved
    Between 10:35 and 12:20 UTC today, MongoDB ingress rules may have experienced errors publishing changestream events. If you were affected by this, please contact [email protected].
Source
Oct 1, 202627mSEV 2 · SECURITY
Elevated API ErrorsA DDoS attack caused elevated token-request errors with 503 responses in all Ably Realtime regions for about six minutes on October 1, 2026. Ably restored service by adding capacity and deployed traffic-blocking and rate-limiting rules. ably.com was also intermittently unavailable.AI-graded
Why this grade
  1. RULES · SEV 1
    Sev 1 from the incident title (no components listed)
  2. CLAUDE · SEV 2 · GLOBAL · MEDIUM CONFIDENCE · SECURITY
    A DDoS attack caused token-request 503 errors across all Ably regions for about six minutes; the website was intermittently unavailable.
  3. Claude Haiku 5.5 · prompt v11 · Oct 24, 2026 · Raw record
3 updates
  1. 15:36 UTC · investigating
    We're experiencing an elevated level of API errors and are currently looking into the issue.
  2. 16:03 UTC · resolved
    Between 15:30 UTC to 15:36 UTC there were elevated errors affecting token requests in all regions resulting in 503 error responses.
  3. 17:20 UTC · postmortem
    On October 1, a distributed denial-of-service \(DDoS\) attack caused elevated token-request errors across all Ably Realtime regions for approximately six minutes and intermittent unavailability of [ably.com](http://ably.com). We restored token-request service by increasing capacity and added targeted traffic-blocking and rate-limiting rules to mitigate the attack. We are strengthening endpoint protection and reviewing our firewall controls to reduce the risk of recurrence.
PostmortemSource
Sep 7, 20267mSEV 4
Ably documentation pages - ably.com/docsAbly documentation pages on ably.com/docs failed to load for about seven minutes; realtime services were not affected.AI-graded · Ably Website
Why this grade
  1. RULES · SEV 4
    Sev 4 because Ably Website had a partial outage
  2. CLAUDE · SEV 4 · GLOBAL · HIGH CONFIDENCE
    Ably documentation pages on ably.com/docs failed to load for about seven minutes; realtime services were not affected.
  3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
3 updates
  1. 11:25 UTC · investigating
    Customers may be experiencing issues viewing documentation pages on our website ably.com/docs. Our realtime services are not affected by this issue at this time.
  2. 11:28 UTC · monitoring
    A fix has been implemented and we are monitoring the results.
  3. 11:32 UTC · resolved
    This incident has been resolved.
Source
Aug 28, 20261h 24mSEV 2
Elevated API Errors - LiveObjectsAbly saw elevated API errors in LiveObjects affecting Pub/Sub in Europe (Frankfurt) and US East (Ohio) from 12:08 to 13:33 UTC on 28 August. They mitigated while investigating, deployed a fix, and resolved the incident. The root cause was not described in the updates.AI-graded · regional · Ably Platform +2
Why this grade
  1. RULES · SEV 1
    Sev 1 because Ably Platform had a partial outage
  2. CLAUDE · SEV 2 · REGIONAL · MEDIUM CONFIDENCE
    LiveObjects API requests failed at elevated rates in Frankfurt and US East Ohio for about 75 minutes; other regions were not named.
  3. Claude Haiku 5.5 · prompt v11 · Oct 24, 2026 · Raw record
6 updates
  1. 12:08 UTC · investigating
    We're experiencing an elevated level of API errors in LiveObjects and are currently looking into the issue.
  2. 12:17 UTC · identified
    We're mitigating the impact while investigating the root cause.
  3. 13:22 UTC · identified
    We are continuing to work on a fix for this issue.
  4. 13:26 UTC · monitoring
    A fix has been implemented and we are monitoring the results.
  5. 13:26 UTC · monitoring
    We are continuing to monitor for any further issues.
  6. 13:33 UTC · resolved
    This incident has been resolved.
Source
Aug 20, 20261h 17mSEV 2
Elevated Error Rate in ap-southeast-12 incidents
  1. Thu, Aug 20 19:07 UTC3mSEV 2
    Elevated Error Rate in ap-southeast-1Pub/Sub in the Singapore region saw elevated errors for about three minutes; other regions were not reported affected.AI-graded · regional · Ably Pub/Sub - Asia-Pacific (Singapore)
    Why this grade
    1. RULES · SEV 1
      Sev 1 because Ably Pub/Sub - Asia-Pacific (Singapore) had a partial outage
    2. CLAUDE · SEV 2 · REGIONAL · MEDIUM CONFIDENCE
      Pub/Sub in the Singapore region saw elevated errors for about three minutes; other regions were not reported affected.
    3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
    2 updates
    1. 19:07 UTC · investigating
      We're experiencing an elevated level of errors in the ap-southeast-1 region and are currently looking into the issue.
    2. 19:10 UTC · resolved
      This incident has been resolved.
    Source
  2. Thu, Aug 20 19:29 UTC55mSEV 2
    Elevated Errors in ap-southeast-1Ably Pub/Sub in the Singapore region (ap-southeast-1) showed elevated error rates from about 19:29 to 20:24 UTC on 2026-08-20. Error rates returned to normal by 19:42 UTC after a fix; the provider kept the incident open while monitoring and then resolved it.AI-graded · regional · Ably Pub/Sub - Asia-Pacific (Singapore)
    Why this grade
    1. RULES · SEV 2
      Sev 2 because Ably Pub/Sub - Asia-Pacific (Singapore) had degraded performance
    2. CLAUDE · SEV 2 · REGIONAL · HIGH CONFIDENCE
      Pub/Sub API in ap-southeast-1 (Singapore) had elevated errors for about 13 minutes before error rates returned to normal; other regions were not reported affected.
    3. Claude Haiku 5.5 · prompt v11 · Oct 24, 2026 · Raw record
    5 updates
    1. 19:29 UTC · investigating
      We're experiencing an elevated level of errors in ap-southeast-1 and are currently looking into the issue.
    2. 19:41 UTC · identified
      The issue has been identified and a fix is being implemented.
    3. 19:42 UTC · identified
      We are continuing to work on a fix for this issue. Error rates have decreased to normal levels and the service is currently fully operational.
    4. 20:12 UTC · monitoring
      A fix has been implemented and we our monitoring the result. Error rates have decreased to normal levels and the service is currently fully operational.
    5. 20:24 UTC · resolved
      This incident has been resolved.
    Source
Feb 17, 202612mSEV 2
Elevated realtime publish error rate in ap-southeast-1Ably saw elevated realtime publish errors in its ap-southeast-1 (Singapore) Pub/Sub region from about 10:19 to 10:32 UTC on 17 February 2026. A fix was implemented and the incident was resolved. The text gives no root cause and no explicit statement of full recovery before resolution.AI-graded · regional · Ably Pub/Sub - Asia-Pacific (Singapore)
Why this grade
  1. RULES · SEV 2
    Sev 2 because Ably Pub/Sub - Asia-Pacific (Singapore) had degraded performance
  2. CLAUDE · SEV 2 · REGIONAL · MEDIUM CONFIDENCE
    Realtime publish requests failed at an elevated rate in the Singapore region for about 12 minutes; other regions were not reported affected.
  3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
4 updates
  1. 10:19 UTC · investigating
    We're experiencing an elevated level of realtime publish errors in ap-southeast-1 and are currently looking into the issue.
  2. 10:24 UTC · identified
    The issue has been identified and a fix is being implemented.
  3. 10:30 UTC · monitoring
    A fix has been implemented and we are monitoring the results.
  4. 10:32 UTC · resolved
    This incident has been resolved.
Source
Feb 14, 20268mSEV 2
Elevated Attach Error Rate in eu-west-1Ably saw elevated channel attachment errors in its eu-west-1 Pub/Sub region from about 05:57 to 06:05 UTC on 14 February 2026. A fix was implemented and monitored, and the incident was resolved after about eight minutes.AI-graded · regional · Ably Pub/Sub - Europe (Ireland)
Why this grade
  1. RULES · SEV 1
    Sev 1 because Ably Pub/Sub - Europe (Ireland) had a partial outage
  2. CLAUDE · SEV 2 · REGIONAL · MEDIUM CONFIDENCE · BACK AFTER 4M
    Realtime channel attach errors elevated in eu-west-1 (Ireland) for about eight minutes; other regions were not reported affected.
  3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
4 updates
  1. 05:57 UTC · investigating
    We're experiencing an elevated level of attachment errors in eu-west-1 and are currently looking into the issue.
  2. 06:00 UTC · identified
    The issue has been identified and a fix is being implemented.
  3. 06:01 UTC · monitoring
    A fix has been implemented and we are monitoring the results.
  4. 06:05 UTC · resolved
    This incident has now been resolved.
Source
Jan 21, 20261h 41mSEV 3
SMS based 2FA is currently unavailableSMS-based two-factor login to the Ably website was unavailable from 18:11 to 19:52 UTC on 21 January 2026. A fix was implemented around 18:36 and the incident was resolved at 19:52. The provider reported no cause beyond the fix.AI-graded · some customers · Ably Website
Why this grade
  1. RULES · SEV 4
    Sev 4 because Ably Website had degraded performance
  2. CLAUDE · SEV 3 · SUBSET · MEDIUM CONFIDENCE
    SMS two-factor login to the Ably website failed for about 85 minutes; API and realtime services were not reported affected.
  3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
4 updates
  1. 18:11 UTC · investigating
    We are currently investigating an issue impacting SMS based 2FA login to the Ably website ably.com.
  2. 18:29 UTC · identified
    The issue has been identified and a fix is being implemented.
  3. 18:36 UTC · monitoring
    A fix has been implemented and we are monitoring the results.
  4. 19:52 UTC · resolved
    This incident has been resolved.
Source
Dec 11, 202524mSEV 2
Push delivery errors for messages published in ap-southeast-2Push notification delivery errors for messages published in ap-southeast-2 for about 23 minutes; other regions unaffected.AI-graded · regional
Why this grade
  1. RULES · SEV 2
    Sev 2 from the incident title (no components listed)
  2. CLAUDE · SEV 2 · REGIONAL · MEDIUM CONFIDENCE
    Push notification delivery errors for messages published in ap-southeast-2 for about 23 minutes; other regions unaffected.
  3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
3 updates
  1. 18:36 UTC · investigating
    Increased level of push delivery errors for messages published in ap-southeast-2 only since 15:28 UTC, we're remediating
  2. 18:36 UTC · identified
    The issue has been identified and a fix is being implemented.
  3. 18:59 UTC · resolved
    This incident has been resolved.
Source
Dec 5, 202519mSEV 3
Ably website, customer dashboard & Control API issuesWebsite, customer dashboards and the Control API had connectivity problems for about 20 minutes; realtime services were not affected.AI-graded · Ably Website
Why this grade
  1. RULES · SEV 4
    Sev 4 because Ably Website had a major outage
  2. CLAUDE · SEV 3 · UNCLEAR · MEDIUM CONFIDENCE
    Website, customer dashboards and the Control API had connectivity problems for about 20 minutes; realtime services were not affected.
  3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
3 updates
  1. 08:59 UTC · investigating
    Customers may be experiencing connectivity issues affecting our website (ably.com), customer dashboards, the Control API (control.ably.net), the Ably blog and CMS. Our realtime services are not affected by this issue at this time.
  2. 09:14 UTC · monitoring
    The service is recovering, we are continuing to monitor.
  3. 09:18 UTC · resolved
    This incident has been resolved.
Source
Nov 25, 202533mSEV 2
High pusher, pubnub and mqtt adaptor error rateAbly's pusher, PubNub and MQTT protocol adaptors had a high error rate. It began in eu-west-1 at 11:25 UTC, then spread globally. EU regions were affected for 36 minutes, US for 15 and AP for 14. The provider identified the issue and applied a fix, and error rates dropped by 12:04 UTC.AI-graded · Ably Pub/Sub Channels - Asia-Pacific (Singapore) +6
Why this grade
  1. RULES · SEV 2
    Sev 2 because Ably Pub/Sub Channels - Asia-Pacific (Singapore) had degraded performance
  2. CLAUDE · SEV 2 · GLOBAL · MEDIUM CONFIDENCE
    Pusher, PubNub and MQTT adaptor requests saw elevated error rates across all regions for about 15 to 36 minutes; some requests failed.
  3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
4 updates
  1. 11:38 UTC · monitoring
    There was a high error rate with pusher, pubnub, and mqtt protocol adaptors in one region for a duration of 11 minutes from 11:25 UTC in the eu-west-1 (Ireland) region. This has now been resolved. No other region was affected
  2. 12:01 UTC · monitoring
    There is a high error rate for all translator protocol adaptors globally (pusher, pubnub and mqtt). We have identified the issue and are working on a fix.
  3. 12:04 UTC · monitoring
    We have resolved the issue and the error rate has now dropped across all affected regions. We are continuing to monitor.
  4. 12:11 UTC · resolved
    The incident is now resolved. The elevated protocol adaptor error rate affected EU regions for 36 minutes from 11.25am-12.01pm UTC, for 15 minutes in US regions from 11.50am-12.05pm UTC, and for 14 minutes in AP regions from 11.48am-12.02pm UTC.
Source
Oct 29, 202545mSEV 2
Webhook delivery failuresA new webhook feature deployment was incompatible with many existing webhook integration rules, causing elevated errors in webhook delivery on 29 October from about 08:58 to 09:42 UTC. The postmortem says impact began 28 October at 11:34 UTC. The fix was deployed at 09:26 UTC and the incident was resolved at 09:42 UTC.AI-graded · some customers · Integrations +1
Why this grade
  1. RULES · SEV 1
    Sev 1 because the provider reported data loss or exposure
  2. CLAUDE · SEV 2 · SUBSET · HIGH CONFIDENCE
    Reactor webhook deliveries to customer endpoints failed or errored for customers with webhook rules for about 45 minutes; other features were unaffected.
  3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
6 updates
  1. 08:57 UTC · investigating
    We are experiencing an elevated level of errors impacting Ably Reactor Webhook delivery. We have identified the issue and are currently deploying a fix.
  2. 08:58 UTC · identified
    The issue has been identified and a fix is being implemented.
  3. 09:21 UTC · identified
    We are continuing to work on a fix for this issue.
  4. 09:26 UTC · identified
    A fix has been deployed and Ably Reactor Webhook delivery service is recovering.
  5. 09:42 UTC · resolved
    This incident has been resolved.
  6. 09:16 UTC · postmortem
    # Summary of incident affecting the Ably production service on 28-29 October - investigation and conclusions ## Overview There was an incident affecting Ably's service between 28 and 29 October 2025, which affected the delivery of webhooks. The impact of the incident lasted from 1134 on 28 October to 0936 on 29 October \(all times UTC\). The incident occurred as a result of the deployment of new webhook functionality that was incompatible with many already-configured webhook rules. Now we have a good understanding of the underlying problem, the various factors that led to that having customer impact, and the effectiveness of our response in this case. ## Background Ably operates a number of production clusters for its service around the globe. There is a main production cluster that services the majority of customer accounts, plus a small number of dedicated clusters for specific accounts. Each of the clusters has a presence in multiple regions in a globally federated system. ### Ably integration rules Ably processes messages that are published on channels, and directs those to realtime subscribers on those channels, and also directs them to other targets as configured by integration rules. Multiple targets are supported for egress integration rules, including message queues \(Kafka, AMQP\) and generic HTTP endpoints \(webhooks\). Rules are configured via a control plane which supports: * direct interactive \("click-ops"\) configuration in the Ably dashboard; * a [Control API](https://ably.com/docs/api/control-api); * a [Terraform Provider](https://ably.com/integrations/ably-terraform-provider) which acts through the Control API. The source of truth for integration rule definitions is Ably's core "realtime" platform, and the other services \(the website and control API service\) obtain access to that via an internal provisioning API. Integration rule definitions comprise some attributes that are generic \(such as `channelFilter` that specifies which channels the rule applies to\), and some that are specific to each type of integration rule target \(such as the target endpoint, credentials, and any protocol-specific attributes\). Some generic attributes - ie attributes that are capable of having a meaning for multiple rule types - are not universally supported, since the evolution of rule features is based on adoption and priority. As a result, the schema for integration rule definitions contains multiple optional generic attributes as well as a wide range of type-specific attributes. One generic feature is an option that enables a custom set of trust anchors \(TLS certificates\) to be configured for a rule; this is to support integrations that target TLS endpoints that have a self-signed certificate rather than certificates issued by a public Certificate Authority \(CA\). This feature is used by customers having self-hosted endpoints, either as part of development/CI infrastructure, or as internally-facing production services where the operational effort of maintaining a certificate from a public CA is not justified. At the time of the incident custom trust anchors were supported for Apache Pulsar and Kafka rules, and support had just been added for webhooks. ### Ably's service deployment process Ably adopts a staggered deployment approach whereby each change is deployed gradually, minimising the adverse impact of any regression that might exist in new software versions having escaped all prior test phases. Changes are typically deployed on a weekly basis; usually in the main \(multi-tenanted\) cluster this means that changes are deployed to the ap-southeast-2 \(Sydney\) region first, and error rates monitored, before deploying to the remaining regions starting the following day. However, if a deploy needs to be expedited for any reason, subsequent regions can be deployed after a change has been in ap-southeast-2 for only a few hours. Most monitoring is automated; metrics are gathered for a wide range of service health metrics, including the rates of logged error events, and the rate of error API responses. When error metrics exceed a threshold then on-call engineers are paged to investigate. Since the majority of Ably's infrastructure comprises immutable EC2 instances, any deploy consists of deploying new instances with the new software version in parallel with retirement of superseded instances. For a normal deploy this process happens over the course of several hours so that the number of instances being retired at any given time is small, minimising any impact of the update on existing traffic. If a deploy needs to be accelerated, such as if making an emergency rollback, then deployment would more typically involve deploying a full complement of replacement instances immediately, followed by termination of old instances at a rate that reflects the urgency of the operation. Immediate and abrupt termination of multiple instances simultaneously leads to a greater risk of loss of ephemeral channel state, so even in accelerated rollbacks the preference is for old instances to be terminated gradually so that there is an orderly shedding of load so as to avoid any data loss. ## As it happened A change to add support for custom trust anchors was implemented, and this change was part of a set of changes deployed in a staggered deployment starting on 28 October. This happened in ap-southeast-2 from 1134 and, since this change set also included a fix for an unrelated problem, it was expedited and started in other regions from around 1500. Starting at 1940 on 28 October there was a handful of customer reports of a problem with webhook delivery. There were two independent customer reports of issues on 28 October and a handful of reports overnight. At 0731 on 29 October Ably's customer support team escalated the problem reports to Ably's engineering team for investigation. At 0745 an incident was declared, on-call engineers were paged and investigation started. The problem was determined to be a regression introduced in the previous day's deploy. At 0746 a rollback process was initiated to revert the changes. Rollback was initiated first in ap-southeast-2. At 0811 rollback was initiated for all regions and dedicated clusters that had received the changes. The rollback completed at 0936 and errors dropped to zero. ## Investigation The investigation aimed to understand: * the cause of failed webhook invocations; * how Ably's testing had failed to detect the regression; * how Ably's monitoring had failed to detect the regression; * the effectiveness of the response, including the time taken to respond following the first customer problem reports. The conclusions of the investigation are summarised below. ### The cause of failed webhook invocations The new code that caused the problem was adding support for custom trust anchors for webhooks. This code would look in the rule configuration for a configured set of trust anchors and, if present, use those instead of the system set of trust anchors for public CAs. The problem was caused by the fact that a significant number of preexisting rule definitions already had an empty set of trust anchors configured. The new code then used that empty set instead of the default set and failed to verify the certificate chain for any TLS endpoint. These failures were logged internally with a severity that classifies the event as a client configuration error rather than a system error. The presence of the empty trust anchor set in existing rule definitions was due to a long-standing behaviour in the web \(dashboard and control API\) services that create and update rules via the realtime platform's provisioning API. When custom trust anchor support was first added for Pulsar rules \(in November 2021\), the set of custom trust anchors became a generic rule attribute; but the web service would configure an empty set instead of a true zero value when custom trust anchors were unset. This meant that any rules that had been created since November 2021, even for types that did not initially support custom trust anchors, would have an empty set configured. Remedial actions identified in this area are as follows. **Fix the immediate regression.** As an interim behaviour, interpret an empty custom trust anchor array as if custom trust anchors are unset \(and so use the default trust anchors\). **Correct web service behaviour.** Update the web service so that a true zero value is set instead of an empty array when custom anchors are not set. **Correct the configuration of existing rules.** Perform a database update for all existing rule definitions that have an empty array configured. ### The failure of tests to detect the regression The relevant tests are a combination of unit tests and integration tests within the test suite of the realtime platform. Unit test coverage includes test coverage of all webhook configuration options \(and new tests were included with the deployed change\), plus tests for different endpoint behaviours \(HTTP 1 vs HTTP/2 for example\). Unit tests mock different parts of the surrounding code and system in order to have tests that are as targeted as possible. Integration tests include full end-to-end tests that create integration rules via the provisioning API that point to a range of endpoints existing as part of the CI system. The reason that none of these tests identified the problem is that none used the configuration in the same way as the web service; in particular, any rule that did not have custom trust anchors would be configured with a null entry. There was no expectation in the tests that an empty set should be interpreted as having no custom configuration. There was no integration test coverage of the end-to-end behaviour when rules are created via the web service. Remedial actions identified in this area are as follows. **Improve web service testing of integrations.** This should at least include: * create web service unit tests that verify that an unset custom trust anchor configuration results in a zero value being configured. * create web service integration tests that verify end-to-end behaviour involving creation of rules via the web service, with and without custom trust anchors set. ### The failure of monitoring to detect the regression Ably monitors a wide range of service metrics and alerts relevant engineering teams when those metrics indicate a service problem. Monitored metrics include errors logged internally whilst servicing customer traffic, the rate of error responses to customer requests, and the rate of errors or failures in traffic that is generated by Ably specifically for service monitoring. For this last category, Ably operates a series of "quality of service", or QoS, traffic generators that exercise a range of service functionality, so certain classes of failure relating to deployment can be identified that would not be identifiable via CI. Errors - both those logged internally and customer-visible error responses - can be classified broadly into: * bad requests - that is, operations that fail because the arguments or parameters of the request are invalid. When these errors are associated with an error code, it is typically in the 4xx range - these include general bad requests, plus authentication and authorisation failures; * internal errors - that is, errors that arise performing some operation that ought to succeed, but fails because of an issue \(eg an unexpected exception, a failure to communicate over the network, or an assertion failure\). When these errors are associated with an error code, it is generally in the 5xx range. A high rate of internal errors - either in our internal logs, or in customer-visible responses - is an indication of a problem with the service. A high rate of bad requests is more typically a problem on the customer side; an application is attempting operations that cannot succeed. All 5xx \(internal error\) metrics are monitored and alerts are generated when they exceed a threshold indicating a problem. Many 4xx \(bad request\) metrics are monitored but in most situations these do not generate alerts. This is for several reasons: * most are not an indication of a system problem; there is simply some invalid operation being attempted by a client; * all public endpoints are subject to a very high rate of spurious requests that fail authentication and authorisation checks, so the general rate of 4xx errors is high. Similarly, problems with invocation of a webhook \(or other rule target\) are usually a result of misconfiguration of the rule, or unavailability of the target endpoint, and these are classified with a 4xx status code. There is a material background level of such errors. Given the very high rate of background 4xx errors on external endpoints, it is generally difficult to identify any pattern on those endpoints that is indicative of a system problem. Notwithstanding this, all Ably Enterprise accounts have alerts set up both for 4xx and 5xx error metrics. 4xx alerts do not generally page an engineer out-of-hours, but both engineering and customer success teams get notified when there is an unexpected elevated rate of 4xx errors for an Enterprise customer. The configuration of those accounts within Ably's monitoring and alerting systems is added when that customer is onboarded as an enterprise customer. In this incident the overall increase in rule invocation failures was not significant in comparison with the normal ambient background error rate so, based on those metrics, there was no clear signal of there being a problem, and it was not sufficient to cross the alerting threshold on any existing enterprise accounts. Many customers with the highest usage volumes had rules created before November 2021, which were unaffected by the problem. At the time of the incident, QoS monitoring simulated traffic for some integrations but not webhooks. Remedial actions identified in this area are as follows. **Improve the ability in metrics and alerts for integration rules to discriminate between Ably problems and end-user problems.** Although most errors are a result of configuration problems, or problems in end-user infrastructure, it is still clearly possible for Ably regressions to also generate errors. We need to have more intelligent metrics and alerts that identify such occasions without creating an unsustainable number of false-positives. One possibility is to create alerts based on individual metrics labelled by rule type, to ensure that an increase in error rate for an individual rule type is alerted, even if that increase is not a material fraction of errors across all rule types. Another possibility is to correlate error levels with deploy events, or with software versions, so if a variation in error rate is correlated with a code change it is more likely to trigger an alert. **Add webhook tests to QoS monitoring.** Continuous testing in a production setting of simulated traffic should be extended to include webhooks. ### The effectiveness of the response From the first customer impact of this issue to the final resolution was 22 hours. This is far too long. Ably's response in this incident led to delays at multiple stages: * from the time of the problematic code being deployed to the first reported issue was around 8 hours; * from the first report to raising an incident was around 12 hours; * the full rollback, from the time that an incident was declared, was nearly 2 hours. Of course, had there been suitable alerting for the production problem we would have reacted sooner; but this timeline reflects the fact that even with the information we did have, the response was slow. This was due to a combination of factors: * the reports received were all received outside of business hours and, given that the customers raising those reports were not enterprise customers, there was no out-of-hours process to review them; * there was some delay before the support team correlated multiple reports to conclude that there was a problem with the delivery of webhooks; * the rollback process was comparatively slow, deliberately avoiding a more disruptive abrupt rollback since the issue had impact for a relatively small number of customers. The remedial actions in this area are as follows. **Review processes for handling and escalating issue reports out-of-hours.** Customers on lower support tiers can legitimately expect slower response times to deal with any individual issues or questions they have, but when an Ably problem arises that affects them there is no means for the relevant response team to be made aware. We will review our processes with the aim of improving support in these situations. ## Conclusion The service issues were the direct result of a regression in the Ably service - specifically, an incompatibility between a new feature \(custom trust anchors for webhooks\) and the existing state of rule configurations. The cause of the extended duration of the incident was that the classification and rate of failed delivery events resulted in no alert being generated, which meant that Ably was not aware of the ongoing problem until it was reported by customers. Although it is difficult to monitor 4xx error rates effectively, because of the high background level of such errors, there is clearly a need to improve on the current situation where Ably regressions can go undetected. There was significant additional delay in triggering a response because the reports were received on a support channel that was not monitored out of business hours. Once alerted to the problem, Ably initiated the response quickly, but the operational nature of the rollback meant that it took nearly two hours until it was completed. We take service continuity very seriously, and this incident represents another opportunity to learn and improve the level of service we are committed to providing for our customers. We are sorry for the disruption that was experienced in this instance, and are committed to identifying and implementing remedial actions so that we eliminate the risk of similar failures in the future.
PostmortemSource
Oct 9, 20250mSEV 3
Ably website, customer dashboard & Control API issuesWebsite, customer dashboards and Control API had connectivity issues; realtime services were stated as unaffected.AI-graded · Ably Website
Why this grade
  1. RULES · SEV 4
    Sev 4 because Ably Website had a partial outage
  2. CLAUDE · SEV 3 · UNCLEAR · LOW CONFIDENCE
    Website, customer dashboards and Control API had connectivity issues; realtime services were stated as unaffected.
  3. Claude Haiku 5.5 · prompt v8 · Oct 23, 2026 · Raw record
1 update
  1. 14:10 UTC · resolved
    Customers may be experiencing connectivity issues affecting our website (ably.com), customer dashboards, the Control API (control.ably.net), the Ably blog and documentation. Our realtime services are not affected by this issue at this time.
Source

Peers, 90 days

Compare
MailerSendGrade A+100.00% up0 incidentsMTTR –LoopsGrade A+100.00% up1 incidentMTTR 1h 34mKindeGrade A+99.99% up2 incidentsMTTR 1h 12mPaddleGrade A+100.00% up3 incidentsMTTR 47mCloudinaryGrade A99.97% up5 incidentsMTTR 1h 47mAblyGrade A99.97% up7 incidentsMTTR 7mMailgunGrade A99.96% up9 incidentsMTTR 42mPusherGrade A99.96% up2 incidentsMTTR 3h 45mResendGrade B99.86% up25 incidentsMTTR 24mInngestGrade B99.75% up14 incidentsMTTR 43mMapboxGrade C99.91% up2 incidentsMTTR 12h 38mClerkGrade C99.88% up10 incidentsMTTR 3h 26mWorkOSGrade C99.43% up21 incidentsMTTR 1h 05mStytchGrade C99.79% up7 incidentsMTTR 57mPlaidGrade F98.14% up18 incidentsMTTR 3h 38mTwilioGrade F96.38% up56 incidentsMTTR 7h 26m

README badge

Ably's live grade, for your README.

Ably grade badge
[![ably on downagain.dev](https://downagain.dev/badge/ably.svg)](https://downagain.dev/p/ably)