- Database upgrade triggered service disruption — recovery in progress – Incident details

All systems operational

Database upgrade triggered service disruption — recovery in progress

Resolved
Major outage
Started 14 days agoLasted about 2 hours

Affected

Authentication Service

Operational from 9:49 PM to 11:56 PM

Catalog Service

Operational from 9:49 PM to 11:56 PM

Checkout Service

Operational from 9:49 PM to 11:56 PM

Updates
  • Resolved
    Resolved

    Quick update on impact: throughout the disruption, our automatic checkout fallback system routed buyer traffic to the underlying Shopify checkout, ensuring buyers were able to continue placing orders. Some Shopflo-specific checkout features (upsells, loyalty point redemption at checkout, custom shipping logic) were unavailable during the fallback window, but the core transaction flow remained operational throughout.

  • Monitoring
    Monitoring

    Update: the checkout flow is back online. We have migrated merchant login, product catalog, and checkout services to replacement databases brought up from point-in-time backups, and these services are now operational. Most merchants should see checkout behaviour returning to normal.

    The remaining background services (settlements, loyalty, communications) are being brought online over the next few minutes.

    Our cloud provider's engineering team is continuing to investigate the original upgrade state separately; that work is no longer on the customer-facing path.

    We're keeping this incident open while we monitor for stability. We will post a final resolution update once we have confirmed clean behaviour across all services.

  • Identified
    Identified
    A scheduled minor-version database engine upgrade — initiated automatically during our cloud provider's maintenance window — entered a degraded state on three of our production databases, affecting merchant login, product catalog, and billing/settlement APIs. Our cloud provider's engineering team has been engaged via a Severity 1 case and is working on resolving the underlying upgrade state. In parallel, we have applied a configuration mitigation that has reduced the failure rate, and are bringing replacement databases online from point-in-time backups to fully restore service. Impact: - Merchant login (dashboard): intermittent failures and retries - Product catalog & storefront product data: intermittent slow / failed responses - Checkout: secondary impact when login or catalog is affected We expect recovery within 30–60 minutes. The next update will be posted in 20 minutes.