Lullops is an AI on-call engineer. When the 3am alert fires, it reads the logs and the last few deploys, finds the likely cause and opens the fix as a draft pull request. It wakes a person only for calls that need judgment.
14 days free. Read-only until you say otherwise.
Investigation
5xx error rate above threshold
14.2%of checkout requests failing. Threshold is 2%.
Paged first, per your escalation policy. Starting triage.
Logs checkout-api, last 15 min
2,418 lines read
412 of 418 errors share one stack trace: formatPrice at format.ts:88. Every one is a cart saved before May 2.
Deploys checkout-api, last 24 h
3 deploys
Trace of one failing checkout52 ms, 500
POST /v2/checkout
cart.load38 ms
pricing.formatPrice× 2 ms
payments.authorizenot reached
Traces agree: 97% of failing requests pass through formatPrice, which #4471 changed.
src/pricing/format.ts
+3 −1
import { money } from "./money" export function formatPrice(cart: Cart) { const currency = cart.currency return money(cart.total, currency.code) // Carts saved before May 2 have no currency. const code = currency?.code ?? cart.store.defaultCurrency return money(cart.total, code)} export function formatTotal(order: Order) {Test added and passing: “formats legacy carts without a currency”. Falls back the same way invoices already do.
Rolled back checkout-api from #4471 to #4470
Allowed by rule 4: “Roll back a deploy under 30 minutes old if 5xx stays above 5%.”
5xx rate, was 14.2% 0.3%
Guard legacy carts without a currency
Watching checkout-api for 15 minutes before I call it resolved.
incidents
14 members, 0 awake
Lullops is typing
LullopsApp3:14 AM
Checkout was failing for about 14% of customers from 03:10.
Cause: deploy #4471 reads currency.code, and carts saved before May 2 have no currency.
Done: rolled back to #4470 at 03:13 (runbook rule 4). Errors are back to 0.3%.
Next: the fix is in PR #4472 for @maya.r to review after coffee.
I didn't page anyone. Full timeline in INC-2291.
Status page updated: checkout degraded 03:10 to 03:13, now operational.
You slept through it.
Mitigated in 1 min 48 s. Nobody paged. The fix waits for review.
Sample workspace: Parcelry, a made-up checkout team. Replays at about five times real speed.
formatPrice, all from carts saved before May 2.formatPrice; errors began 94 seconds later. A trace of a failing checkout stops in formatPrice and never reaches payments.to
This is her sleep tracker, with Lullops on the pager. Flip the switch to see the same night without it.
Disk at 85% on worker-3
Cleared 40 GB of stale build cache, per runbook step 2.Priya wakes up, clears the cache and lies awake until 00:15.
Search health check flapping
Matched a known flaky check, muted it for 2 hours and filed a ticket.Priya acknowledges it from bed and takes twenty minutes to fall back asleep.
Checkout failing 14% of requests Replayed above
Rolled back #4471 in under two minutes. The fix is waiting in PR #4472.Priya finds the bad deploy at 03:51 and rolls it back at 04:02.
Email queue 9 minutes behind
Scaled workers from 4 to 6, inside the limit in lullops.yml.Priya scales it by hand, then watches the graph until 04:55.
TLS certificate expires in 9 days
Not urgent, so it filed a ticket for the morning.A 5:52am page for something due next week.
If a call is outside your runbooks, or it touches customer data or money, Lullops pages one person. The page says what it found and which of two fixes it would pick.
Needs a human Answered To Jonas K., primary on call
This is outside my runbooks, so I'm asking before I touch it.
orders_archive grew 38 GB since Monday. Backfill job #812 is still writing to it.You need read tokens for your alerting, logs and repo, plus your runbooks folder. Shadow mode starts on the next alert, and everything Lullops may do stays in a file you review like code.
Point it at your alerts, logs, traces, deploys and repos. It starts with read access to everything and write access to nothing.
Paste in the docs you already have, messy ones included. Lullops turns them into steps it can follow and flags the ones too vague to trust.
Decide what it may do alone, like rolling back a deploy, and what always goes to a person. You set this per service.
For a week it works every alert next to your on-call engineer, and you grade its calls. Change one word when you trust it.
lullops.yml
checkout-api
# What Lullops may do. Reviewed like code.
service: checkout-api
reads: [alerts, logs, traces, deploys, repo]
runbooks: ./runbooks/checkout.md
may:
rollback: { age: "< 30m", when: "5xx > 5%" }
restart: { max: 2, per: hour }
scale: { replicas: "+50%" }
never: [merge_prs, primary_db, billing]
page_a_human_when:
- confidence < 0.8
- customer_data_at_risk
mode: shadow # promote to: on_callNothing runs on your servers. Lullops connects through the APIs your tools already have, and anything that sends a webhook or speaks OpenTelemetry works too. Here's each tool it touched during INC-2291.
Here's the permissions screen for checkout-api, as maya.r left it two days ago, and what the audit log kept from INC-2291.
| Action | Setting |
|---|---|
| Read logs, traces and metrics | Allowed |
| Read code and deploy history | Allowed |
| Open draft pull requests | Allowed |
| Roll back a deploy under 30 min old | Allowed, logged |
| Restart pods, twice an hour at most | Allowed, logged |
| Change infrastructure config | Asks a human |
| Merge pull requests | Never |
| Touch the primary database | Never |
Every incident Lullops handles gets a write-up, with the timeline filled in from what it actually did. This is the one for last night's checkout outage.
docs / postmortems / INC-2291.md Draft
From 03:10:58 to 03:13:52, about 14% of checkout requests failed with a 500. Nobody was paged. Lullops rolled back the deploy that caused it and opened a fix for review.
formatPrice.#4471 reads currency.code. Carts saved before May 2 have no currency, so the field is undefined and formatPrice throws. Invoices already fell back to the store's currency; checkout didn't.
Parcelry and its people are made up for this concept site.
Releases go out on Tuesdays.
Lullops now follows a failing request across services before it blames one.
Test a runbook against last month's alerts before you let it act on tonight's.
Pages wait for each person's own time zone, holidays and leave.
A first draft lands in your docs by 9am, with the timeline already filled in.
Lullops works your next alerts next to your on-call engineer and shows its work. It stays read-only until you change that, and you won't need a card.