Keeping an Eye on 4,000 Discord Servers: Grafana, PostHog and What They Showed Us

AlphaCloud runs on a little over 4,000 Discord servers, spread across four shards. Those are four processes, each serving a part of the servers, all talking to the same API.

At that size a problem rarely shows itself by breaking something. Far more often everything works, just more expensively, more slowly or more noisily than it should. That is why we rebuilt our monitoring in September 2026. This article shows what it looks like and what it showed us in the first weeks.

Two tools, two questions

Grafana answers the question: what is the bot doing? Bot and API report metrics to Prometheus (requests, response times, error rates, Discord responses, AI usage), and the logs go to Loki. Every server node has its own stack, and Grafana is only reachable through a private network, not from the internet.

PostHog answers the question: what do people experience on our websites? Page views, browser errors, load times and, with consent, recorded sessions. PostHog only starts after you agree in the cookie banner.

The split is deliberate. Grafana looks at machines, PostHog at interfaces. Squeezing both into one tool gets you either user data in the server logs or server load in the product analytics.

What it found

A loop that produced half of the traffic

Polls with an end time are closed by a background task that checks every minute whether anything has expired. It did that for every server individually. With roughly 1,000 servers per shard and four shards, that came to about 70 requests per second, almost all of them answering "nothing expired".

We only noticed in Grafana, once we broke the bot's requests to the API down by path: this one endpoint was more than half of all traffic. Everything worked. It just carried a load nobody had asked for.

Now each shard asks once a minute: "Which servers have an expired poll at all?" and only fetches details for those. Roughly 4,200 requests per minute became about four.

Three million times "not set up"

Same pattern, different place: several modules (support entries, absences, team list, bot stats, newsletter) regularly read every server's settings in their loops. On most servers a given module is not set up at all, so the answer was "not found". About three million times a day.

The fix is unspectacular: a "not set up" is now remembered for 30 minutes. That is only safe because each of these modules sends the bot an update the moment it is saved in the web panel, so setting a module up never means waiting 30 minutes.

Discord's outages are not our bugs

Our team error channel posted every failed action as a "Runtime Error", with a unique reference and everything. That included moments when Discord itself answered with a server error. That is neither a bug on our side nor something we can fix, and it sat in the channel looking exactly like a real defect.

Today it is logged as a warning, and the member is told to try again in a few seconds instead of being offered a link to file a bug report. An error channel where half the messages are not errors eventually stops being read. Then you miss the real one.

An attack that lasted one minute

One afternoon at the end of September our home page received a little over 500,000 requests within one minute, from many countries at once. Most of it was stopped before it reached our servers, and a minute later traffic was back to normal.

We did not see it because something went down. We saw it because the number did not fit the rest of the chart. That is the actual job of monitoring: not raising the alarm when everything is on fire, but showing where something does not fit normal.

What our monitoring deliberately does not see

The rules here matter as much as the dashboards:

• No visitor IP addresses in logs. Our privacy policy promises that, so we remove them in two places: when the access log is written and once more before it is stored.
• No server, user or message IDs in metrics. Every metric is only split by coarse attributes (module, status class, shard). That protects data and keeps the metrics database small.
• Session recordings show our interface, not your content. Only text that comes word for word from our translation files stays readable. Server, channel and user names, inputs and previews are masked. If the list of allowed texts cannot be loaded, everything is masked.
• Everything on the website only after consent. Which also makes the numbers there a lower bound. For real page counts we look at the anonymous totals from our network provider.

The details are in the privacy policy.

What we take from it

The expensive thing is rarely the broken thing. None of the findings above produced an error. They produced load, cost and noise, and without measuring you never see any of that.

Asking per server does not scale. Both big findings were the same pattern: a loop over every server that asks about each one individually. The answer is almost always the same question, asked once for all of them.

An alert has to be rare or it is not one. That is why only Grafana pings the team, and only for things that need someone to step in. New error types from PostHog land in the channel without a ping.

If you run a Discord server yourself and want to see the modules mentioned here: polls, support entries and absences are in the web panel, and they are described in the documentation.