What Running a Bot on 2,000+ Discord Servers Taught Us

A Discord bot on three servers and the same bot on over two thousand are not the same program at a larger size. A handful of assumptions that are completely harmless in the small become, past a certain size, exactly the bugs that are hardest to find.

Here are four that cost us real outages.

1. Failures do not spread evenly, they hide

AlphaCloud runs split across several processes, called shards. Discord decides which server lands on which shard, and each shard only knows its own servers.

That produces a class of failure that looks like randomness from outside: welcome messages work on some servers and not on others. No error in the log, no alert, nothing. The cause was a single network call without a timeout. If the other side accepts the connection and then goes quiet, the code waits for the remaining lifetime of the process. Everything after it never happens, and because no error is thrown, nothing reports an error.

The lesson is unspectacular and universal: every outgoing call needs an upper bound. A call that waits forever is worse than one that fails, because failing is at least visible.

2. A restart is not a fix, it is a deadline

While we did not know the cause, restarting the affected shard reliably helped. That is precisely the problem with restarts: they work, and that removes the pressure to find the cause.

We now treat "does a restart help?" as a diagnosis, not a remedy. If it does, the broken state lives in memory, and there is a place in the code that builds it up and never tears it down.

The converse holds too: anything that has to survive a restart belongs in the database. If a multi-step operation is running, say a role change across thousands of members, and the process dies mid-way, the operation must not simply vanish. It has to be detected and continued on the next start.

3. Discord's limits apply per bot, not per process

Discord limits how many requests a bot may make per second. The obvious assumption: more processes, more throughput. Wrong. The limit is on the bot, not the process. Three shards each with a full budget add up to three times the allowance, and Discord then throttles all three.

So each of our processes works out how many exist in total and takes only its share. That sounds obvious, but it was not, and the bug stayed invisible for a long time because every individual process looked perfectly well behaved on its own.

A related trap: recurring jobs run on fixed intervals, say every five minutes. If all shards start at the same time, they fire their jobs at the same instant forever. Since then every process gets a random offset at startup.

4. What people actually configure is not what you expect

The technical part is one half. The other: for most servers, the default is the setting. A module whose defaults are wrong is misconfigured on the vast majority of servers, however good the settings page is.

Two things follow for us. First, every default costs more thought than the setting itself. Second, the better question for a new module is not "what can be configured here" but "what does this module do when nobody configures anything, and is that right for a normal server".

That is also the reason behind a decision that sounds like the opposite at first: since v1.8.0, almost every message the bot sends can be freely designed. But required blocks like the reason in a warning DM or the category menu in a ticket panel can be moved and not deleted. Design freedom, yes, but not to the point where a server accidentally builds something that looks like it works and does not.

The common thread

Three of the four are the same thought in different costumes: at this size, a silent failure is more expensive than a loud one. A crash gets reported and fixed. A feature that simply stops doing anything can stay that way for months, and you find out through a bug report from somebody who took the trouble to write.

So the question we now ask first about new code is not "does this work", it is "what does it look like when it does not, and would we notice".