How I Accidentally Hit My Own Server With A DDoS Attack

I hit the send button and waited on the summary page of my newsletter campaign. Suddenly, the site became extremely slow. I was immediately worried if something was wrong and checked my queue system, but I couldn't load this page. Next, my server was down. Neither my blog nor my newsletter software (living in the same app) wasn't responding anymore. I had no idea what's going on, and I also checked my Mastering PhpStorm course website; no response either. It lives on another server of mine. I was like: "Come on, are you kidding me?". Most important, in my newsletter, I was presenting this new course. That's a bad start.

It was the first time I was dealing with such a situation where I didn't know what was going on. I knew it was related to my newsletter, but I sent them from my server for a year now without any problems. I wanted to look at the logs on my server, but I couldn't reach them. So the only solution for me was to restart the server, which did not help.

After trying to figure out what was happening, my sites started to work again. Overall we are talking about a timeframe from just 20-30 minutes. It felt much longer to me, though. I was most afraid that not all emails were sent from my campaign, but it looked like all went out. That's good. Still, I'm pretty sure a lot of people weren't able to check my new course. This sucked!

A distributed denial-of-service (DDoS) attack is a malicious attempt to disrupt the normal traffic of a targeted server, service, or network by overwhelming the target or its surrounding infrastructure with a flood of Internet traffic. (Cloudflare)

What Has Happened

The first hint of what could have lead to the issue was given by Marcel Pociot. In his screenshot, he is showing the network tab of my course's website. At the bottom of the website, I have a counter for subscribers. I'm checking them every few seconds, and if there is a new one, I'm showing a little message. These API calls are usually pretty fast, but you can see they take forever in the screenshot.

So the first guess was, that my course's website sent too many API calls to my main website, where I have hosted my newsletter system. And since I'm checking the counter every three seconds, this can add up to many requests. After Marcel's tweet, I immediately removed polling from this counter. Additionally, Chris Fidao, the server guru of the Laravel community, pointed me to the right location to check some logs. Indeed, the server had trouble processing tasks. (requests maybe?):

WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

Still, I had some problems with this theory. A significant number of people would be needed simultaneously on my course website to produce that many requests. From my experience, many people do not open up newsletters right away. Also, the API calls were pretty small and fast.

What Has Really Happened

When I thought about what had changed since my last newsletter, I remembered one thing. I never set up email tracking for open and click events. In the last weeks, when I changed a few things, I fixed that. Meaning every time someone opens my newsletter or clicks a link in one of my emails, my email delivery service will send a request to my newsletter system so that I see those statistics for my campaign. That's nothing unusual. But that day, I realized the first time what that means. It means a lot of requests to my server.

Again, open and clicks are not events that happen at the same time. Mostly they are spread out and, also, shouldn't be an issue. When I check my database, I found over 4.000 entries in that webhook table. Now I was curious. I have around 3.000 subscribers, but the campaign's open rate was at 20% at that time. This doesn't sum up. Even if all those 20% also clicked a link, which they didn't, it would only be 1.600 entries.

Here comes the "Ahh 😲" moment. Besides click and open events, I have also subscribed to deliveries on the new system. Finally, it all made sense. 3.000 emails mean 3.000 delivery events in a short amount of time. This is something my simple server was not configured for. 💥

Learnings

All the mistakes you make have one thing in common: They teach you something new!

Besides all the stress and issues this event caused, I'm still thankful for the experience. As a developer, we tend to work with the tools we have and are used to. Many times we do not run into issues with our decisions, but sometimes we do. I'm glad this only happened to me and not to any of my clients.

Laravel Forge

I've been using Laravel Forge for many years now and love it. It makes it super easy to spin up new servers and connect them to your domains and applications. Besides having Forge handling many things for you, you now own a server, and you are responsible for updates, maintenance, security, etc.

I wasn't aware of the possible wave of requests my server could face. I could have enhanced it to deal with the amount.

Queues

Even though queues were in place to handle those outgoing and incoming tasks, it was too much for my server. I thought I was safe for little spikes in traffic due to the queue system.

Laravel Livewire

The real-time counter was a nice-to-have little gimmick for visitors on Mastering Phpstorm. First, I wanted to set up a WebSockets server to achieve real-time updates, but I thought it was too much overhead. That's why I decided to use the polling feature of Laravel Livewire. It was just so easy to set up that I didn't even think of if it could be a problem.

The more comfortable it becomes to use something, the more careful you have to make decisions. I would never blame any tools; it's always the developer's responsibility in the end.

Newsletters Are Hard

A year ago, I moved away from ConvertKit. My list was growing, and my monthly plan has become pretty expensive, with 75$/month. Especially if you only send one newsletter every 1-2 months.

Now a year later, I know exactly what you pay this money for. It's for all those things that might not happen to you but could, and like in my case, did. (IP ban list, IP reputation, domain reputation, servers, etc.)

Conclusion

I'm currently taking some actions to prepare myself better for these situations. I also unsubscribed from my emails' delivery events because I don't even use or need them. I'll probably write about my actions in detail in another post.

This was how I hit my server with a DDoS attack. Be better than me and try not to make the same mistakes as I did.