Skip to content

Commit 78691df

Browse files
Add Emergency Rescue investigation page
Add a troubleshooting page that explains what an Emergency Rescue is, why it shows up in the Control Panel logs and livelog, and how to use the incident files and PHP-FPM log to find out what caused it. Link to it from the 24/7 Monitoring and Alerting page.
1 parent 3282295 commit 78691df

2 files changed

Lines changed: 79 additions & 1 deletion

File tree

‎docs/about-hypernode/support/24-7-monitoring-and-alerting.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ At Hypernode we understand like no other that the availability and stability of
1717

1818
## Monitoring
1919

20-
It is our responsibility to monitor all the offered services on the Hypernode. As such we have setup a global monitoring network that checks your Hypernode’s availability from locations in Europe, the United States, and Eastern Asia. Each location checks the availability roughly every 2 minutes. An alert is raised if too many errors are received globally, and an Emergency Rescue is started. If the automated Emergency Rescue can’t fix the issue, the alert is escalated to the Hypernode Emergency Recovery Operative (Hero).
20+
It is our responsibility to monitor all the offered services on the Hypernode. As such we have setup a global monitoring network that checks your Hypernode’s availability from locations in Europe, the United States, and Eastern Asia. Each location checks the availability roughly every 2 minutes. An alert is raised if too many errors are received globally, and an Emergency Rescue is started. If the automated Emergency Rescue can’t fix the issue, the alert is escalated to the Hypernode Emergency Recovery Operative (Hero). See [Emergency Rescue on Hypernode and How to Investigate It](../../troubleshooting/performance/emergency-rescue-on-hypernode-and-how-to-investigate-it.md) for what a rescue does and how to find out what caused it.
2121

2222
### What Do We Monitor
2323

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
myst:
3+
html_meta:
4+
description: Learn what an Emergency Rescue on Hypernode is, why it shows up in
5+
your logs, and how to use the incident files to find out what caused it.
6+
title: Emergency Rescue on Hypernode and How to Investigate It | Hypernode
7+
---
8+
9+
# Emergency Rescue on Hypernode and How to Investigate It
10+
11+
An Emergency Rescue is started automatically when our global monitoring sees too many errors on your Hypernode. It tries to fix the problem without anyone having to act, usually by restarting services that fail under heavy load. If it can't, the alert goes to the Hypernode Emergency Recovery Operative (Hero). See our guide on [24/7 monitoring and alerting](../../about-hypernode/support/24-7-monitoring-and-alerting.md) for how this works.
12+
13+
You do not need to do anything when a rescue is started.
14+
15+
## Why It Shows Up With Your Name
16+
17+
In the logs of the Control Panel you may see "Emergency Rescue" entries with your name, or the name of the account owner, and an IP of `None` with source `Hypernode Events`. Nobody logged in. The entry is created by the automated system and uses the account owner's name for the audit trail.
18+
19+
## The Livelog Line
20+
21+
In `livelog` or the Control Panel logs you may see a line like `restore from backup <appname>`. This is how an Emergency Rescue shows up. It is an action by the platform. Nothing in the Control Panel or in your shop started it, and it does not mean an attacker or a team member did something.
22+
23+
The line says nothing about your data. To check that nothing was rolled back, look at the data itself. For example, an order placed after the time of the rescue should still be there.
24+
25+
## What It Does and Does Not Look At
26+
27+
- It checks if your Hypernode services (Nginx, MySQL, Redis, Varnish and RabbitMQ) are available and responding.
28+
- It does not look at your application. It does not trigger on slow PHP requests, so it can't be set to start at a PHP response time of, for example, 4 seconds. Use an APM tool such as [Tideways](../../best-practices/performance/how-to-set-up-tideways.md), [Blackfire](../../best-practices/performance/how-to-use-blackfire-on-hypernode-to-find-performance-issues.md) or [New Relic](../../best-practices/performance/how-to-find-your-performance-bottleneck-with-new-relic.md) for that.
29+
- A rescue can start without needing to change anything. This happens when the errors are gone again by the time it checks.
30+
31+
## Services Restarting Without a Crash
32+
33+
After a rescue you may see a service, for example MySQL, stop and start again in the system logs without an out-of-memory kill or a crash. This is the rescue restarting it. Several rescues in a row point to overload that lasts, often caused by bots.
34+
35+
## Find Out What Caused It
36+
37+
For every rescue, Hypernode saves a snapshot of the state of your Hypernode in a folder named after the time of the rescue:
38+
39+
```console
40+
app@abcdef-example-magweb-cmbl:~$ ls /data/web/incidents
41+
```
42+
43+
```{important}
44+
The folder names are in UTC. Convert the time shown in the Control Panel to UTC first, or you will open the wrong folder.
45+
```
46+
47+
Each folder contains files like these:
48+
49+
- `fpm-status`: the PHP-FPM workers and what they were doing. `BUSY` workers show requests that were running.
50+
- `mysql-process-list`: the MySQL queries that were running.
51+
- `process-list`: the running processes.
52+
- `tcp-connections`: the open network connections.
53+
54+
### Most Common Cause: PHP-FPM Is Full
55+
56+
Most rescues we see are caused by all PHP-FPM workers being busy. New requests then have to wait, and the monitoring sees errors. Check the PHP-FPM log:
57+
58+
```console
59+
app@abcdef-example-magweb-cmbl:~$ grep "reached pm.max_children" /var/log/php-fpm/php-fpm.log
60+
```
61+
62+
If you see lines here shortly before the rescue, the workers were all in use. The usual causes are:
63+
64+
- Bots and crawlers requesting uncacheable pages, such as layered navigation filter URLs. See our guide on [bots and crawlers](../../best-practices/performance/how-to-fix-performance-issues-caused-by-bots-and-crawlers.md).
65+
- Scanners requesting hundreds of paths that don't exist. See our guide on [blocking spammers and scanners](../../hypernode-platform/nginx/how-to-block-spammers-and-scanners.md).
66+
- An integration that requests your REST API very often.
67+
- Slow requests or slow database queries in your own code. An APM tool helps to find these.
68+
69+
To deal with unwanted traffic, work in this order:
70+
71+
1. Return `410 Gone` to bots on layered navigation query strings. See [How to Fix Performance Issues Caused by Bots and Crawlers](../../best-practices/performance/how-to-fix-performance-issues-caused-by-bots-and-crawlers.md).
72+
1. [Rate limit](../../hypernode-platform/nginx/how-to-resolve-rate-limited-requests-429-too-many-requests.md) the crawler that causes most of the requests.
73+
1. Filter the rest with [Botstopper](../../hypernode-platform/botstopper/how-to-use-botstopper.md).
74+
1. Only then look at capacity.
75+
76+
Check the access log afterwards to confirm the `410` and `429` responses are given.
77+
78+
The number of PHP workers is fixed by your plan. If your shop needs more headroom after you have dealt with unwanted traffic, a larger plan gives you more.

0 commit comments

Comments
 (0)