|
| 1 | +--- |
| 2 | +myst: |
| 3 | + html_meta: |
| 4 | + description: Learn what an Emergency Rescue on Hypernode is, why it shows up in |
| 5 | + your logs, and how to use the incident files to find out what caused it. |
| 6 | + title: Emergency Rescue on Hypernode and How to Investigate It | Hypernode |
| 7 | +--- |
| 8 | + |
| 9 | +# Emergency Rescue on Hypernode and How to Investigate It |
| 10 | + |
| 11 | +An Emergency Rescue is started automatically when our global monitoring sees too many errors on your Hypernode. It tries to fix the problem without anyone having to act, usually by restarting services that fail under heavy load. If it can't, the alert goes to the Hypernode Emergency Recovery Operative (Hero). See our guide on [24/7 monitoring and alerting](../../about-hypernode/support/24-7-monitoring-and-alerting.md) for how this works. |
| 12 | + |
| 13 | +You do not need to do anything when a rescue is started. |
| 14 | + |
| 15 | +## Why It Shows Up With Your Name |
| 16 | + |
| 17 | +In the logs of the Control Panel you may see "Emergency Rescue" entries with your name, or the name of the account owner, and an IP of `None` with source `Hypernode Events`. Nobody logged in. The entry is created by the automated system and uses the account owner's name for the audit trail. |
| 18 | + |
| 19 | +## The Livelog Line |
| 20 | + |
| 21 | +In `livelog` or the Control Panel logs you may see a line like `restore from backup <appname>`. This is how an Emergency Rescue shows up. It is an action by the platform. Nothing in the Control Panel or in your shop started it, and it does not mean an attacker or a team member did something. |
| 22 | + |
| 23 | +The line says nothing about your data. To check that nothing was rolled back, look at the data itself. For example, an order placed after the time of the rescue should still be there. |
| 24 | + |
| 25 | +## What It Does and Does Not Look At |
| 26 | + |
| 27 | +- It checks if your Hypernode services (Nginx, MySQL, Redis, Varnish and RabbitMQ) are available and responding. |
| 28 | +- It does not look at your application. It does not trigger on slow PHP requests, so it can't be set to start at a PHP response time of, for example, 4 seconds. Use an APM tool such as [Tideways](../../best-practices/performance/how-to-set-up-tideways.md), [Blackfire](../../best-practices/performance/how-to-use-blackfire-on-hypernode-to-find-performance-issues.md) or [New Relic](../../best-practices/performance/how-to-find-your-performance-bottleneck-with-new-relic.md) for that. |
| 29 | +- A rescue can start without needing to change anything. This happens when the errors are gone again by the time it checks. |
| 30 | + |
| 31 | +## Services Restarting Without a Crash |
| 32 | + |
| 33 | +After a rescue you may see a service, for example MySQL, stop and start again in the system logs without an out-of-memory kill or a crash. This is the rescue restarting it. Several rescues in a row point to overload that lasts, often caused by bots. |
| 34 | + |
| 35 | +## Find Out What Caused It |
| 36 | + |
| 37 | +For every rescue, Hypernode saves a snapshot of the state of your Hypernode in a folder named after the time of the rescue: |
| 38 | + |
| 39 | +```console |
| 40 | +app@abcdef-example-magweb-cmbl:~$ ls /data/web/incidents |
| 41 | +``` |
| 42 | + |
| 43 | +```{important} |
| 44 | +The folder names are in UTC. Convert the time shown in the Control Panel to UTC first, or you will open the wrong folder. |
| 45 | +``` |
| 46 | + |
| 47 | +Each folder contains files like these: |
| 48 | + |
| 49 | +- `fpm-status`: the PHP-FPM workers and what they were doing. `BUSY` workers show requests that were running. |
| 50 | +- `mysql-process-list`: the MySQL queries that were running. |
| 51 | +- `process-list`: the running processes. |
| 52 | +- `tcp-connections`: the open network connections. |
| 53 | + |
| 54 | +### Most Common Cause: PHP-FPM Is Full |
| 55 | + |
| 56 | +Most rescues we see are caused by all PHP-FPM workers being busy. New requests then have to wait, and the monitoring sees errors. Check the PHP-FPM log: |
| 57 | + |
| 58 | +```console |
| 59 | +app@abcdef-example-magweb-cmbl:~$ grep "reached pm.max_children" /var/log/php-fpm/php-fpm.log |
| 60 | +``` |
| 61 | + |
| 62 | +If you see lines here shortly before the rescue, the workers were all in use. The usual causes are: |
| 63 | + |
| 64 | +- Bots and crawlers requesting uncacheable pages, such as layered navigation filter URLs. See our guide on [bots and crawlers](../../best-practices/performance/how-to-fix-performance-issues-caused-by-bots-and-crawlers.md). |
| 65 | +- Scanners requesting hundreds of paths that don't exist. See our guide on [blocking spammers and scanners](../../hypernode-platform/nginx/how-to-block-spammers-and-scanners.md). |
| 66 | +- An integration that requests your REST API very often. |
| 67 | +- Slow requests or slow database queries in your own code. An APM tool helps to find these. |
| 68 | + |
| 69 | +To deal with unwanted traffic, work in this order: |
| 70 | + |
| 71 | +1. Return `410 Gone` to bots on layered navigation query strings. See [How to Fix Performance Issues Caused by Bots and Crawlers](../../best-practices/performance/how-to-fix-performance-issues-caused-by-bots-and-crawlers.md). |
| 72 | +1. [Rate limit](../../hypernode-platform/nginx/how-to-resolve-rate-limited-requests-429-too-many-requests.md) the crawler that causes most of the requests. |
| 73 | +1. Filter the rest with [Botstopper](../../hypernode-platform/botstopper/how-to-use-botstopper.md). |
| 74 | +1. Only then look at capacity. |
| 75 | + |
| 76 | +Check the access log afterwards to confirm the `410` and `429` responses are given. |
| 77 | + |
| 78 | +The number of PHP workers is fixed by your plan. If your shop needs more headroom after you have dealt with unwanted traffic, a larger plan gives you more. |
0 commit comments