Support Kubernetes 1.34+ on DigitalOcean

[Exairnous]

What?

Adds a memory request/limit to Haproxy.

Why?

Starting with Kubernetes 1.34, DigitalOcean kills Haproxy (and also Reticulum and Photomnemonic, for some reason) shortly after startup because it thinks it's using too much memory. Adding the limit to Haproxy seems to fix everything (possibly because it increases the QoS Class of the pod from "BestEffort" to "Burstable", though I'm not sure why this fixes the Reticulum and Photomnemonic pods as they're still "BestEffort"). The extremely low request and extremely high limit should essentially preserve the previous behavior and make it so it just uses whatever memory it needs (which is normally around 30-40 Mi to begin with).

Examples

N/A

How to test

1. Regenerate your hcce.yaml file with npm run gen-hcce (remember to create a backup of hcce.yaml first if you have customized it).
2. Comment out the - --default-ssl line in hcce.yaml
3. Run these commands to completely restart your instance:
   • kubectl delete deployment --all -n hcce
   • kubectl delete pods --all -n hcce
   • npm run apply
4. Visit your instance and see that everything works.
5. If you are using DigitalOcean and/or on a version of Kubernetes before 1.34, you should see no difference with these changes.
6. If you are using DigitalOcean and/or on a version of Kubernetes of 1.34+, you should see that without these changes, your instance won't work and will be stuck with pods in a CrashLoopBackOff state due to being continually killed for being out of memory; however, with these changes you should see that things work normally.

Documentation of functionality

This PR preserves the previous behavior, so no documentation update is needed.

Known limitations

This doesn't add memory requests/limits to Reticulum, Photomnemonic, or any other pods.

Alternative implementations considered

Adding more constrained memory requests/limits, but it is unclear how much memory Haproxy really needs, and the default example has it with a minimum of 2Gi (and this was apparently in response to a bug: haproxytech/kubernetes-ingress@ff82484). Since that's much higher than we usually see it with Hubs, giving it an essentially unlimited amount to preserve the previous behavior seemed better than more constrained settings.

Adding requests/limits to other pods, but this isn't required to fix the issue and it would be more complicated, and potentially require changes to allow auto-calculation based on a user provided value for the total memory of their instance (which would also necessitate a docs update).

Open questions

None.

Additional details or related context

The memory used by the Reticulum pod has increased dramatically (from beginning with a few hundred Mi to beginning with around seventeen hundred Mi), but it's unclear why.

[github-actions]

Thank you for the pull request.

ROADMAP STATUS: This pull request isn't currently on any roadmap. Updates will be conveyed here as its place on/off a roadmap changes.

You can view the roadmaps here: Roadmaps Google Drive folder.

For more information on how the roadmaps work, see our roadmaps policy on GitHub.

[DougReeder]

This doesn't break anything using versions
Client Version: v1.34.2
Kustomize Version: v5.7.1
Server Version: v1.33.12

so we could implement it as-is, but I'd like to set a memory resource request of 65Mi, as after restart + 2 users logged in, I see the HAProxy memory useage as 62Mi.

[Exairnous]

What advantage does increasing the minimum memory allocated have? Currently with 2 users logged into my instance HAProxy is using 58Mi.

update: actually about 2.5 users because Hubs Bot was connected

[DougReeder]

If you want to go with something like 60 MI, I won’t argue, but if we have a memory request, it should mean something.

[Exairnous]

Normally I would agree; however, in this case the only reason for the memory request is to work around the issues introduced with the Kubernetes update. Basically, the idea was for it to preserve the previous behaviour which was to have no requests/limits, so the meaning behind the values was to indicate these were dummy values (though perhaps it would have been better to make it have a request of 1Ki and a limit of 1Pi and include a comment).

It is true, though, that having an extremely low memory request does make it more likely that the pod would get evicted due to memory pressure (which for something like HAProxy we probably don't want), so to avoid that we would need to make the request a little bigger than the standard amount of memory HAProxy normally uses over time. So, potentially a value of 256Mi would be appropriate; however, we'd still need to use a dummy value for the limit to not artificially constrain it (which of course isn't great).

If I understand correctly, memory requests should ideally be added to all the pods to ensure they have enough memory for their normal operation and limits shouldn't be added unless there is some reason to (e.g. a memory leak); however, this isn't possible currently (we could set it for haproxy, but not for all of them - I tried and it wasn't happy).

Anyway, all of this got me to thinking that potentially this isn't the correct solution and that the correct solution is to get Reticulum's memory back down to normal and relieve the memory pressure so that we can have a proper request/limit setup for things. So, I did some investigation and I think I did find the underlying problem (see this Discord post: https://discord.com/channels/498741086295031808/1158476691384062012/1544236901840592916) and I no longer think that adding memory requests/limits is the correct way to solve it (we can always add those later, though, and they should work much better with this underlying issue solved).

So, I think I'm actually going to recommend closing this PR and possibly putting up a new one with the (more correct) workaround I mentioned in my Discord post.

[Exairnous]

Closing in favour of #406