Trying to catch panics in Rust is best-effort only: see https://doc.rust-lang.org/std/panic/fn.catch_unwind.html . It's pretty strongly discouraged for general exception handling. You can check, but I have a feeling that stack unwinding requires some heap allocations, in which case an OOM panic will cause a double panic and force an abort, and thus cannot be caught.
The proper solution in Rust is to use the Result type.
As for what Linux does: it will invoke the "OOM Killer" which will kill user processes to clean up space.
"As for what Linux does: it will invoke the "OOM Killer" which will kill user processes to clean up space."
Which is also pretty catastrophic. Honestly, I'm not sure which I would prefer more: a debuggable kernel panic that would be easy enough to recover from, or a still-running but possibly lobotomized machine.
You can configure the OOM killer to simply panic on OOM. You can also control which processes are prioritized for OOM killing, as the sibling comment mentions.
Not mentioned in the article is that you can set per-process memory usage limits, where subsequent calls to malloc() will return NULL if trying to allocate more. Some applications will try to properly handle that condition, many will just segfault or do some kind of controlled abort. For a lot of applications failing is the right behavior on memory allocation failure. I mean, presumably you weren't trying to allocate that memory on a whim, you need that memory to function properly! So there really isn't any sane way to continue functioning without either disabling functionality or hanging until memory becomes available. Either way, I generally prefer something to unambiguously fail so I can restart it than have a "gray failure" where the system tries to keep limping along.
In a decent amount of cases this will kill the offending process taking up too much memory, and then your service manager (systemd etc) will restart it. If problem was due to a memory leak etc this typically recovers the situation for a while.
Of course the OOM killer might kill the wrong process, which can be annoying. This can be tweaked on a per process basis using /proc/$process/oom_score_adj
or using OOMScoreAdjust in the systemd service file
The proper solution in Rust is to use the Result type.
As for what Linux does: it will invoke the "OOM Killer" which will kill user processes to clean up space.