Skip to main content

Raymii.org Raymii.org Logo

Quis custodiet ipsos custodes?
Home | About | All pages | Cluster Status | RSS Feed

Limit specific process memory on desktop linux with cgroups and earlyoom

Published: 13-02-2021 | Author: Remy van Elst | Text only version of this article


❗ This post is over three years old. It may no longer be up to date. Opinions may have changed.


On my laptop I recently had trouble with out of memory issues when running clion, firefox, thunderbird, teams and a virtualbox VM. To combat this, I've setup cgroups to limit how much RAM specific applications can use and configured earlyoom, a very nifty tool that checks available memory and kills the process with the highest oom_score if available memory falls below 5%. Otherwise, my laptop would first grind to a halt (even without swap) and only after half an hour of seemingly being stuck would the OOM killer kick in. With earlyoom this hanging behavior is gone, although sometimes applications get killed when I don't expect it. I've given firefox, thunderbird and teams a cgroup with memory limit and clion and virtualbox use their own configuration to limit their RAM usage.This post details how to setup cgroups to limit memory of specific processes including automatically placing process inside a cgroup.

teams requirements

I'm using Microsoft Teams in this example, that abomination of a chrome-browser / glorified IRC client has hardware requirements stating at least 4 GB of RAM, and that is way too much for what it's worth. Even my java-based CLion IDE doesn't use as much memory as Microsoft Teams. I've now given it 2 GB RAM max and it works just fine. 1.5 GB RAM also works in my experience. I think Teams has a memory leak because it starts with around 600 MB in use, but after running for a few hours, it is up and over 1 GB, eventually being killed by the OOM-killer in the cgroup. After restarting it's back around 600 MB again.

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $200 credit for 60 days. Spend $25 after your credit expires and I'll get $25!

This post is tested on Ubuntu 20.04. As far as documentation tells me, it should work on Debian 10 and Ubuntu 18.04 as well, but I've not tested that. Documentation on cgroups is spread out and not coherent. When looking up documentation, make sure you know which cgroup version you use and for what cgroup version the guide is written. I try to avoid systemd-specific cgroup configuration (slices) since this post should also be applicable for non-systemd users. Therefore you'll see me poking around in /sys/fs/cgroup/ instead of using systemd tools.

I've ended up with the following RAM limits:

  • Teams: 2 GB
  • Firefox: 2 GB
  • Thunderbird: 1 GB
  • CLion: 4 GB
  • Virtualbox VM: 2 GB

About 11 GB reserved, leaving a bit for all other applications such as the desktop. This limited configuration runs for a few days and I haven't had the out of memory issues. Using munin and the multips plugin I can see that the processes stay inside their given limits. Here is the munin graph that shows the specific processes I monitor:

multips

java is CLion, and since the machine is not on all the time, there are gaps in the graph, but it is more than enough to give a general overview of usage. The default memory usage graph is fun to look at as well:

ram munin

Earlyoom, a more desktop friendly oom-killer

Starting off with earlyoom, as of Debian 10 and Ubuntu 18.04, available in the repositories, install it using the package manager:

apt install earlyoom

You don't have to do anything more, when your memory usage drops below 10% it will start killing processes, by default the one with the highest oom_score. In my case that was often teams, but sometimes it was firefox. I've changed the settings to only kick in when memory usage drops below 5% and have added a few processes which I'd rather not have killed. kwin is the KDE window manager, when that was killed my window borders were gone. Fixed by a kwin --replace, but annoying. VirtualboxVM killing could give disk corruption inside the VM, which is something I'd also rather avoid.

vim /etc/default/earlyoom

Contents:

# Print every 60 seconds, act if free memory comes below 5% and avoid killing KDE and virtualbox
EARLYOOM_ARGS="-r 60 -m 5 --avoid '(^|/)(kwin_x11|kwin|ssh|VirtualBoxVM)$'"

Limiting memory per process with cgroups

cgroups (abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes. It's the magic behind linux containers (lxc/docker) and I'm using it to make sure specific processes and their children cannot allocate more than a given amount of RAM. On Ubuntu you must install the following package:

apt install cgroup-tools

Create a cgroup, I named mine cgTeams:

sudo cgcreate -t remy:remy -a remy:remy -g memory:/cgTeams

Replace remy:remy by your username and group.

Set the maximum amount of RAM for the newly created cgroup. The calculation to bytes isn't required on modern linux, you could just enter 2048m but on older Debian systems you do need to specify the exact bytes:

echo $(( 2048 * 1024 * 1024 )) | sudo tee /sys/fs/cgroup/memory/cgTeams/memory.limit_in_bytes #2 GB RAM

If you have swap enabled, you can set a limit on that as well:

echo $(( 2049 * 1024 * 1024 )) | sudo tee /sys/fs/cgroup/memory/cgTeams/memory.memsw.limit_in_bytes #2GB swap, only works if you have swap

Launch Teams in the freshly created cgroup:

cgexec -g memory:cgTeams teams 

In the next few sections I'll discuss automatic creation of cgroups at boot and automatic placement of processes inside cgroups. If all you wanted was to give a specific process a RAM limit, not making it persistent, this is all there is to it.

What happens when a process tries to allocate more RAM than it is allowed?

On modern systems, the OOM killer will kill the cgroup-ed process. Quoting the kernel documentation on this one:

When a cgroup goes over its limit, we first try to reclaim memory from the cgroup so as to make space for the new pages that the cgroup has touched. If the reclaim is unsuccessful, an OOM routine is invoked to select and kill the bulkiest task in the cgroup.

I first tried to give Teams one gigabyte of RAM, which wasn't enough. Teams showed the splash screen and failed to start, dmesg -T showed me that it was killed right away inside the cgroup:

[Thu Feb 11 12:46:42 2021] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/cgTeams,task_memcg=/cgTeams,task=teams,pid=22920,uid=1000
[Thu Feb 11 12:46:42 2021] Memory cgroup out of memory: Killed process 22920 (teams) total-vm:2513452kB, anon-rss:322756kB, file-rss:57980kB, shmem-rss:0kB, UID:1000 pgtables:3444kB oom_score_adj:300
[Thu Feb 11 12:46:42 2021] oom_reaper: reaped process 22920 (teams), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Firefox behaves fun as well with a RAM limit. Large webpages get killed and a funny message is shown:

firefox tab kill

dmesg -T shows that it is not the firefox process, but a child named Web Content:

[Fri Feb 12 12:49:50 2021] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/cgFirefox,task_memcg=/cgFirefox,task=Web Content,pid=98779,uid=1000
[Fri Feb 12 12:49:50 2021] Memory cgroup out of memory: Killed process 98779 (Web Content) total-vm:4292056kB, anon-rss:1657332kB, file-rss:108680kB, shmem-rss:95364kB, UID:1000 pgtables:8884kB oom_score_adj:0
[Fri Feb 12 12:49:50 2021] oom_reaper: reaped process 98779 (Web Content), now anon-rss:0kB, file-rss:0kB, shmem-rss:94228kB

Oh well, at least my machine doesn't crash when I open a large merge request on gitlab with hundreds of changes.

Activate cgroups at boot with cgconfigparser

When this is all working you can make it permanent by placing this configuration in the file /etc/cgconfig.conf:

group cgTeams {
    perm {
        admin {
            uid = remy;
        }
        task {
            uid = remy;
        }
    }
    memory {
        memory.limit_in_bytes = 1585446912;
    }
}

Test the file for syntax errors:

sudo cgconfigparser -l /etc/cgconfig.conf   

The command should not give any output. For reference, I have three groups configured:

 group cgTeams {
     perm {
         admin {
             uid = remy;
         }
         task {
             uid = remy;
         }
     }
     memory {
         memory.limit_in_bytes = 2048m;
     }
 }
 group cgFirefox {
     perm {
         admin {
             uid = remy;
         }
         task {
             uid = remy;
         }
     }
     memory {
         memory.limit_in_bytes = 2048m;
     }
 }
 group cgThunderbird {
     perm {
         admin {
             uid = remy;
         }
         task {
             uid = remy;
         }
     }
     memory {
         memory.limit_in_bytes = 2048m;
     }
 }

systemd service for cgconfigparser

On Ubuntu 20.04 there is no systemd service to start cgconfigparser at boot. Here is a relatively simple service file I use to start cgconfigparser:

vim /lib/systemd/system/cgconfigparser.service

Contents:

[Unit]
Description=cgroup config parser
After=network.target

[Service]
User=root
Group=root
ExecStart=/usr/sbin/cgconfigparser -l /etc/cgconfig.conf
Type=oneshot

[Install]
WantedBy=multi-user.target

Enable and start the service:

systemctl enable cgconfigparser
systemctl start cgconfigparser

On boot the cgroups will then be created via this service. Easy to adapt to other init systems like openrc.

You can continue on with the next section to automatically place specific processes into specific cgroups via cgrulesengine but, for a simpler solution, you can edit your desktop launcher to run the command prefixed with cgexec. Or a simple cronjob every minute that runs cgclassify -g memory:cgTeams $(pidof teams) to put every running teams process in that cgroup. Before I had set up cgrulesengined I used three cronjobs every minute running cgclassify.

Automatically put processes into a specific cgroup with cgrulesengined

This part was a bit vague online, but there exists a daemon that automatically puts processes inside cgroups based on a few rules. The startup scripts in most versions of Debian and Ubuntu are either broken or missing, but the daemon seems to work. The manpage had more information about the syntax, I'll limit the example to our usecase. Edit the following file:

vim /etc/cgrules.conf

Contents:

remy:teams              memory          cgTeams/
remy:firefox            memory          cgFirefox/
remy:thunderbird        memory          cgThunderbird/

Replace remy by your username, the processes (teams) and the cgroups (cgTeams) by your own. Save and check the file with:

/usr/sbin/cgrulesengd -vvv

No output means no errors. You can omit :processname to limit everything by a user, memory can be replaced by other cgroup categories like cpu, but in our situation that is not applicable. I want to limit the memory of a few specific applications, not CPU cores or other resources.

systemd service for cgrulesengined

Just as with cfconfigparser, there is no default service on Ubuntu, but adding one is just as simple as the previous one.

Copy one of the configuration files:

cp /usr/share/doc/cgroup-tools/examples/cgred.conf /etc/cgred.conf

Add a systemd unit file:

vim /lib/systemd/system/cgrulesgend.service

Contents:

[Unit]
Description=cgroup rules generator
After=network.target cgconfigparser.service

[Service]
User=root
Group=root
Type=forking
EnvironmentFile=-/etc/cgred.conf
ExecStart=/usr/sbin/cgrulesengd
Restart=on-failure

[Install]
WantedBy=multi-user.target

Enable and start the service:

systemctl enable cgrulesgend
systemctl start cgrulesgend

After a reboot you should have the processes you configured in

To check if the process actually launches in the correct cgroup after a reboot you need to use the cgroup filesystem. The file /sys/fs/cgroup/memory/cgTeams/tasks lists all the process ID's that run in that cgroup. I can see that it is working for teams:

$ for pid in $(cat /sys/fs/cgroup/memory/cgTeams/tasks); do pgrep $pid; done
remy        3305  2.4  2.0 3688372 327456 ?      Sl   06:47   3:18 /usr/share/teams/teams
remy        3307  0.0  0.2 189960 39656 ?        S    06:47   0:00 /usr/share/teams/teams --type=zygote --no-sandbox
remy        3367  0.0  0.5 1832608 87364 ?       Sl   06:47   0:00 /usr/share/teams/teams --type=renderer
Tags: articles , cgroups , debian , desktop , docker , linux , lxc , microsoft , teams , ubuntu