This is a sysadmin post. The other day I was running a background process on a production machine. I thought it wasn't going to eat up many resources, but I was wrong. It turned out it was doing a lot of random I/O and things ground to a halt.Now often I will have top, vmstat and iostat open and renice annoying background processes to 20 when appropriate (r in top). If you don't already know, 'nix processes (I use FreeBSD) can have priorities, which the scheduler takes into account when giving out resources.
These priorities range from -20 to 20 (on FreeBSD at least), and you can see them in top under the PRI column. 0 is neutral. If you set something to 20, it will be tied for lowest priority process in the system.
nice is another command that messes with process priorities. It starts them out at a particular priority, as opposed to changing a priority via renice. For example 'nice -n 20 ./process' will start process at priority 20.
Then I got to thinking, why don't I do more of this initial nice stuff? Maybe seasoned sysadmins all already do this, but I went back through all my scripts and crontabs and explicitly set nice values. Then I went through my daemontools run scripts and set nice values there as well.
My web server (nginx) already did this via the 'worker_priority' variable, which I had previously set to -6. I set negative values to my most important scripts and relative values between them in order of importance. I set positive values to less important scripts, and then 20 to various background processes kicked off via cron. For example, backups are now niced to 20.
The system was already runnnig pretty smoothly, but now it is even more so. And I think, more importantly, it will react better in times of need.
Final tip. When I want to renice a bunch of stuff, I usually do so on the command line instead of top, e.g.:
ps auxww | grep -i crawl | awk '{print $2}' | xargs renice 10
That will take all the processes that match 'crawl' and renice them to 10.
Follow my blog. I'd be eternally grateful.
