Linux

Linux stuff

Limit a threaded Linux app's CPU consumption

I rarely bother to "rate limit" the amount of CPU my Linux processes can do, but since I unfortunately have to share my desktop with some CPU intensive tasks, sometimes it makes sense to do so, for multi-threaded applications at least. Linux has, for some time now, a feature to set CPU affinity for a process. This is btw something that Sun/Solaris have had for almost as long as I can remember. With this feature, you can specify which CPUs (cores) a process is allowed to use.

As an example, lets assume we want the application traffic_server to only consume core 2 and 3, you'd simply start the application with

# taskset -c 2,3 traffic_server

Now, some applications have this built in (unfortuantely traffic_server does not, yet), and there are APIs in linux to control this (see the man page for sched_setaffinity() ). You can of course also "nice" your application accordingly, but the above lets you get a somewhat more controlled behavior (again, for threaded apps that can use more than one core).

Hacking: 

Fedora Core 15 as a VirtualBox guest OS

Fedora Core 15 was released today! I of course went straight into VirtualBox, after the download finished of course, to create a new couple of VMs. As I usually do, I assigned 512MB of RAM for the new VMs (one 32-bit and one 64-bit). Booting up, the kernel would crash, wth? It turns out, 512MB is not enough to boot up FC15, at least not under VirtualBox. I increased the memory allocated to each VM to 768MB, and things works just fine now.

Now of course to deal with the disaster that is Gnome 3.0. It's probably time to switch over to KDE, the Gnome org is just going crazy removing options and configurations for no apparent reason other than telling people they know best. I mean, come on, not even allowing me to use "Select windows when the mouse moves over them"?? Grrrr...

Update: You can achieve this particular setting ("focus follows mouse") using gconf-edit, setting

/apps/metacity/general/focus_mode

 

to "sloppy" (without the quotes). Sigh. Of course, turning on "fallback" mode is almost a requirement.

Hacking: 

Apache Traffic Server recipes

Apache Traffc Server is a high performance, customizable HTTP proxy server. This is a collection of small "recipes", showing how to accomplish various tasks. This cookbook is work in progress, I haven't quite decided yet how I want to handle this "documentation". Perhaps it belongs in Apache official docs, but for now, I find it easier to use the tools I'm used to here on ogre.com to maintain this.

Hacking: 

Forcing a "check" on a Linux md RAID device

To be proactive, I've found that once in a while (perhaps via a cron job) it might be a good idea to force a Linux RAID (mirror or RAID5/6) to be checked for consistency. This can easily be done from command line, with something like

% sudo echo check >  /sys/block/md0/md/sync_action

Also, to repair bad raid device, perhaps something like

% sudo echo repair >/sys/block/md0/md/sync_action

This second command solved a problem for me, where I'd get an email warnings once a week, saying "WARNING: mismatch_cnt is not 0 on /dev/md0". This was verified with

% sudo cat /sys/block/md1/md/mismatch_cnt
256

 

Hacking: 

Upgrading Fedora with yum

I've been doing a number of upgrades of Fedora Core recently, and was semi-disappointed with how slow the upgrade process from DVD was. On every upgrade, it had to first install well over 1000 packages from DVD, and to finish it off with a "yum update" after the upgrade, at least another 1000 packages upgraded again. I decided for my next upgrade to do an update on the live system, using yum directly. I decided to follow the excellent upgrade instructions from the Fedora project, and it worked surprisingly well (and much, much faster). On top of that, I also cleaned out some groups that I no longer thought were necessary.

Here's the quick summary of the commands I ran:

# Run an update first, just in case
yum update

# Cleanup/merge config updates
yum install rpmconf
rpmconf -a

# Find and review unused packages
yum install yum-utils
package-cleanup --leaves
# Now yum remove the packages you think should be removed

# Find and review lost packages
package-cleanup --orphans

# Cleanup
yum clean all

# Do the upgrade, preferably in run-level 3 (no GUI)

# For FC13 -> 14 upgrade
rpm --import https://fedoraproject.org/static/97A1071F.txt
# For FC14 -> 15 upgrade
rpm --import https://fedoraproject.org/static/069C8460.txt
# For FC15 -> 16 upgrade
rpm --import https://fedoraproject.org/static/A82BA4B7.txt
# For FC16 -> 17 upgrade
rpm --import https://fedoraproject.org/static/10D90A9E.txt
# For FC17 - > 18 upgrade
rpm --import https://fedoraproject.org/static/DE7F38BD.txt
# For FC18 - > 19 upgrade
rpm --import https://fedoraproject.org/static/FB4B18E6.txt
# For FC19 -> 20 upgrade
rpm --import https://fedoraproject.org/static/246110C1.txt

yum update yum
# The following steps are a bit different than Fedora recommendations, they
# say to run the updates with --skip-broken. But I don't like it.
yum check
# and then resolve any duplicates. When running the following, you will get
# conflicts, resolve those by uninstalling the offending packages.
# For FC13 -> 14
yum --releasever=14 distro-sync
# For FC14 -> 15
yum --releasever=15 distro-sync
yum groupupdate Base
# For FC15 -> 16
yum --releasever=16 --disableplugin=presto distro-sync
yum groupupdate Base
# For FC16 -> 17
yum --releasever=17 --disableplugin=presto distro-sync
yum groupupdate Base
# For FC17 -> 18
yum --releasever=18 --disableplugin=presto distro-sync
yum groupupdate Base
# For FC18 -> 19
yum --releasever=19 --disableplugin=presto distro-sync
yum groupupdate Base
# For FC19 -> 20
yum --releasever=20 --disableplugin=presto distro-sync
yum groupupdate Base
 

# Examine other groups and update (or remove) as necessary yum grouplist yum groupupdate "Administration Tools" "Server Configuration Tools" ... # Prepare for boot, assuming your boot device is /dev/sda (change as appropriate) /sbin/grub-install /dev/sda # Or, if you are switching to grub (as in FC16) /sbin/grub2-mkconfig -o /boot/grub2/grub.cfg /sbin/grub2-install /dev/sda # Update startup order; This is not for FC16 or later! cd /etc/rc.d/init.d; for f in *; do /sbin/chkconfig $f resetpriorities; done # Find packages that haven't been upgraded package-cleanup --orphans

I made a small shell script available to make it a little easier to manage the update (and additions / removals) of yum groups. It's far from perfect, and does indeed have bugs, but I still find it helpful. Right now, it still runs the necessary yum commands "interactively", to give you some extra safety deciding if you want to proceed with a particular operation. The script supports upgrading, removing and adding yum groups. It is available from github, at https://github.com/zwoop/scripts, it is named yumgroup.sh.

 

Note: I take no responsibilities for failed upgrades, but this process worked nicely for me, upgrading from FC13 to FC14.

Hacking: 

Linux / Fedora Core and nf_conntrack:

While testing and benchmarking some new features in Apache Traffic Server on my dev box, I started having really odd problems with inconsistently lost connections or really slow performance. Upon examing the logs, I found a large number of kernel barf like

Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.
Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.
Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.
Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.

Clearly this could not be good. I poked around, and could not find any ways to turn off this kernel module, at least on my Fedora Core 13 it seems compiled straight into the kernel (i.e. no .ko). There are ways to increase the table size (sysctl's) but that seemed like a band aid at best. Bummer. So, I went to the mountain (Noah F.) and asked for advice. After some poking around, we found that forcing iptables to ignore the conntracking helps, a lot. E.g.

iptables -t raw -I OUTPUT -j NOTRACK

 seems to do the trick for my particular case. This turns it off for all protocols, but you could probably limit it further (e.g. for tcp only with a "-p tcp" option). Also, you might want to do this on the input as well, e.g.

iptables -t raw -I PREROUTING -j NOTRACK

We found this old report against Fedora Core 10 as well, which describes the same problem, but no solution.

Hacking: 

Modifying a Linux software RAID mirror

While upgrading my desktop to Fedora Core 12, I also decided to modify a software RAID mirror that I had on the old system. In particular, I wanted to do two things, without losing data:

  1. Change the partition layout, the old disks had partitions for OS and swap allocated to them, which I no longer needed. I know, LVM would have been nice here, but alas, I wasn't using it on this older box.
  2. I also wanted to upgrade to EXT4, and I've heard that upgrade process can potentially corrupt the entire disk (ask Bryan Call if you don't believe me).

So, since I'm using mirroring, I figure the "right" approach would be to just break the mirror, and do a migration safely that way. And of course, that does work, and there's plenty of information in the man-pages how to do this, but I figure I'll write down the steps I took so I can remember it myself (and maybe someone else finds it useful too).

The first step is to break the mirror, in my case, the RAID mirror is /dev/md2 using /dev/sdb and /dev/sdd, and then mount the broken mirror half on the file system:

# mdadm --manage /dev/md2 --fail /dev/sdb1
# mdadm --manage /dev/md2 --remove /dev/sdb1
# mkdir /mnt/old-data
# mount /dev/sdb1 /mnt/old-data

We now have a complete "copy" of the mirrored data, make sure /mnt/old-data (or whatever you named it) looks good. The next step is to get rid of the old RAID device, and create a new one. In my case, I need to run fdisk as well, to change the partition layout, but I'm not going into details here exactly what I did, since it's very specific to my setup.

# mdadm --stop /dev/md2
# fdisk /dev/sdd
# mdadm --create /dev/md0 --level 1 --raid-devices=2 missing /dev/sdd1

Finally, we can now create the new filesystem, and migrate the old data over to it:

# mke2fs -t ext4 -j /dev/md0
# mkdir /mnt/new-data
# mount /dev/md0 /mnt/new-data
# rsync -av /mnt/old-data/ /mnt/new-data

The final step is to add back the "old-data" partition into the new mirror, but before doing so, make sure the "new-data" looks alright.

# umount /mnt/old-data
# fdisk /dev/sdb
# mdadm --add /dev/md0 /dev/sdb1

 

That's it! Easy, and very little risk for lost data or corruptions.

Hacking: 

Pages

Subscribe to RSS - Linux