six degrees of freedom

OpenStack “backing file” cleanup script

MD1032 — Wed, 12 Dec 2012 09:30:40 +0000

Sometimes OpenStack leaves the “backing file” of the sparse root filesystem behind after a VM is destroyed. Those files are located in /var/lib/nova/instances/_base/ and can use up hundreds of GB. To find out which of those backing files are orphans and delete them, someone has posted a script, which I slightly improved:

#!/bin/sh
cd /var/lib/nova/instances
find -name "disk*" | xargs -n1 qemu-img info | grep backing | sed -e 's/.*file: //' -e 's/ .*//' | sort | uniq > /tmp/ignore
while read i; do
ARGS="$ARGS \( \! -path $i \) "
done < /tmp/ignore
eval "find /var/lib/nova/instances/_base/ -type f $ARGS -delete"

Someone in the original post claimed that adding remove_unused_base_images = True to nova.conf would do this automatically, however I could not reproduce that on Folsom.

How to create a bootable volume in Openstack Folsom

MD1032 — Sat, 10 Nov 2012 02:33:14 +0000

Wouldn’t it be nice to have the root filesystem of your Openstack VM inside a volume? This way you could install a lot of packages on the root partition without spreading the files over several volumes. You could take a snapshot of the entire system (OS & data) and boot right off it.

Openstack Folsom supports this feature. Here’s how you can create your own bootable volume:

On your desktop machine, download the cloud image of the OS that you want to boot from a volume. I chose Ubuntu 12.10:
wget http://cloud-images.ubuntu.com/releases/quantal/release/ubuntu-12.10-server-cloudimg-amd64-disk1.img
Now convert it to RAW format using qemu-img
qemu-img convert -O raw ubuntu-12.10-server-cloudimg-amd64-disk1.img ubuntu-12.10-server-cloudimg-amd64-disk1.raw
Start up a regular VM from the Ubuntu cloud image (or any other image you have)
Create a volume (that you later want to boot from) and attach it to the VM
SSH into the VM and become root
check if your volume is attached with fdisk -l. The volume device name is /dev/vdb in my case
ssh into your desktop machine (or wherever you have the RAW image) from the VM and write the image data directly into the volume device
ssh user@desktop cat ubuntu-12.10-server-cloudimg-amd64-disk1.raw > /dev/vdb

If you want your root partition to be larger than 2GB, you can now resize it to the volume size. To do that, start parted, get the partition information, delete (!) the partition, create a new one with the same “start” value, but with (total size-1) as the end value. Partition type is “primary”, filesystem “ext4″.

root@3:/home/ubuntu# parted /dev/vdb GNU Parted 2.3 Using /dev/vdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) unit B (parted) p Model: Virtio Block Device (virtblk) Disk /dev/vdb: 21474836480B Sector size (logical/physical): 512B/512B Partition Table: msdos

Number Start End Size Type File system Flags
1 8225280B 2146798079B 2138572800B primary ext4 boot

(parted) rm 1
(parted) mkpart
Partition type? primary/extended? p
File system type? [ext2]? ext4
Start? 8225280
End? 21474836479
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? i

(parted) set 1 boot on
(parted) p
Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 21474836480B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 8225280B 21474836479B 21466611200B primary ext4 boot

(parted) q
Information: You may need to update /etc/fstab.

Now you can detach the volume in Horizon and create a new instance. Choose “Boot from Volume” in “Volume Options” and choose your volume. You may also take a snapshot of the volume first, if you want to preserve it in a fresh state for later. You still need to choose an image in the “Details” section, which makes no sense in this case, since the VM is entirely booted off the volume.

collectd libvirt plugin build dependencies

MD1032 — Fri, 31 Aug 2012 07:03:33 +0000

When configuring collectd using ./configure –prefix=/usr/local –enable-libvirt , the modules section that’s displayed after configuring on my machine shows:
libvirt . . . . . . . no (dependency error)
The solution here is to install the libxml2-dev package in addition to libvirt-dev, since that is needed by collectd to parse the XML VM configuration stanzas that libvirt provides.

Airport Extreme: update dynamic hostname through BIND

MD1032 — Mon, 11 Jun 2012 05:43:10 +0000

If you are the owner of an Apple Airport Extreme base station, you may have wondered if it is possible to assign a static hostname to your dynamically changing ADSL IP address. Many other routers offer this feature through one of the popular dynamic DNS services such as DynDNS. Apple however decided not to support the proprietary interfaces of those commercial services, but instead use a generic approach as described in RFC 2136 Dynamic Updates in the Domain Name System (DNS UPDATE).

Here’s how you can use this feature.

Requirements:

Apple Airport Extreme Base Station
Airport Utility version 5.6 (the DNS update feature is not accessible in 6.0+)
a domain name where you can choose your own nameserver (most domain sellers allow this)
a (Linux) server running BIND9 on a static IP address

Setup

for this setup I assume that your Linux box has the static hostname mybox.com on IP address 46.19.92.98. The domain name you want to use for your airport extreme is airport.mydomain.com. This means mydomain.com is owned by you and has its nameserver set to mybox.com.
Install BIND version 9 on your Linux server (e.g. “sudo apt-get install bind9″) and create a zone file for your domain. You can put it in /etc/bind/zones/mydomain.com.db and make it look like this:

$ORIGIN .
$TTL 86400	; 1 day
mydomain.com		IN SOA	mybox.com. mailaddress.gmail.com. (
				2012060524 ; serial
				28800      ; refresh (8 hours)
				7200       ; retry (2 hours)
				864000     ; expire (1 week 3 days)
				86400      ; minimum (1 day)
				)
			NS	mybox.com.
			A	46.19.92.98
			MX	10 mydomain.com.
$ORIGIN mydomain.com.
*			A	46.19.92.98
$ORIGIN _dns-sd._udp.mydomain.com.
b			PTR	mydomain.com.
db			PTR	mydomain.com.
dr			PTR	mydomain.com.
lb			PTR	mydomain.com.
r			PTR	mydomain.com.
$ORIGIN _udp.mydomain.com.
_dns-update		SRV	0 0 53 mybox.com.

google for some BIND tutorials if you need more information on the zone file configuration
make sure port 53 UDP is open in your Linux box’s firewall
create a RNDC key. This is the “password” used to update your DNS zone. Run “rndc-confgen -a -c /etc/bind/rndc.key”
add the following to your /etc/bind/named.conf.local:

include "/etc/bind/rndc.key";
zone "mydomain.com" {
      type master;
      file "/etc/bind/zones/mydomain.com.db";
      update-policy {
        grant rndc-key name airport.mydomain.com A;
      };
};

ensure that your zone file and the named.conf.local have permissions -rw-r–r– and are owned by root, group bind.
on Ubuntu 12.04 I had to edit the file /etc/apparmor.d/usr.sbin.named and change the line “/etc/bind/zones/** r,” to “/etc/bind/zones/** rw,”
restart apparmor and bind
check the syslog for any bind errors
if you’ve just changed the nameserver of domain.com to mybox.com, it may take 24h or more for this update to reach your provider’s DNS cache
check your nameserver configuration with “dig”. “dig a domain.com” should give you an answer section with your Linux box’s IP address. “dig ns domain.com” should return mybox.com. “dig PTR b._dns-sd._udp.domain.com” should return domain.com.
when everything works as expected, we can now configure Airport Extreme. Open “Airport Utility 5.6″, choose the “base station” tab and click “Edit…”. Configure it as in the screenshot below. The password is the “secret” as mentioned inside your rndc.key file. Enter it without the quotes.

click “Done” and “Update”. Your Airport Extreme should update the DNS A record for airport.domain.com every 15 minutes. In your syslog it looks like this:

Jun 11 07:15:07 alderaan named[31953]: client 40.224.233.149#5353: updating zone 'domain.com/IN': deleting rrset at 'airport.domain.com' A Jun 11 07:15:07 alderaan named[31953]: client 40.224.233.149#5353: updating zone 'domain.com/IN': adding an RR at 'airport.domain.com' A

you can now reach your home network from anywhere through the hostname “airport.domain.com”. Success!!

Script to convert a SYMPA mail archive to mbox format (sympa2mbox)

MD1032 — Mon, 14 May 2012 06:31:31 +0000

Export the mail archive using the SYMPA web interface, extract the zipfile and run the following script over the mail directory. You need to have “formail” installed.

#!/bin/bash
 
# This scripts takes a sympa email archive and converts it into a
# single mbox file
 
path="$1"
dest="$2"
 
function die()
{
	echo "$1"
	exit
}
 
# ensure arguments are passed
[ -z "$path" ] && die "Pass archive directory as first argument."
[ -z "$dest" ] && die "Pass destination mbox file name as second argument."
 
# sanity checking
[ ! -d "$path" ] && die "Can't find '$path'"
touch "$dest" || die "Can't write to '$dest'"
 
# we rely on directories being name in alpha order oldest -> newest 
for dir in $(ls "$path"); do
	if [ -d "$path/$dir" ]; then
		echo -n "Working on $path/$dir..."
		files=$(ls "$path/$dir" | sort -n)
		for file in $files; do
			formail -ds < $path/$dir/$file >> $dest
		done
		echo "done"
	fi
done

Remove failing VMs in OpenStack

MD1032 — Fri, 04 May 2012 05:40:32 +0000

Occasionally OpenStack VMs fail to spawn or fail to terminate and cannot be removed from the VM list. Deleting the VM using the command line or web interface has no effect, they are stuck in the “spawning” or “deleting” state. To get rid of these zombie entries I remove them from the database using the following bash script (pass the machine UUID as parameter):

#!/bin/bash mysql -uroot -ppassword << EOF use nova; DELETE a FROM nova.security_group_instance_association AS a INNER JOIN nova.instances AS b ON a.instance_id=b.id where b.uuid='$1'; DELETE FROM nova.instance_info_caches WHERE instance_id='$1'; DELETE FROM nova.instances WHERE uuid='$1'; EOF

Force library refresh on Nook Simple Touch

MD1032 — Sat, 31 Mar 2012 07:20:07 +0000

If you have rooted a Nook Simple Touch e.g. using http://forum.xda-developers.com/showthread.php?t=1343143, you may want to run some scripts to download some books directly onto your device. The problem is that these books you saved to /media or /sdcard only show up in the library after you reboot the Nook.

To force a library refresh, create a script containing the line

busybox killall vold

of course you need to have busybox installed through Google Play and you have to run this script with root permissions.

RAID5 array of 6 SSDs Performance Evaluation

MD1032 — Tue, 20 Mar 2012 03:34:39 +0000

I’ve got six Crucial m4 SSDs with 256GB in a RAID5 array, resulting in 1.2TB useable disk space.The SSDs are connected via SATA3 to a Gigabyte 990FX UD7 board. I’m running 64bit Ubuntu 11.10 server.

The following two articles offer some good ideas on performance tuning the SSD RAID:

http://h3x.no/2011/07/09/tuning-ubuntu-mdadm-raid56

https://wiki.archlinux.org/index.php/Solid_State_Drives

The chuck size on /dev/md127 is 512k, which is the default in Ubuntu and suits the 512k erase block size of the Crucial m4 disks. I’ve decided to partition the disks with “gdisk”, using a GPT. The partition starts at sector 2048 to align with the SSD chunk size. The filesystem is ext4, using 4k block size and 128 blocks stride, which again matches the 512k EBS. There is no need to pass any special options to mkfs.ext4, it was all autodetected.

I’m using the following optimizations in /etc/rc.local:

md=`ls -l /dev/disk/by-id/md-uuid-f70a5a71:577d6424:8ca4b9cf:265a61a3 | 
awk '{ print $NF }' | sed -e 's/[/\.]//g'`
 
echo 32768 >  /sys/block/$md/md/stripe_cache_size
blockdev --setra 4096 /dev/$md
 
SSD=/dev/disk/by-id/ata-M4*
for s in $SSD
do
  NODE=`ls -l $s | awk '{ print $NF }' | sed -e 's/[/\.]//g'`
  echo noop > /sys/block/$NODE/queue/scheduler
done

Make sure /etc/rc.local is run by bash, not sh. The block device scheduler for the SSDs is set to “noop”, since there is no seek time on the SSD.

Here are some benchmarks:
root@localhost:/mnt/ssd# dd of=file.bin if=/dev/zero bs=1G count=10 10+0 records in 10+0 records out 10737418240 bytes (11 GB) copied, 20.4972 s, 524 MB/s root@localhost:/mnt/ssd# dd if=file.bin of=/dev/null 20971520+0 records in 20971520+0 records out 10737418240 bytes (11 GB) copied, 16.53 s, 650 MB/s

650MB/s read speed is not too bad!

Probably add GRUB_CMDLINE_LINUX_DEFAULT=”bootdegraded=true” to /etc/default/grub to allow the array to boot in degraded state.

Note that there is no TRIM support for software raid5 yet (only levels 0,1,10 as of Linux 3.3), but support for levels 4,5,6 is in the making. Until then, a regular wipe of all SSDs may allow for steady performance.

How I built my custom Mac

MD1032 — Mon, 21 Nov 2011 06:56:00 +0000

Here’s a list of the parts:

Intel Core i7 2600K $335.00 Corsair Vengeance CMZ16GX3M4A1600C9B 16GB (4x4GB) DDR3 $149.00 Antec Sonata IV with 620W $187.00 Crucial M4 SSD 256GB $439.00 LG CH12LS28 12X BD-R Blu-ray DVD Combo Drive $69.00 Arctic Cooling Freezer 13 Pro CO CPU Cooler $59.00 Gigabyte GA-Z68MX-UD2H-B3 Motherboard $165.00
First I had to update the BIOS to F12 and configure a few settings:

enable HPET in 64bit mode
Set all SATA controllers to AHCI
give 64MB of memory to the integrated graphics
turn off the SATA RAID BIOS

I then prepared a UniBeast stick on my other Mac, according to the instructions here. I followed the guide but could not get to the Lion installer. After fiddling around with a few parameters I realised that passing PCIRootUID=1 on the UniBeast loader did the trick. After partitioning and formatting the SSD I was able to install Lion and boot it through the USB stick.

I then installed Multibeast with the DSDT for my mainboard. After rebooting Lion got stuck on the white apple screen. To go ahead I passed GraphicsEnabler=false to the boot loader and got into Lion at a low resolution. System Info reports that the machine is a Mac Pro, and that was the problem since the Mac Pro doesn’t have an Intel HD 3000 graphics card. So I ran Multibeast again and installed the Mac Mini System Definition. I had to add it to my /Extra/org.chameleon.Boot.plist:

< ?xml version="1.0" encoding="UTF-8"?>
< !DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
 version="1.0">
>
	>GraphicsEnabler>
	>Yes>
        >Timeout>
        >2>
        >Legacy Logo>
        >Yes>
	>SMBIOS>
	>/Extra/smbios.plist>
	>EthernetBuiltIn>
	>y>
>
>

After a reboot I got it running in full resolution. What was still missing is the USB3 driver and the Lnx2Mac RTL81xx driver from MultiBeast, which installed with no problems. Everything is working now, even the sound and the USB Bluetooth stick. It’s a rock stable system overclocked to 4GHz and I have never seen a faster Mac. And it is silent, too. CPU core temperature never goes over 60 degrees, even overclocked & under full load.

Mounting Drobo b800i on Linux using iSCSI

MD1032 — Fri, 11 Nov 2011 07:21:10 +0000

iSCSI support in the new Drobo is a really neat feature. Here’s how to mount the Drobo as a block device in Linux:

install some hard disks and connect the Drobo via USB cable to a Windows or Mac computer
open the Drobo Dashboard and create a volume. I recommend to set it to 16TB size, no matter how many disks you actually have installed. This allows you to grow the filesystem later when you add more disks.
give the Drobo a static IP address on the iSCSI1 interface and connect it via Ethernet cable to your Linux machine (directly or using a gigabit switch)
unplug the USB cable
in Linux you need to install the iSCSI daemon. On Ubuntu just become root and run apt-get install open-iscsi
make sure your Ethernet card is configured to be on the same network as the Drobo
run iscsiadm -m discovery --type sendtargets --portal 169.254.2.0 where the IP address is your Drobo iSCSI1 IP address
you should now see your volumes listed, like 169.254.2.0:3260,0 iqn.2005-06.com.drobo:b800i.tdb1119b0269.id1
connect to your id1 volume by running iscsiadm -m node --targetname iqn.2005-06.com.drobo:b800i.tdb1119b0269.id1 --portal 169.254.2.0 --login
run dmesg | tail to see whether the block device was detected by the Linux kernel
run parted /dev/sdb to partition your device. Use mklabel to create a GUID partition table (GPT). Then use mkpart to create a partition. Drobo apparently does not support ext4, but ext3 works fine for me.
the block device will appear to be of size 16TB. However your actual useable space depends on the number and size of the disks you are using and the amount of redundancy you selected. Make sure that the partition you create is smaller than the actually available space. You can calculate the capacity of your Drobo here. Leave the rest of the 16TB disk as free space.
exit parted and format the disk using mkfs.ext3 /dev/sdb1
mount /dev/sdb1 and enjoy blazing fast storage
for automatic iSCSI target login on boot, run iscsiadm -m node -T iqn.2005-06.com.drobo:b800i.tdb1119b0269.id1 -p 169.254.2.0 --op update -n node.startup -v automatic
If the target login fails, it helped me once to disconnect the USB cable
If the Drobo volume should mount on boot, add an entry to the /etc/fstab with options auto,_netdev