Thursday, May 28, 2009

Linux partition advice

When I first partioned my disk I put /boot on the first partition and gave it 96M, following some advice found online I assume. This is not enough. I suggest giving it 256M. It is still a fraction of your hard disk. And because /boot is special it is impossible, or at least complex, to move some files to other partitions and link to them.

About a week ago ubuntu upgrade started complaining about a kernel upgrade problem. I ignored it for a few days assuming it would sort itself out. But it kept happening. Then when I viewed details of the upgrade I noticed (just briefly before it vanished off-screen) it was saying not enough diskspace on /boot.

Not again! When I tried to upgrade from Ubuntu 7 to Ubuntu 8 lack of space on /boot caused problems then too.

Poking around I also found I still had linux-generic packages installed, even though I'd switched to the linux-server kernel (see 6Gb on 32-bit linux). I deleted all packages that had the word "generic" in their name. After a reboot I still had one "generic" file left in /boot which I then just deleted. There was also a *.bak file for my current kernel. Datestamp was for a week ago, so I deleted that too.

It doesn't look like I've broken anything, and I'm now down to using 40M on /boot with 48M free (and I still have Ubuntu 7's linux-server kernel in there, which I think is now pointless, so I could reduce it even more).

Therefore, you can get away with a mere 96M /boot partition, it just requires more time and effort.

Conversely, I think a /boot partition above a certain size (1G?) causes problem at boot time, which is the whole reason for have a separate /boot partition. But I'm no expert, and that may be old-fashioned advice, and every BIOS on every motherboard made in the last 10 years may in fact be fine.

I dunno, and am too busy with more interesting stuff to study up on it, which is why I'll go for a 256M partition on my next computer.

Monday, May 25, 2009

Japanese NLP mailing list

A new mailing list for discussing Japanese NLP (natural language processing) in English has been set up, by Jim Breen (of JMDict fame):

There is a lot of software for processing Japanese text which is only documented in Japanese, and even then only minimally documented. So the new list is an ideal place (for those of us more comfortable in English than Japanese) for asking about how to use chasen, cabocha, namazu, etc. Or to describe what you are trying to do and get program and data suggestions. Hopefully people will also post about new software and data releases, related conferences, new academic papers, and so on.

Also, if the above interests you then you will also want to know about this Ubuntu repository for all kinds of NLP software:

Much of the Japanese stuff is UTF-8 ready (as opposed to the EUC-JP that academic Japanese software still likes to default to).

Thursday, May 21, 2009

Adobe PDF reader and Japanese fonts

Another casualty of my recent enforced ubuntu upgrade was Japanese fonts in pdf files. Adobe acroread has also moved from version 8 to version 9. When you meet Japanese in a PDF file it tells you the URL to go to to get asian font pack. Unfortunately that page only has asian fonts for acroread 8 and earlier!

The link should be:

(I'm mentioning this as it took a bit of work to discover it.)

Scroll down to the add-ons section, and it seems each language is now its own file, and the files are much bigger. (I don't know the difference between a "Font Pack" and a "Font Packs"; the files I are identical so I chose the latter.)

Unzip the bz2 file with "tar xjf FontPack910_jpn_i486-linux.tar.bz2"
Then move into the JPNKIT directory and type "./INSTALL"

The install process asks:
"Enter the location where you installed the Adobe Reader? /opt"

I didn't install it, Ubuntu did. However it seems Ubuntu is putting it in /opt!
Strange for a package-based distro to put anything there, but I accepted the /opt default and it worked. (This was different in Ubuntu 7, as I remember having to try lots of paths until I guessed the one it was after.)

Incidentally I have already got as an extra repository, but there is no acroread-fonts or similar package. Perhaps there is some legal issue (though I thought's raison d'etre was packages with legal issues).

Saturday, May 9, 2009

Moving encrypted partition to software RAID

I moved most of my partitions to software RAID a few weeks ago. But I left /home/darren/ because it was encrypted. However a few days ago I moved it too. Here is how.

The quick overview: it is exactly like moving any other type of existing partition to software RAID, except where you would format /dev/md7, prior to copying the existing data over to it, you would set up crypt on /dev/md7 instead.

Detailed Steps

These instructions assume that you have moved other partitions to software RAID, or are at least familiar with the process (see previous article). All commands are run as root.

If you have not got an existing crypt partition you need to prepare for it:
* Install cryptsetup
* modprobe dm-crypt
* Add dm-crypt to /etc/modules

I will be setting up /dev/md7 for software RAID 1, but with just /dev/sdb7 in the raid array initially, i.e.
mdadm --create /dev/md7 --level=1 --raid-disks=2 missing /dev/sdb7

Next I setup /dev/md7 for crypt with these commands:
cryptsetup luksFormat /dev/md7 -c aes -s 256 -h sha256
cryptsetup luksOpen /dev/md7 somecryptraid
mke2fs -j /dev/mapper/somecryptraid -L somecryptraid

Each command only takes a few seconds to run. You will be prompted for a password. Forget that password and the data on your partition is lost forever; there is no recovery ability. So choose carefully.

The next command sets chkdsk to run every 100 days, however many times you boot. This is just personal preference, and completely optional (the default is to run chkdsk more frequently):
tune2fs -c 0 -i 100 /dev/mapper/somecryptraid

Run "mkdir /mnt/md7" then edit /etc/fstab and add this line:
/dev/mapper/somecryptraid /mnt/md7 ext3 defaults,noatime 0 0

What we are doing here is saying we want our new encrypted software raid partition to be mounted somewhere temporary, so we can copy our existing data over to it.
(the "noatime" flag is optional, and nothing to do with software RAID or crypt)

And edit /etc/crypttab to add this line:
somecryptraid /dev/md7 none luks

This is the command that will cause it to prompt for password on boot time. If you already have an encrypted partition then you are adding the above in addition to your existing entry: you will have two encrypted partitions for the next boot.

Now reboot into single-user mode (the recovery kernel). You should be prompted for the password for your new crypt partition (in addition to any existing crypt partition). Now I move /home/darren to the new crypt partition with:
cd /home/darren/
cd -dpRx . /mnt/md7

This took over an hour to run for me.

Once it finishes edit /etc/fstab to change the /dev/mapper/somecryptraid entry from /mnt/md7 to be /home/darren. And comment out the previous entry for /home/darren. I.e. the new entry looks like:
/dev/mapper/somecryptraid /home/darren ext3 defaults,noatime 0 0

Reboot. When running df you should see a line like:
/dev/mapper/somecryptraid ... ... ... ... /home/darren

You can now remove (or comment out) the old "somecrypt" entry from crypttab. Also use fdisk to change the /dev/sda7 entry from "83 Linux" to "fd Linux RAID autodetect" (use the fdisk "t" command to do this). Reboot again.

Now you should be able to run:
mdadm --add /dev/md7 /dev/sda7

This will take a while to run, as it mirrors the data from /dev/sdb7 to /dev/sda7. "watch cat /proc/mdstat" will show its progress.

"rmdir /mnt/md7" as a tidyup step at the end. You might also want to delete commented out lines in /etc/cryptab and /etc/fstab if you like to keep those files lean and clean.


When I ran "mdadm --add /dev/md7 /dev/sda7" I got this error:
"mdadm: add new device failed for /dev/sda7 as 2: No space left on device"

I tracked this down to slightly different sizes as reported by fdisk:
/dev/sda7 16988 23361 51199092 fd Linux RAID autodetect
/dev/sdb7 16988 23361 51199123+ fd Linux RAID autodetect

Even though start and end cylinders are the same, the number of sectors is different! When I change the view ("u" command in fdisk), to show start and end sector, the problem is clearer:
/dev/sda7 272896281 375294464 51199092 fd Linux RAID autodetect
/dev/sdb7 272896218 375294464 51199123+ fd Linux RAID autodetect

In other words /dev/sda7 starts a few sectors later. My fix was to delete /dev/sda7 from the partition table, and then recreate it. Then /dev/sda7 showed the same start/end sector as /dev/sdb7. Weird and spooky, but now the "mdadm --add" command worked (after another reboot of course).

Tip: use fdisk to check your partition sizes are exactly the same before starting! The other way to fix this would have been to make /dev/sdb7 smaller but, by the time I realized, it was too late to do that.