Preparation

First, I nuke everything. I already performed an ATA Secure Erase on the SSD in the previous post. So I boot the Arch installation USB and similarly do an ATA Secure Erase for the HDD. Then, I create the EFI System Partition on the SSD. This time around, I'm going to let it double as my /boot partition.

Here, I initialized the SSD to GPT and create the ESP partition:

# gdisk /dev/sdb

(create a 512M partition with partition type EF00)

Then format the partition with a FAT32 filesystem (I don't think the volume label matters):

# mkfs.vfat -F32 /dev/sdb1 -n ESP

With the ESP taken care of, I reboot and start the Windows 7 installation. I am installing Windows completely on the HDD (since I rarely use it), so it doesn't even touch the SSD except to put its bootloader information in the ESP on the SSD. I otherwise follow this very excellent guide for installing Windows 7 on a T420.

As far as the multiple drives goes, Windows wouldn't let me install to the HDD at first. But after I loaded the Intel SATA RST drivers (details in the link above), then it was just fine.

After installing Windows, I performed all the updates, used ninite, and set the hardware clock to UTC.

/dev/sda (HDD)
128 MiB Microsoft reserved partition (created by Windows 7 install)
60 GiB Windows 7
240 GiB (remainder) LVM-on-LUKS

/dev/sdb (SSD)
512 MiB EFI System Partition
59 GiB (remainder) LVM-on-LUKS

Linux Install

As before, I use the FDE guide from the Arch wiki.

I booted into the Arch installation USB and created the Arch partitions with gdisk. The first one, on the SSD, will store the / filesystem as well as swap, so I will use LVM. (I can't have the swap partition on the HDD if I want hibernation support, so that's why I have the swap partition on the SSD.) Its GPT partition hexcode is 8e00. The second one, on the HDD, will store /home, but I will also make it LVM to make future adjustments easier later (also GPT partition hexcode 8e00).

Next is to setup LUKS with dm-crypt. Because I have two LUKS partitions (one each on the SSD and HDD), this setup is a little more complicated. I setup LUKS on both partitions and use a passphrase. For convenience, I use the same passphrase. From what I can tell, I would need to enter in two passphrases when I boot the system (one for each LUKS partition). This is a little cumbersome, so the way around this is to also use a keyfile. The LUKS partition on the HDD containing /home can be opened with either the passphrase or a keyfile, and the keyfile will be stored in the / partition. When / is decrypted and mounted, /etc/crypttab will be used to look up the keyfile and unlock the second LUKS partition. This way, only one passphrase needs to be entered upon bootup.

Since my T420 uses an Intel chip with the Sandy Bridge microarchitecture, it includes support for AES-NI, or hardware-accelerated AES. This is confirmed using the benchmark:

# cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       405795 iterations per second
PBKDF2-sha256     231167 iterations per second
PBKDF2-sha512     152942 iterations per second
PBKDF2-ripemd160  334367 iterations per second
PBKDF2-whirlpool  184608 iterations per second
#  Algorithm | Key |  Encryption |  Decryption
aes-cbc       128b   525.4 MiB s  1886.0 MiB/s
serpent-cbc   128b    68.5 MiB/s   279.7 MiB/s
twofish-cbc   128b   161.4 MiB/s   307.0 MiB/s
aes-cbc       256b   418.0 MiB/s  1444.0 MiB/s
serpent-cbc   256b    68.2 MiB/s   280.2 MiB/s
twofish-cbc   256b   160.2 MiB/s   304.7 MiB/s
aes-xts       256b  1616.0 MiB/s  1626.0 MiB/s
serpent-xts   256b   285.0 MiB/s   271.0 MiB/s
twofish-xts   256b   290.0 MiB/s   295.1 MiB/s
aes-xts       512b  1241.0 MiB/s  1270.0 MiB/s
serpent-xts   512b   288.6 MiB/s   271.0 MiB/s
twofish-xts   512b   295.1 MiB/s   302.0 MiB/s

So, I format the two partitions with LUKS:

cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random --verify-passphrase luksFormat /dev/sdb2
cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random --verify-passphrase luksFormat /dev/sda3

And open them:

cryptsetup open --type luks /dev/sdb2 ssdlvm
cryptsetup open --type luks /dev/sda3 hddlvm

Now that I have the LUKS partitions opened, I use dd if=/dev/zero of=/dev/mapper/hddlvm and dd if=/dev/zero of=/dev/mapper/ssdlvm to easily wipe the disks. By checking the progress of dd occasionally (send SIGUSR1 to the dd process), this serves as a benchmark for how fast I can write to the encrypted HDD and SSD. For the HDD, it varied between from high 80's MB/s down to 66 MB/s (due to faster accesses near beginning of drive). For the SSD, it started around 190 MB/s (burst writing) but slowly decreased to about 113 MB/s (sustained writing).

Now I create my LVM partitions, I use two separate volume groups so I can control whether the logical volumes are the HDD or SSD:

pvcreate /dev/mapper/hddlvm
pvcreate /dev/mapper/ssdlvm

vgcreate hddvg /dev/mapper/hddlvm
vgcreate ssdvg /dev/mapper/ssdlvm

lvcreate -L 4G ssdvg -n swap
lvcreate -l 100%FREE ssdvg -n root
lvcreate -l 100%FREE hddvg -n home

I create a 4 GiB swap partition and fill up the rest with / and /home (I don't use the contiguous option for swap because it's on an SSD, so there's no benefit).

Create filesystems, also label them:

mkfs.ext4 -L arch-root /dev/ssdvg/root
mkswap -L arch-swap /dev/ssdvg/swap
mkfs.ext4 -L arch-home /dev/hddvg/home

Tune how much space is reserved for the superuser:

tune2fs -m 1.0 /dev/ssdvg/root
tune2fs -m 0.0 /dev/ssdvg/home

Add metadata to check the partitions after so many mounts or after so much time:

tune2fs -c 31 -i 31d /dev/hddvg/home
tune2fs -c 30 -i 6w /dev/ssdvg/root

And finally, mount everything:

swapon /dev/ssdvg/swap
mount /dev/ssdvg/root /mnt
mkdir /mnt/{home,boot}
mount /dev/hddvg/home /mnt/home
mount /dev/sdb1 /mnt/boot # this is the EFI System Partition

I install Arch and follow the rest of the guide here: https://wiki.archlinux.org/index.php/Installation_Guide#Configure_the_system. In the generated /etc/fstab, I update the relatime mount options to noatime for better performance. For the initrd to know about the encrypted / and LVM, in the HOOKS array in /etc/mkinitcpio.conf, I add encrypt lvm2 before filesystems and add shutdown at the end.

Last time around, I used GRUB, but this time I use the gummiboot bootloader. In order for gummiboot to create boot loader entries, the EFI variables needs to be accessible to the chroot. Outside the chroot:

modprobe efivars && mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars

Now install gummiboot (back inside the chroot):

pacman -S gummiboot
gummiboot --path=/boot install

Gummiboot needs to know the kernel arguments so it can (1) handle LUKS, (2) know where / is (due to LVM), and (3) resume from hibernation. The contents of my /boot/loader/entries/arch.conf:

title       Arch Linux
linux       /vmlinuz-linux
initrd      /initramfs-linux.img
options     root=/dev/mapper/ssdvg-root cryptdevice=/dev/sdb2:ssdvg:allow-discards resume=/dev/mapper/ssdvg-swap rw

Note the allow-discards option, so LUKS will allow me to use TRIM.

Finally, in order for /home to mount, I added the following line to /etc/crypttab:

hddlvm          /dev/sda3       /etc/luks-keys/hddlvm-key

The third field can be left blank if you want it to ask for the password, otherwise, it uses the specified keyfile. Here's how I created the keyfile. I create a random binary file:

dd if=/dev/urandom of=mykeyfile bs=512 count=4

and save it to /etc/luks-keys/hddlvm-key. Then I can add it to a keyslot for the HDD LUKS volume:

# cryptsetup luksAddKey /dev/sda3 /etc/luks-keys/hddlvm-key

and verify with sudo cryptsetup luksDump /dev/sda3 that key slot 0 (the passphrase) and 1 (the keyfile) are enabled. As a reminder, the initrd asks for a password to open the LUKS partition on the SSD (/dev/sdb2), and uses LVM to mount / and swap. Later in the boot process, systemd sees in /etc/crypttab to open the LUKS partition on the HDD (/dev/sda3) using the keyfile stored on /, and then uses LVM to mount /home.

With that in place, everything should boot! Here's the final layout of everything:

$ lsblk
NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                8:0    0 298.1G  0 disk
├─sda1             8:1    0   128M  0 part
├─sda2             8:2    0  59.9G  0 part
└─sda3             8:3    0 238.1G  0 part
  └─hddlvm       254:3    0 238.1G  0 crypt
    └─hddvg-home 254:4    0 238.1G  0 lvm   /home
sdb                8:16   0  59.6G  0 disk
├─sdb1             8:17   0   511M  0 part  /boot
└─sdb2             8:18   0  59.1G  0 part
  └─ssdvg        254:0    0  59.1G  0 crypt
    ├─ssdvg-swap 254:1    0     4G  0 lvm   [SWAP]
    └─ssdvg-root 254:2    0  55.1G  0 lvm   /

Final Optimizations

Everything is good and working, but there are a few final things to do due to the SSD.

1. TRIM support. There are several layers between the filesystem and the SSD that all have to cooperate. Going top-down, the filesystem itself needs to support TRIM. ext4 does, but I opt to not do it automatically (with a discard mount option), but to do the TRIM-ing in bulk with fstrim. Then LVM needs to support it. It does after enabling an option in /etc/lvm/lvm.conf. Finally, on the LUKS level, it is with the allow-discards option used for gummiboot above. I tried # fstrim -v / and it said 46 GiB trimmed, which was my free space on /. I could add this as a cron job, but I just do it manually every week or so when I remember.
2. IO scheduler. I change to the deadline IO scheduler for the SSD, but not for the HDD. This is easily done by adding a udev rule, as detailed at the Arch wiki.
3. SSD swappiness. Since the swap file is on the SSD, I have plenty of RAM, so I don't want to use swap unless absolutely necessary (and for hibernation). This is done by changing the swappiness value.

Final Observations

I've finished the installation a couple of months ago, so I've had some time to observe the changes. Honestly, I don't notice performance to be that much different. I mean, it's nice to have the extra room, but speed-wise things don't seem that much different to me. Except for one thing: when I hibernate and resume, it's noticeably faster. I've not timed it, but I'm quite sure it's no placebo effect.