Unfinished Development Web Development, Infrastructure & Testing

Creating a custom CentOS 6.6 AMI

Whilst there are plenty of options for a CentOS AMI in the AWS Marketplace it can be useful to create you own base image.

One of the reasons we ended up creating our own image was due to the restrictions on resizing the root filesystem of the Marketplace images, which became a real issue for us when automating the creation of instances through Foreman's EC2 Compute plugin; we needed multiple base images with different sized root volumes available.

Creating your own AMI is not an especially complex task, although it does involve a fair few number of steps. In this guide we'll create on image on an empty filesystem that we mount be loopback. We'll then convert the image we create into an EBS backed image.

We are going to perform the following to get our EBS backed AMI into AWS:

  1. Gather the required information
  2. Setup our environment
  3. Prepare the image
  4. Build the AMI
  5. Convert the AMI to an EBS Backed AMI

What you'll need before starting

  • You need an AWS account (obviously) which has access to the EC2 and S3 services, with the ability to create new instances and volumes
  • You will need to know the account number of the AWS account. This is normally displayed in the top right hand corner of the console
  • You need an AWS Access Key and Secret Key. View the AWS documentation for details on creating these.
  • You are also going to need a Signing certificate and private key, with the certificate added to your AWS user. Again, the AWS documentation is very helpful here
  • You will need a build machine on which to create the image with enough space available on a volume to create them image - this guide uses 2GB. I used a new t2.micro image with a marketplace CentOS image (ami-30ff5c47 in Ireland region). The main reason I used this was it increased the speed I could download packages and upload bundles into AWS but any install of a RHEL derivative will work.

Setting Up the Environment

After logging in to your build machine, you need to set a few environment variables and install the AWS Command Line tools.

cp /root/.bashrc /root/.bashrc.bak
vi /root/.bashrc

# .bashrc
# User specific aliases and functions
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

export EC2_HOME=/opt/ec2/tools
export EC2_URL=https://ec2.amazonaws.com
export AWS_ACCOUNT_NUMBER=<THE ACCOUNT NUMBER>
export AWS_ACCESS_KEY=<YOUR ACCESS KEY>
export AWS_SECRET_KEY=<YOUR SECRET KEY>
export EC2_CERTIFICATE=/opt/ec2/certificates/ec2-cert.pem
export EC2_PRIVATEKEY=/opt/ec2/certificates/ec2-pk.pem
export AWS_AMI_BUCKET=<AWS S3 BUCKET PATH TO UPLOAD BUNDLE TO>
export AWS_DEFAULT_REGION=<REGION TO CREATE IMAGE IN>
export PATH=$PATH:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:$EC2_HOME/bin
export JAVA_HOME=/usr

# Source global definitions
if [ -f /etc/bashrc ]; then
  . /etc/bashrc
fi

source /root/.bashrc

I found it easier to work through the steps with SELinux disabled. If you do not want to disable it, you will be on your own if steps don't work due to SELinux.

setenforce 0

We'll need a few packages installed on the build machine, which are required to create the image

yum install e2fsprogs ruby java-1.7.0-openjdk unzip MAKEDEV

Copy your signing certificate and private key onto the build machine. First create the directory on the build machine

mkdir -p /opt/ec2/certificates

I then used SCP to transfer the certificate and private key file. You could of course create the certificate and private key on the build machine directly and add that into AWS

scp <your certificate file> root@<your_instance_ip>:/opt/ec2/certificates/ec2-cert.pem

scp <your private key file> root@<your_instance_ip>:/opt/ec2/certificates/ec2-pk.pem

The last steps in setting up our build environment is to install the EC2 API and AMI tools. First, download and install the API tools

mkdir -p /opt/ec2/tools
curl -o /tmp/ec2-api-tools.zip http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
unzip /tmp/ec2-api-tools.zip -d /tmp
cp -r /tmp/ec2/api-tools-*/* /opt/ec2/tools

And finally, download and install the AMI tools

mkdir -p /opt/ec2/tools
curl -o /tmp/ec2-ami-tools.zip http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip
unzip /tmp/ec2-ami-tools.zip -d /tmp
cp -r /tmp/ec2/ami-tools-*/* /opt/ec2/tools

Preparing the Image

In this section we will prepare an image that can bundled into an Instance Store backed AMI on an empty filesystem that we have mounted via loopback. The AMI will be created will involve doing a full operating system install on a clean root filesystem.

We start by creating a disk image with an empty ext4 filesystem. We then mount it via loopback, which allows us to use a normal file as a raw device. This presents it as part of the normal filesystem, so we can use our usual tools to manage and modify it. Think if it as a filesystem within a file.

mkdir -p /opt/ec2/images
dd if=/dev/zero of=/opt/ec2/images/centos-6.6-x86_64-base.img bs=1M count=2048
mkfs.ext4 -F -j -L 'ROOTFS' /opt/ec2/images/centos-6.6-x86_64-base.img
mkdir -p /mnt/ec2-image
mount -o loop /opt/ec2/images/centos-6.6-x86_64-base.img /mnt/ec2-image

Before we can install the OS we need to prepare the image by creating the required directories in the root filesystem to hold the system files and devices

mkdir -p /mnt/ec2-image/{dev,etc,proc,sys}
mkdir -p /mnt/ec2-image/var/{cache,log,lock,lib/rpm}

Next up, we populate the /dev directory with a minimal set of devices. You can safely ignore any MAKEDEV: mkdir: File exists warning you see here

/sbin/MAKEDEV -d /mnt/ec2-image/dev -x console
/sbin/MAKEDEV -d /mnt/ec2-image/dev -x null
/sbin/MAKEDEV -d /mnt/ec2-image/dev -x zero
/sbin/MAKEDEV -d /mnt/ec2-image/dev -x urandom

Mount dev, pts, shm, proc and sys into the new root filesystem

mount -o bind /dev /mnt/ec2-image/dev
mount -o bind /dev/pts /mnt/ec2-image/dev/pts
mount -o bind /dev/shm /mnt/ec2-image/dev/shm
mount -o bind /proc /mnt/ec2-image/proc
mount -o bind /sys /mnt/ec2-image/sys

Install the Operating System

We're now ready to install the operating system. We will create a Yum configuration file that we will use to install the base OS. This file can be located anywhere on the main filesystem (not on the loopback filesystem) and is only used during the base installation.

mkdir -p /opt/ec2/yum
vi /opt/ec2/yum/yum-xen.conf

[base]
name=CentOS-6 - Base
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

[updates]
name=CentOS-6 - Updates
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=updates
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

[extras]
name=CentOS-6 - Extras
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=extras
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6

[centosplus]
name=CentOS-6 - Plus
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=centosplus
gpgcheck=1
enabled=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

[contrib]
name=CentOS-6 - Contrib
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=contrib
gpgcheck=1
enabled=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

Install the base package and other required packages into the new image. I am installing cloud-init here (which is why I added the EPEL repository above). This isn't a required package, however if you want to be able to provide userdata to your instance then you will need this.

yum -c /opt/ec2/yum/yum-xen.conf --installroot=/mnt/ec2-image -y groupinstall Base
yum -c /opt/ec2/yum/yum-xen.conf --installroot=/mnt/ec2-image -y install *openssh* dhclient grub e2fsprogs yum-plugin-fastestmirror.noarch selinux-policy selinux-policy-targeted vi cloud-init

This is the point you can add any other packages you require to your new image. Remember, however, you may need to manually add the repository to the yum configuration file we created and you will need to provide the --installroot parameter.

Configure the Operating System

After the successful install of the base OS, the next step is to configure the networking, volumes, security settings and any other services you installed

Login Script

First up, we'll create a shell login script for the root user account

vi /mnt/ec2-image/root/.bashrc

# .bashrc

# User specific aliases and functions
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi
vi /mnt/ec2-image/root/.bash_profile

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

# Use specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH

Networking

Now we need to configure the networking options

vi /mnt/ec2-image/etc/sysconfig/network

NETWORKING=yes
HOSTNAME=localhost.localdomain
vi /mnt/ec2-image/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"
NM_CONTROLLED="yes"
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no

NOTE: Amazon EC2 DHCP servers ignore hostname requests. If you attempt to set DHCP_HOSTNAME it will be used for the local hostname and set on the instances, but not externally. Also, the local hostname will be the same for every instance of the AMI, which may prove confusing

Ensure that the the network service will be started on boot

/usr/sbin/chroot /mnt/ec2-image /sbin/chkconfig --level 2345 network on

We also want to ensure that the SSH service starts on boot

/usr/sbin/chroot /mnt/ec2-image /sbin/chkconfig --level 2345 sshd on

SELinux

By default, SELinux is set to enforcing; however in certain cases it doesn't get labeled correctly, so we'll assume that for the first boot of the instance it is not properly labelled. We can force labelling on first boot by adding a .autorelabel file

touch /mnt/ec2-image/.autorelabel

If you decide to disable SELinux, you will need to edit the /mnt/ec2-image/etc/sysconfig/selinux file and set it to DISABLED

vi /mnt/ec2-image/etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled

# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted

Cloud-Init

If you have decided to use cloud-init, you will need to configure it. There are a lot of options for cloud-init and they are beyond the scope of this post, however below is a very simple configuration that will allow us to log in as root and run basic commands in our userdata

vi /mnt/ec2-image/etc/cloud/cloud.cfg

disable_root: false
manage_etc_hosts: true
syslog_fix_perms: null

datasource_list: [Ec2]
datasource:
    Ec2:
        metadata_urls: ['http://169.254.169.254']

cloud_init_modules:
    - bootcmd
    - set_hostname
    - update_hostname
    - update_etc_hosts

cloud_config_modules:
    - runcmd

cloud_final_modules:
    - scripts-per-once
    - scripts-per-boot
    - scrits-per-instance
    - scripts-user
    - phone-home
    - final-message
    - power-state-change

system_info:
    distro: rhel
    ssh_svcname: sshd

Remember, the cloud.cfg file is YAML and that is extremely sensitive to whitespace. Ensure your file is valid.

Disks & Volumes

There are lots of options for mounting instance storage volumes but the aim of this post is to create an EBS backed AMI, so I am going to keep the disk configuration as simple as possible

Create an fstab file for the new image. We'll mount the root file system by label

vi /mnt/ec2-image/etc/fstab

LABEL=ROOTFS    /          ext4      defaults        1 1
none            /dev/pts   devpts    gid=5,mode=620  0 0
none            /dev/shm   tmpfs     defaults        0 0
none            /proc      proc      defaults        0 0
none            /sys       sysfs     defaults        0 0

Now we need to create a grub configuration file for the image and boot settings so that the Amazon Kernel Image (AKI) can boot into the new image kernel

vi /mnt/ec2-image/boot/grub/grub.conf

default=0
timeout=0
title CentOS 6 (x86_64)
root (hd0)
kernel /boot/vmlinuz ro root=LABEL=ROOTFS
initrd /boot/initramfs

Create a symlink to menu.lst in the new image

ln -s /boot/grub/grub.conf /mnt/ec2-image/boot/grub/menu.lst

Next, we need to modify the grub.conf file to add the correct kernel version

kern=`ls /mnt/ec2-image/boot/vmlin*|awk -F/ '{print $NF}'`
ird=`ls /mnt/ec2-image/boot/initramfs*.img|awk -F/ '{print $NF}'`
sed -ie "s/vmlinuz/$kern/" /mnt/ec2-image/boot/grub/grub.conf
sed -ie "s/initramfs/$ird/" /mnt/ec2-image/boot/grub/grub.conf

SSH

Modify the sshd configuration for the new image to allow root login via private key authentication only and disable DNS lookups. Feel free to make any further changes you wish to sshd configuration but remember if you block access for the root user you'll have no way of connecting to your instance

vi /mnt/ec2-image/etc/ssh/sshd_config

UseDNS no
PermitRootLogin without-password

Create a script that will run on boot and capture the SSH key for your root user you provided when creating the instance

vi /mnt/ec2-image/etc/rc.local

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# set a random pass on first boot
if [ -f /root/firstrun ]; then
   dd if=/dev/urandom count=50|md5sum|passwd --stdin root
   passwd -l root
   rm /root/firstrun
fi
if [ ! -d /root/.ssh ]; then
   mkdir -m 0700 -p /root/.ssh
   restorecon /root/.ssh
fi

# Get the root ssh key setup
ReTry=0
while [ ! -f /root/.ssh/authorized_keys ] && [ $ReTry -lt 5 ]; do
   sleep 2
   curl -f http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > /root/.ssh/authorized_keys
   ReTry=$[Retry+1]
done
chmod 600 /root/.ssh/authorized_keys && restorecon /root/.ssh/authorized_keys

Cleanup

Clean up the file system of the new image and unmount the directories

yum -c /opt/ec2/yum/yum-xen.conf --installroot=/mnt/ec2-image -y clean packages
rm -rf /mnt/ec2-image/root/.bash_history
rm -rf /mnt/ec2-image/var/cache/yum
rm -rf /mnt/ec2-image/var/lib/yum
umount /mnt/ec2-image/dev/shm
umount /mnt/ec2-image/dev/pts
umount /mnt/ec2-image/dev
umount /mnt/ec2-image/sys
umount /mnt/ec2-image/proc
umount /mnt/ec2-image

Build the AMI

When we register the AMI with AWS we need to set the default kernel to one which supports the GRUB boot loader. Amazon has published Amazon Kernel Images (AKIs) that use a system called PV-GRUB, which is basically a boot manager for XEN virtual machines.

In order to find the latest available AKI we can use the AWS CLI tools

ec2-describe-images --owner amazon | grep "amazon\/pv-grub-hd0" | awk '{ print $1, $2, $3, $5, $7 }'

So long as you have used the setting / templates above you can use an AKI with hd0 in the name. If you have added a partition table, then you will need hd00 in the name. You also need to ensure that you choose an AKI for the correct architecture, in our case x86_64. When I wrote this post the latest suitable AKI for eu-west-1 was aki-52a34525.

Next up we will bundle the image so that we can upload it to our S3 bucket. This command uses the signing certificate and key we set up earlier to you need to ensure you have followed that part

ec2-bundle-image --cert $EC2_CERTIFICATE --privatekey $EC2_PRIVATEKEY --user $AWS_ACCOUNT_NUMBER --image /opt/ec2/images/centos-6.6-x86_64-base.img --prefix CentOS-6.6-Base-x86_64 --destination /opt/ec2/images --arch x86_64 --kernel aki-52a34525 --region $AWS_DEFAULT_REGION

Make sure you set the correct kernel ID here

We need to upload the bundle into S3 so that we can register it with AWS. You may need to ensure that the bucket path you set already exists in S3

ec2-upload-bundle -b $AWS_AMI_BUCKET -a $AWS_ACCESS_KEY -s $AWS_SECRET_KEY -m /opt/ec2/images/CentOS-6.6-Base-x86_64.manifest.xml --region $AWS_DEFAULT_REGION

After the upload completes we can register the AMI with EC2 - once again, ensure you set the correct kernel ID here

ec2-register $AWS_AMI_BUCKET/CentOS-6.6-Base-x86_64.manifest.xml --name "CentOS 6.6 (x86_64)" --description "Base CentOS 6.6 AMI" --architecture x86_64 --kernel aki-52a34525 --region $AWS_DEFAULT_REGION

If all of that ran through successfully, you should now be able to launch a new instance of the AMI, using either the CLI tools or AWS Console. The unique ID of your AMI should have been returned by the ec2-register command above.

Convert to an EBS Backed AMI

There doesn't appear to be a simple command to convert an instance store backed AMI to an EBS backed AMI. Again, it's not an overly complex task but does involve more steps than I would like.

I won't go into as much detail here when it comes to launching instances and attaching volumes, that is heavily documented in a number of places.

  1. Launch a new instance of your instance store based AMI
  2. Create a new EBS volume of the size you wish your new images root volume to be
  3. Attach the new EBS volume to your instance that you launched
  4. Log in to your new instance

Now we need to prepare our new volume and copy the image onto it. First up we need to find our new volume and create an ext4 filesystem on it.

/bin/egrep '[xvsh]d[a-z].*$' /proc/partitions

202 65 2097152 xvde1
202 80 4188672 xvdf
202 144 16777216 xvdj

Now we can make a filesystem on our volume

mkfs.ext4 /dev/xvdj

Create a mount point and mount our EBS volume

mkdir -p /opt/ec2/mnt
mount -t ext4 /dev/xvdj /opt/ec2/mnt

We now want to sync our root and dev filesystems to the new volume

rsync -avHx / /opt/ec2/mnt
rsync -avHx /dev /opt/ec2/mnt

Remove the authorized_keys file for the root user so we can add a different key at launch

rm /opt/ec2/mnt/root/authorized_keys

Next, we label the volume so that it matches the fstab file we created earlier

tune2fs -L 'ROOTFS' /dev/xvdj

Flush all of the writes and unmount the volume

sync;sync;sync;sync
umount /opt/ec2/mnt

Now we can create our EBS image

  1. Detach the EBS volume from the instance
  2. Create a snapshot of the volume
  3. Create an image from the snapshot. You need to ensure you select the correct kernel version again when creating the image, whether through CLI tools or AWS Console

You now have an EBS backed AMI that you can launch into new instances. If you create a new instance with a larger root volume that the image was created with (you cannot create instances with a smaller volume) then you may need to resize the volume.

resize2fs /dev/xvda1

Alternatively you could create further EBS backed AMIs based on your original instance store backed AMI with the required root volume sizes already configured. Just follow the steps to convert your AMI again, setting the correct volume size.

Phew...

That was a long post. As I said at the start, it's not a massively complex task but it's a long winded task. Hopefully you found this post helpful and you can go and start creating AMIs to meet your exact needs.