AWS EC2 Amazon Linux or Where is my ephemeral storage?
I like Amazon Linux. It is basically Centos with a bit of Fedora mixed in. Thus it comes with yum, and even better yet, with amazon supported repositories preconfigured. Because amazon provides it, they support it, and answer questions about it, and fix problems with it and keep it up to date and patched so that it runs well as a paravirtualized guest inside Amazon's Xen based infrastructure. They insure that it's fairly secure, has a fairly minimal set of installed packages, and comes with the ec2 api tools already installed, which can be a daunting task to setup for people new to aws and ec2.
However one thing they don't do by default is configure their AMI's to make the ephemeral storage you are entitled to available by default. When you boot up an instance, you find yourself with a 10 GB EBS volume, and that can become pretty filled once you've installed a few packages.
What is Ephemeral storage?
Ephemeral storage is the "instance storage" that's advertised for each instance type and can range from 150 GB on up to over a terabyte depending on the instance type you're running. This storage is called "Ephemeral" by Amazon because it comes from the local hard drives in the server your instance is running on, but does not persist or survive an instance stop. If you put anything important on it, you'll need to back that data up using traditional means, and won't have the ability to snapshot it like you can with EBS volumes. Although ephemeral drives can provide excellent IO performance and a substantial storage at no extra cost, they are subject to the types of failures that you'd expect with single server disks.
In spite of these concerns, they certainly are a great place for things like temporary files, or swap files, or logs that you're going to rotate or purge anyways.
So how can you use Amazon Linux and still get access to the ephemeral disks?
You need to utilize the Amazon API and use the "-b" parameter to map one or more block devices to your ephemeral storage. Although the web interface "Launch Instance" wizard includes a tab for storage options, the instance storage tab is not accessible. I typically use the command line api to start aws instances.
For example, this command will launch a large instance based on an Amazon Linux AMI, only, while at the same time adding back mappings to the ephemeral storage that is part of the Large instance profile.
ec2-run-instances --region us-east-1 ami-aecd60c7 -k yourkey -t "m1.large" -z us-east-1d -g your-security-group -b "/dev/sdb=ephemeral0" -b "/dev/sdc=ephemeral1"
The important parameters to note, are the two "-b" parameters, which specify that ec2 should establish block device mappings to our two ephemeral drives.
When your instance starts, you'll find that you now have a formated ext3 volume ready for use:
You might have noticed that we specified a mapping for 2 different block devices. You'll find that /dev/sdc is configured, but you'll have to format it and mount it yourself.
If this is the first time you've seen the command line api used, amazon linux is a great way to get started. Just start up a micro instance using the recommended Amazon Linux AMI, add your certs, and you're ready to go.
In summary, Amazon Linux is an excellent choice for your EBS backed instances -- just remember that unless you override the default, you'll be missing out on the Instance storage you're paying for!
However one thing they don't do by default is configure their AMI's to make the ephemeral storage you are entitled to available by default. When you boot up an instance, you find yourself with a 10 GB EBS volume, and that can become pretty filled once you've installed a few packages.
What is Ephemeral storage?
Ephemeral storage is the "instance storage" that's advertised for each instance type and can range from 150 GB on up to over a terabyte depending on the instance type you're running. This storage is called "Ephemeral" by Amazon because it comes from the local hard drives in the server your instance is running on, but does not persist or survive an instance stop. If you put anything important on it, you'll need to back that data up using traditional means, and won't have the ability to snapshot it like you can with EBS volumes. Although ephemeral drives can provide excellent IO performance and a substantial storage at no extra cost, they are subject to the types of failures that you'd expect with single server disks.
In spite of these concerns, they certainly are a great place for things like temporary files, or swap files, or logs that you're going to rotate or purge anyways.
So how can you use Amazon Linux and still get access to the ephemeral disks?
You need to utilize the Amazon API and use the "-b" parameter to map one or more block devices to your ephemeral storage. Although the web interface "Launch Instance" wizard includes a tab for storage options, the instance storage tab is not accessible. I typically use the command line api to start aws instances.
For example, this command will launch a large instance based on an Amazon Linux AMI, only, while at the same time adding back mappings to the ephemeral storage that is part of the Large instance profile.
ec2-run-instances --region us-east-1 ami-aecd60c7 -k yourkey -t "m1.large" -z us-east-1d -g your-security-group -b "/dev/sdb=ephemeral0" -b "/dev/sdc=ephemeral1"
The important parameters to note, are the two "-b" parameters, which specify that ec2 should establish block device mappings to our two ephemeral drives.
When your instance starts, you'll find that you now have a formated ext3 volume ready for use:
/dev/xvdb 414G 199M 393G 1% /media/ephemeral0
You might have noticed that we specified a mapping for 2 different block devices. You'll find that /dev/sdc is configured, but you'll have to format it and mount it yourself.
If this is the first time you've seen the command line api used, amazon linux is a great way to get started. Just start up a micro instance using the recommended Amazon Linux AMI, add your certs, and you're ready to go.
In summary, Amazon Linux is an excellent choice for your EBS backed instances -- just remember that unless you override the default, you'll be missing out on the Instance storage you're paying for!