Skip to content

AWS EC2 Amazon Linux or Where is my ephemeral storage?

I like Amazon Linux. It is basically Centos with a bit of Fedora mixed in. Thus it comes with yum, and even better yet, with amazon supported repositories preconfigured. Because amazon provides it, they support it, and answer questions about it, and fix problems with it and keep it up to date and patched so that it runs well as a paravirtualized guest inside Amazon's Xen based infrastructure. They insure that it's fairly secure, has a fairly minimal set of installed packages, and comes with the ec2 api tools already installed, which can be a daunting task to setup for people new to aws and ec2.

However one thing they don't do by default is configure their AMI's to make the ephemeral storage you are entitled to available by default. When you boot up an instance, you find yourself with a 10 GB EBS volume, and that can become pretty filled once you've installed a few packages.

What is Ephemeral storage?
Ephemeral storage is the "instance storage" that's advertised for each instance type and can range from 150 GB on up to over a terabyte depending on the instance type you're running. This storage is called "Ephemeral" by Amazon because it comes from the local hard drives in the server your instance is running on, but does not persist or survive an instance stop. If you put anything important on it, you'll need to back that data up using traditional means, and won't have the ability to snapshot it like you can with EBS volumes. Although ephemeral drives can provide excellent IO performance and a substantial storage at no extra cost, they are subject to the types of failures that you'd expect with single server disks.

In spite of these concerns, they certainly are a great place for things like temporary files, or swap files, or logs that you're going to rotate or purge anyways.

So how can you use Amazon Linux and still get access to the ephemeral disks?

You need to utilize the Amazon API and use the "-b" parameter to map one or more block devices to your ephemeral storage. Although the web interface "Launch Instance" wizard includes a tab for storage options, the instance storage tab is not accessible. I typically use the command line api to start aws instances.

For example, this command will launch a large instance based on an Amazon Linux AMI, only, while at the same time adding back mappings to the ephemeral storage that is part of the Large instance profile.


ec2-run-instances --region us-east-1 ami-aecd60c7 -k yourkey -t "m1.large" -z us-east-1d -g your-security-group -b "/dev/sdb=ephemeral0" -b "/dev/sdc=ephemeral1"
 


The important parameters to note, are the two "-b" parameters, which specify that ec2 should establish block device mappings to our two ephemeral drives.

When your instance starts, you'll find that you now have a formated ext3 volume ready for use:

/dev/xvdb 414G 199M 393G 1% /media/ephemeral0


You might have noticed that we specified a mapping for 2 different block devices. You'll find that /dev/sdc is configured, but you'll have to format it and mount it yourself.

If this is the first time you've seen the command line api used, amazon linux is a great way to get started. Just start up a micro instance using the recommended Amazon Linux AMI, add your certs, and you're ready to go.

In summary, Amazon Linux is an excellent choice for your EBS backed instances -- just remember that unless you override the default, you'll be missing out on the Instance storage you're paying for!

Centos, Amazon Linux, NFS and the dreaded "nobody" problem

I have been fighting a problem with mounting volumes from an NFS Server in AWS for a few days. With more pressing issues at hand, I had to try and google for solutions for an hour or so before bed, and nothing I was doing was having any effect. Curiously, an Ubuntu based machine that was mounting the drive using NFS3 was not having the same problem. Only the Amazon Linux servers that were using NFS4 were having the issue, and were showing all files and directories as being owned by nobody:nobody.


drwxr-xr-x    2 nobody nobody   22 Jan  9 19:58 installervc
drwxr-xr-x    2 nobody nobody 4.0K Jan  9 19:56 avatar
drwxr-xr-x    2 nobody nobody 4.0K Jan  9 19:56 accessories
 


I had previously insured that the user UID and group GID for the user that would be writing files (in my case "apache" was the same (with the same UID and GID) on the NFS server and the servers mounting the nfs volume.

As it turned out the problem was with the configuration (or lack thereof) of the rpcidmapd service. NFS4 relies on this service to map users between machines. The "idmapd" requires that the domain of both the client and server should match for the UID/GID mapping to work, and in my case it wasn't. Probably many people with proper DNS configuration don't hit this problem, but we did not have a proper DNS setup, as these machines are part of a growing cluster. Compounding the problem I had set the configuration files to have meaningless host names rather than a domain.

You can tweak this setup by editing the: /etc/idmapd.conf file, and find the "Domain" variable:


Domain =  yourdomain.com
 


Set these to be the same for the server and all the clients.

The last problem was that I had to restart the idmapd process, which has an /etc/init.d control script named /etc/rpcidmapd


[root@web1 init.d]# ls -lath | grep rpcid
-rwxr-xr-x  1 root root 2.7K Jul 28  2011 rpcidmapd
 


Restart the process on both the nfs server and any nfs clients. If the source of your problem is the same as mine, your user & group mapping problems should be solved:


sudo /etc/inti.d/rpcidmapd restart
 


Finally!!!!


drwxr-xr-x    2 apache apache   22 Jan  9 19:58 installervc
drwxr-xr-x    2 apache apache 4.0K Jan  9 19:56 avatar
drwxr-xr-x    2 apache apache 4.0K Jan  9 19:56 accessories
 

Defined tags for this entry: , , , , , , , ,

Calculate a person's age in a MySQL query -- continued

The serendipity cache blew up on me for this article so I had to split it into 2 parts.

DATE_ADD to the rescue


MySQL knows how to work with the Gregorian calender, and I believe it is easier to let MySQL do the work, rather than trying to figure out how to implement the different cases. I chose to implement the "last day of the month" technique. In this case, we'll do this by creating a mysql date based on March 1st of the current year, and subtract one day from it to get to the last day of February.

If you read my article on "Finding next monday" using mysql date functions, you would have seen how DATE_ADD() can be used to help solve a lot of problems, even though the name can be misleading when you are actually using it to subtract. Using it with the following test dates proves that mysql has fully implemented the leap year logic accurately:

Continue reading "Calculate a person's age in a MySQL query -- continued"

Calculate a person's age in a MySQL query

Recently a question was posed on the Phpfreaks.com forums, as to the best way for someone to store a user's birth date using the MySQL database. One person suggested using a varchar as they were most familiar with string functions. My answer was to use the MySQL date type.

I provided a quick "advantages of using date" comparison list:
Storing as a DATE
•A MySQL Date requires 3 Bytes of storage
•You can do Date arithmetic inside mysql (search for dates within ranges) and use mysql functions to calculate values directly in a query
•It intrinsically will only store valid dates
•You can format it in numerous ways

Storing as a Varchar
•A string will require minimum 8 bytes, or 10 with separators
•can't do any form of efficient native range queries
•can't reformat it easily in SQL
•will allow completely invalid dates

I thought a great proof of the benefit of this approach was to show that you could have MySQL calculate the person's current age in a query, using their birthday. Certainly with a string, using PHP for example, you would typically query the database to pull out the string and for each row, turn it into a PHP date and do some calculations in your code, and you'd arrive at the same place, but I wanted to show just how capable SQL -- and in this case MySQL can be especially when you use the native data types and some functions.
Continue reading "Calculate a person's age in a MySQL query"

SQL UPDATE for strings in MongoDB

MongoDB currently does not offer the equivalent of a SQL Update statement when you want to change the value of a string to something that is based on the current value. In my case I needed to find spaces and change them to underscores -- something done easily in php with str_replace(), and in MySQL with REPLACE.

If you had a table named foo with a column named "mystr" you could do this sort of update like this:


UPDATE foo SET mystr = REPLACE(mystr, ' ', '_');
 


MongoDB does have collection.update() that works with a series of "modifier" operations, but none of them are useful for doing modifications to the current value of a document field when that field is a string.

MongoDB does however support javascript, and the javascript replace() function implements a regular expression search and replace. I was able to accomplish the update using the forEach() method to process the resulting documents one at a time. While not as efficient as having mongo do this internally with a specially built update modifier, at least this method only needs to operate on documents that are known to require modifications, and could be run in the mongo interactive client easily.

Again, assuming the collection is named "foo" and the document field is named "mystr"


db.foo.find({ mystr: /[ ]+/ }).forEach( function(u) { u.mystr = u.mystr.replace(/[ ]/g, "_"); db.foo.save(u); } );
 


An existing patch promises to add a new update() modifier but until such a time as it exists, this approach can be used when you need to update strings in a collection, referencing the existing values in the transformation.
Defined tags for this entry: , , , , , ,

Load the Url symfony helper in a model or form class

I'm doing some symfony work and needed to use the Url helper in a form class to return an error message that included a link. I would consider this one of those exceptions where you really need to be able to have markup in a class rather than in a template.

I should probably mention that this is the way that works in version 1.4.3, and that the method that works in actions will not work in a form class. That method is:


sfContext::getInstance()->getConfiguration()->loadHelpers(...);
 


That method was deprecated and there is also this article to consider.

In this case, the solution was to access the sfApplicationConfiguration object. In order to use the Url helper so you can call link_to() you need to also include the 'Tag' helper.


sfApplicationConfiguration::getActive()->loadHelpers(array('Url', 'Tag'));

// now you can use link_to(...)
 


Obviously this would allow you to load other helpers in the same way ... i18n for example.