Cost analysis: Storage costs in AWS

Earlier this week I was working with a cloud business to review their expenditure in AWS. The business had advised me of 3 challenges around their storage:

  1. Cost – EBS storage represented over 90% of the MySQL estate cost.
  2. Performance – The business was regularly purchasing more storage capacity than they needed (AWS EBS provides a baseline performance of 3 IOPS per GiB).
  3. Cloning – Cloning large 10TB databases resulted in reduced performance of the destination EBS volume whilst the blocks were rehydrated from the source S3 snapshot. It has taken 5 days in the past until full throughput was available to the system.

This was a really interesting experience that I’ve learnt a lot from, so I decided to take it further and do some modelling of the costs using 5 x MySQL servers each with 11TB of EBS storage attached:

In my modelling, the storage cost represented 94.3% of the total cost of running the 5 MySQL servers.

Now it’s no secret that I work for a company that develops a cloud storage platform that overlays native EBS storage and adds storage efficiencies such as deduplication and compression, plus the added advantage of instant cloning with no performance warm up time.

So, here’s how that same architecture looks with the ONTAP Cloud storage running:

This first thing you’ll notice is that you no longer have to manage the EBS volumes, ONTAP takes care of this for you. That means no worrying about raid types, capacity or performance on a host-to-host basis. Win.

The second benefit is reduced costs. ONTAP brings deduplication and compression to the party. Your mileage will vary depending on your data’s suitability for storage efficiencies but it’s not uncommon to see >30% space savings. Win.

If we apply that logic to this scenario, our resulting storage expenditure will be reduced from £5,188.92/mo to £3,329.56/mo. That’s a saving of £1859.36/mo!

But wait, this all assumes that I am not using the instant cloning functionality that ONTAP has. This scenario has 2 development servers (dev/test) that currently use EBS snapshots (S3). This is costly as each clone of the 10TB source database results in the full space consumption and costs from AWS. In comparison, clones created with ONTAP consume no additional space, therefore the cost of clones is significantly reduced. If the customer changes anything within the clone, those writes are the overall storage cost – again, significantly less that doing a full copy based clone from EBS to S3 back to EBS.

So what does this now look like if we start to use storage efficiencies AND instant cloning for the two dev/text databases?

The storage cost has reduced even further, from our original £5,188.92/mo to £1997.73/mo! That’s a saving of £3,191.19/mo or £38,294.28/year.

That’s all well and good, but of course ONTAP has a licencing cost, what would our final savings look like when we take that into account. I’m basing this on the pay & go licencing (most expensive option) over 12 months:

ONTAP Cloud Pay & Go (R3.2xlarge) / 12 months = $19,307 ~ £15,183.22/year

Therefore, the customer would be in line for a total saving of £23,111.06/year, saving almost 35% of their overall AWS cost, pretty impressive!

This solves the cost problem for the business I was working with, and also fixes the performance issue that they currently experience when working with native AWS cloning.

Not only that, ONTAP cloud has APIs, PowerShell, Kubernetes and CLI management, meaning that it can be seamlessly integrated into their developer toolsets and processes.

So, the customer is now able to rapidly develop in AWS with on-demand clones and further reduce their storage footprint with storage efficiencies.

So, it’s a no-brainer – try it today – free of charge for 30 days here.

Amazon Linux: Migrate MySQL from EBS to iSCSI

This week, I was working with a 100% cloud business, who have been using  AWS from their incorporation over 5 years ago.

The business wanted to take advantage of the instant cloning capabilities offered by running ONTAP in AWS and migrate the existing MySQL databases to the new cloud storage system.

These steps will get you up and running quickly, including some handy gotchas that you may run into.

First, I assume that you already have your Amazon Linux (or other instance) up and running and that you have your iSCSI devices mapped and mounted in your instance. Don’t worry if you haven’t – I created a handy post here that covers it.

Let’s check the current location of your MySQL datadir, there are two methods for checking this:

Method #1: Check within MySQL

mysql -u root -p

select @@datadir;

Method #2: Check from the console and look at the current value of datadir (for example datadir=/var/lib/mysql)

sudo cat /etc/my.cnf

Now that we know the current data directory we can move the database to the new location with these steps. In this example, my iSCSI device is mounted to /mnt/ontap

Shutdown the MySQL service (this ensures data and application consistency):

sudo service mysqld stop

Next, copy the database to the new location:

sudo rsync -av /var/lib/mysql /mnt/ontap

Once complete it’s best to rename the old directory to .bak to prevent confusion:

sudo mv /var/lib/mysql /var/lib/mysql.bak

Now it’s time to change the working directory path to the new one:

sudo nano /etc/my.cnf


It would seem natural at this point to start the service, however, there is at least one more step:

(Optional: If you are using AppArmour):

sudo nano /etc/apparmor.d/tunables/alias

At the bottom of the file, add the following alias rule:

. . .
alias /var/lib/mysql/ -> /mnt/netapp1/mysql/,
. . .

Restart the AppArmour servicesudo

sudo service restart apparmour

Now for the final step, MySQL runs an environment check script upon service start. The script simply checks for the existence of /var/lib/mysql and /var/lib/mysql/mysql . We need to create a minimal directory structure to pass the environment check:

sudo mkdir /var/lib/mysql/mysql -p

That’s it! Now we can start the MySQL service:

sudo service mysqld start

Finally, let’s check that the new data directory is indeed in use:

mysql -u root -p

select @@datadir

Thanks for reading, I hope you found this useful!

Amazon Linux: iSCSI – Install and connect to your storage

iSCSI is a great way to attach enterprise class storage to your EC2 instances and it’s very easy to setup, even if you have zero storage management experience. The best part is that all of the steps are completed directly within your Linux instance itself.


  1. You’ll need some iSCSI storage within your VPC. This is really easy to deploy with ONTAP Cloud – grab yourself a free 30-day trial.
  2. An instance running in your VPC that you can SSH to.

Simply follow the steps below:

First, log into your instance via SSH/terminal (most use PuTTy for this).

Update your packages (Optional – but best practice):

sudo yum update -y

Next, we install iSCSI into our host:

sudo yum install iscsi-initiator-utils

We will now discover the iSCSi system (I’m using ONTAP in AWS, but you can use it in Azure, whitebox or on-prem). Simply replace with your own IP address.

iscsiadm -m discovery -t st -p

Optional: Heres how to find your iSCSI IP address for ONTAP.  SSH to your ONTAP management IP ( in this example):

ssh admin@
network interface show

Next, confirm that your host is seeing the iSCSI portal(s) correctly. they will be listed with the following command:

iscsiadm -m node

iSCSI requires your host to log in to the discovered portal, simply run the following command:

iscsiadm -m node --targetname "" --portal "<ip-address:port>" --login

Optional but recommended: Restart the iSCSI service:

/etc/init.d/iscsi restart

Now, make a note of the initiator name on your host – this will come in handy later:

cat /etc/iscsi/initiatorname.iscsi

The next few steps are for ONTAP users, if you are using other storage please refer to their reference manuals:

Login to your ONTAP system, create a volume and LUN:

ssh admin@
volume create -volume <myvolname> -aggregate aggr1 -size 1024GB -space-guarantee none
lun create -volume <myvolname> -lun <lunname> -ostype linux -space-reserve disabled -size 1024GB

iSCSI uses the concept of igroups in order to securely and logically share resources. We want our host iSCSI initiator to be part of a new igroup (for example I could create an igroup called mysql for all of my database instances). In this example my host initiator was

igroup create -igroup mygroup -ostype linux
igroup add -igroup <myigroup> -initiator
lun map -volume <myvolname> -lun <lunname> -igroup <myigroup>

That’s it – your Linux instance should now see all LUNS that are available to the igroup for which it is a member. You can view these LUNs with:

fdisk -l

Now you have  a LUN you’ll need to format it, partition it and mount it:

fdisk /dev/sdb
mkfs.ext3 /dev/sdb1
mount /dev/sdb1 /mnt

You’ve ready to go!


ONTAP goodness:

If you happen to be running ONTAP, you can try instant cloning of your LUNs / databases /etc. Unlike AWS Snapshot clones, these are instant and have the full performance the minute they are created! Simply SSH into ONTAP and run:

volume clone create -parent-volume myvolume -flexclone myclonename

And just like that, you have created a clone of your data! Clones take up no additional space, are instantly available and can be mounted to any other (or the source) instance, great for speeding up development and saving money at the same time!