Amazon Linux: Migrate MySQL from EBS to iSCSI

This week, I was working with a 100% cloud business, who have been using  AWS from their incorporation over 5 years ago.

The business wanted to take advantage of the instant cloning capabilities offered by running ONTAP in AWS and migrate the existing MySQL databases to the new cloud storage system.

These steps will get you up and running quickly, including some handy gotchas that you may run into.

First, I assume that you already have your Amazon Linux (or other instance) up and running and that you have your iSCSI devices mapped and mounted in your instance. Don’t worry if you haven’t – I created a handy post here that covers it.

Let’s check the current location of your MySQL datadir, there are two methods for checking this:

Method #1: Check within MySQL

mysql -u root -p

select @@datadir;

Method #2: Check from the console and look at the current value of datadir (for example datadir=/var/lib/mysql)

sudo cat /etc/my.cnf

Now that we know the current data directory we can move the database to the new location with these steps. In this example, my iSCSI device is mounted to /mnt/ontap

Shutdown the MySQL service (this ensures data and application consistency):

sudo service mysqld stop

Next, copy the database to the new location:

sudo rsync -av /var/lib/mysql /mnt/ontap

Once complete it’s best to rename the old directory to .bak to prevent confusion:

sudo mv /var/lib/mysql /var/lib/mysql.bak

Now it’s time to change the working directory path to the new one:

sudo nano /etc/my.cnf

datadir=/mnt/ontap

It would seem natural at this point to start the service, however, there is at least one more step:

(Optional: If you are using AppArmour):

sudo nano /etc/apparmor.d/tunables/alias

At the bottom of the file, add the following alias rule:

. . .
alias /var/lib/mysql/ -> /mnt/netapp1/mysql/,
. . .

Restart the AppArmour servicesudo

sudo service restart apparmour

Now for the final step, MySQL runs an environment check script upon service start. The script simply checks for the existence of /var/lib/mysql and /var/lib/mysql/mysql . We need to create a minimal directory structure to pass the environment check:

sudo mkdir /var/lib/mysql/mysql -p

That’s it! Now we can start the MySQL service:

sudo service mysqld start

Finally, let’s check that the new data directory is indeed in use:

mysql -u root -p

select @@datadir

Thanks for reading, I hope you found this useful!

Amazon Linux: iSCSI – Install and connect to your storage

iSCSI is a great way to attach enterprise class storage to your EC2 instances and it’s very easy to setup, even if you have zero storage management experience. The best part is that all of the steps are completed directly within your Linux instance itself.

Prerequisites:

  1. You’ll need some iSCSI storage within your VPC. This is really easy to deploy with ONTAP Cloud – grab yourself a free 30-day trial.
  2. An instance running in your VPC that you can SSH to.

Simply follow the steps below:

First, log into your instance via SSH/terminal (most use PuTTy for this).

Update your packages (Optional – but best practice):

sudo yum update -y

Next, we install iSCSI into our host:

sudo yum install iscsi-initiator-utils

We will now discover the iSCSi system (I’m using ONTAP in AWS, but you can use it in Azure, whitebox or on-prem). Simply replace 0.0.0.0 with your own IP address.

iscsiadm -m discovery -t st -p 0.0.0.0

Optional: Heres how to find your iSCSI IP address for ONTAP.  SSH to your ONTAP management IP (1.2.3.4 in this example):

ssh admin@1.2.3.4
network interface show
exit

Next, confirm that your host is seeing the iSCSI portal(s) correctly. they will be listed with the following command:

iscsiadm -m node

iSCSI requires your host to log in to the discovered portal, simply run the following command:

iscsiadm -m node --targetname "iqn.xxx.netapp:xxxxx" --portal "<ip-address:port>" --login

Optional but recommended: Restart the iSCSI service:

/etc/init.d/iscsi restart

Now, make a note of the initiator name on your host – this will come in handy later:

cat /etc/iscsi/initiatorname.iscsi

The next few steps are for ONTAP users, if you are using other storage please refer to their reference manuals:

Login to your ONTAP system, create a volume and LUN:

ssh admin@1.2.3.4
volume create -volume <myvolname> -aggregate aggr1 -size 1024GB -space-guarantee none
lun create -volume <myvolname> -lun <lunname> -ostype linux -space-reserve disabled -size 1024GB

iSCSI uses the concept of igroups in order to securely and logically share resources. We want our host iSCSI initiator to be part of a new igroup (for example I could create an igroup called mysql for all of my database instances). In this example my host initiator was iqn.1994-05.com.redhat:265535cef94:

igroup create -igroup mygroup -ostype linux
igroup add -igroup <myigroup> -initiator iqn.1994-05.com.redhat:265535cef94
lun map -volume <myvolname> -lun <lunname> -igroup <myigroup>

That’s it – your Linux instance should now see all LUNS that are available to the igroup for which it is a member. You can view these LUNs with:

fdisk -l

Now you have  a LUN you’ll need to format it, partition it and mount it:

fdisk /dev/sdb
mkfs.ext3 /dev/sdb1
mount /dev/sdb1 /mnt

You’ve ready to go!

 

ONTAP goodness:

If you happen to be running ONTAP, you can try instant cloning of your LUNs / databases /etc. Unlike AWS Snapshot clones, these are instant and have the full performance the minute they are created! Simply SSH into ONTAP and run:

volume clone create -parent-volume myvolume -flexclone myclonename

And just like that, you have created a clone of your data! Clones take up no additional space, are instantly available and can be mounted to any other (or the source) instance, great for speeding up development and saving money at the same time!