Elastic Storage in AWS Cloud

What is Elastic EBS
Elastic EBS Use Cases
Elastic EBS Testing
Elastic EBS Limitations

What is Elastic EBS

Public clouds are known for elasticity of compute resources. For example, AWS AutoScaling Group (ASG) can dynamically scale up a compute farm (VM or cloud instances) to service higher load or scale down to safe cost during off hours. What missing was a dynamic scaling of block storage such as: online volume expansion, storage tiering or temporary boosting volume's iops and throughput. Recent announcement of Elastic EBS feature addressed some of these shortcoming.  SAN features like online modification of volume attributes (size, iops and type) without detaching or attaching volume or restarting an instance are now possible in public cloud. No down time is required!

EBS (Elastic Block Storage) is AWS network storage offering that allows attaching block storage to a running cloud instance in a given Availability Zone (AZ).  EBS volumes support both: solid state disks (SSD), called io1, gp2, for improved random iops or lower latencies, and magnetic disks (HD), called st1 and sc1, for higher sequential throughput.  Elastic EBS feature makes it possible to mix and match EBS volume capabilities to achieve best storage performance for a given workload.

Elastic EBS Use Cases

Elastic EBS fits well for various use cases listed below:

Performance Boost: 

• To improve IOPS performance temporarily during nightly batch or quarterly end processing
• To improve sequential IO throughput for Analytics and DSS queries, io1 or gp2 volumes can be modified to st1 volumes that offer higher throughput than io1 and gp2 volumes.

Storage Tiers to Save Cost:

• Modify volume types to save cost. One can build online storage tier without impacting performance using various EBS types available: io1, gp2, st1 and sc1

On Demand Storage Capacity

• Instead of over-provisioning storage, start with a fixed size EBS volumes for all instances in your cluster. Monitor EBS volume capacity and raise it when reaches 80-90%. This may save cost as the capacity increase is performed on as needed basis.

Reduced Administration Overhead

• EBS storage saves great deal of administration work as compared to ephemeral storage (direct attach storage). EBS is persistent storage and thus does not require additional steps in protecting data such as copying or refreshing data, as in the case of ephemeral storage, on instance launch. 
• Elastic EBS feature makes EBS more attractive as it takes away the size consideration out of capacity planning decision. Both volume and file system can be expanded online without requiring instance reboot.

Elastic EBS Testing

Two new EBS API calls: ModifyVolume and DescribeVolumesModifications were introduced to support Elastic Volume feature.  awsconsole, awscli or API calls can be used to modify volume. Completion status can then be polled via Cloudwatch metrics or via DescribeVolumesModifications API to trigger some action.

 
" For example: File system size can be expanded online once volume change completion
      notification is received"

Modify Volume -- iops Test

Provision iops on io1 volume can be modified (increase or decrease) dynamically. Modification request takes few minutes to complete. Volume continue to perform at original iops level until volume modification is completed

To test the feature, I created and attached 500 GB ebs io1 volume type. DescribeVolumesModification API is used to poll volume changes. Iops are increased in every iteration by 2000. fio benchmark tool was running in the background to perform read via direct IO path. Measured storage iops, tput and latency. See bash script used for testing below

#!/bin/sh while true do #increase iops iops=0 while [ $iops -lt 21000 ] do  iops=$[$iops+2000]  echo $iops >> iops-test.out 2>&1  date >> iops-test.out 2>&1  error=`aws ec2 modify-volume --volume-id vol-0db5e81a6fe4e5b4b --iops=$iops --region us-east-1 2>&1 >/dev/null |awk '{print $2}'`  echo $error >> iops-test.out 2>&1  if [[ $error =~ "error" ]]  then   sleep 21700  else   notfound=true   while ($notfound)   do    modifyState=`aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-0db5e81a6fe4e5b4b | jq .VolumesModifications | jq ".[] | .ModificationState"`    if [[ $modifyState =~ "completed" ]]    then     aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-0db5e81a6fe4e5b4b >> iops-completion.out 2>&1     notfound=false   else    echo "Waiting for ModificationState to reach completion" >> iops-test.out 2>&1    sleep 5   fi done fi done # Reduce iops iops=20000 while [ $iops -gt 1900 ] do iops=$[$iops-2000] echo $iops >> iops-test.out 2>&1 date >> iops-test.out 2>&1 error=`aws ec2 modify-volume --volume-id vol-0db5e81a6fe4e5b4b --iops=$iops --region us-east-1 2>&1 >/dev/null |awk '{print $2}'` echo $error >> iops-test.out 2>&1 if [[ $error =~ "error" ]] then   sleep 21700 else notfound=true while ($notfound) do   modifyState=`aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-0db5e81a6fe4e5b4b | jq .VolumesModifications | jq ".[] | .ModificationState"`   if [[ $modifyState =~ "completed" ]]   then    aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-0db5e81a6fe4e5b4b >> iops-completion.out 2>&1    notfound=false   else    echo "Waiting for ModificationState to reach completion" >> iops-test.out 2>&1    sleep 5   fi done fi done done

VOLUME --IOPS MODIFICATION COMPLETION STATUS json returned by DescribeVolumesModifications notifying volume ModificationState and EndTime {    "VolumesModifications": [        {            "TargetSize": 500,            "TargetVolumeType": "io1",            "ModificationState": "completed",            "VolumeId": "vol-0db5e81a6fe4e5b4b",            "TargetIops": 6000,            "StartTime": "2017-03-06T11:11:06.375Z",            "Progress": 100,            "OriginalVolumeType": "io1",            "OriginalIops": 2000,            "EndTime": "2017-03-06T11:16:28.742Z",            "OriginalSize": 500        }    ] } {    "VolumesModifications": [        {            "TargetSize": 500,            "TargetVolumeType": "io1",            "ModificationState": "completed",            "VolumeId": "vol-0db5e81a6fe4e5b4b",            "TargetIops": 10000,            "StartTime": "2017-03-06T17:18:16.190Z",            "Progress": 100,            "OriginalVolumeType": "io1",            "OriginalIops": 6000,            "EndTime": "2017-03-06T17:19:39.691Z",            "OriginalSize": 500        }    ] }

Modify Volume --size Test
Created and attached 500 GB ebs gp2 volume. DescribeVolumesModification API is used to poll volume modification status. Size of gp2 volume is modified (expanded) to an additional 200GB in every test iteration. xfs_grow is invoked to grow xfs file system to utilize additional volume space. fio read IO load on a file system was running in endless loop. Measured storage iops, tput and latency. See batch script used for testing below:

#!/bin/sh size=1200 while [ $size -lt 100001 ]   do size=$[$size+100] echo $size >> size-test.out 2>&1 date >> size-test.out 2>&1 error=`aws ec2 modify-volume --volume-id vol-0add7a47226f23fa6 --size=$size --region us-east-1 2>&1 >/dev/null |awk '{print $2}'` echo $error >> size-test.out 2>&1 if [[ $error =~ "error" ]] then  sleep 21700 else  notfound=true  while ($notfound)  do   modifyState=`aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-0add7a47226f23fa6 | jq .VolumesModifications | jq ".[] | .ModificationState"`   if [[ $modifyState =~ "completed" ]]    then     aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-0add7a47226f23fa6 >> size-completion.out 2>&1     sudo fdisk -l /dev/sdg >> size-test.out 2>&1     sudo xfs_growfs /dev/sdg>> size-test.out 2>&1     notfound=false   else    echo "Waiting for ModificationState to reach completion" >> size-test.out 2>&1    sleep 5  fi done fi done

VOLUME --SIZE MODIFICATION COMPLETION json returned by DescribeVolumesModifications notifying volume ModificationState and EndTime {    "VolumesModifications": [        {            "TargetSize": 1400,            "TargetVolumeType": "gp2",            "ModificationState": "completed",            "VolumeId": "vol-0add7a47226f23fa6",            "TargetIops": 4200,            "StartTime": "2017-03-06T05:26:50.957Z",            "Progress": 100,            "OriginalVolumeType": "gp2",            "OriginalIops": 3900,            "EndTime": "2017-03-06T05:58:48.002Z",            "OriginalSize": 1300        }    ] } {    "VolumesModifications": [        {            "TargetSize": 1600,            "TargetVolumeType": "gp2",            "ModificationState": "completed",            "VolumeId": "vol-0add7a47226f23fa6",            "TargetIops": 4800,            "StartTime": "2017-03-06T12:00:34.867Z",            "Progress": 100,            "OriginalVolumeType": "gp2",            "OriginalIops": 4200,            "EndTime": "2017-03-06T12:26:24.884Z",            "OriginalSize": 1400        }    ] } {    "VolumesModifications": [        {            "TargetSize": 1800,            "TargetVolumeType": "gp2",            "ModificationState": "completed",            "VolumeId": "vol-0add7a47226f23fa6",            "TargetIops": 5400,            "StartTime": "2017-03-06T18:28:08.889Z",            "Progress": 100,            "OriginalVolumeType": "gp2",            "OriginalIops": 4800,            "EndTime": "2017-03-06T18:49:31.960Z",            "OriginalSize": 1600        }    ] }

Modify Volume --type Test
Created and attached 3TB sc1 ebs type volume. DescribeVolumesModification API is used to poll modification status. Dynamically change 3TB volume from one EBS volume type to other in a loop:
sc1->st1 | st1->gp2 | gp2->io1 | io1-> gp2 | gp2->st1 | st1->sc1
fio read IO load via direct IO was running in endless loop. Measured storage iops, tput and latency. See the batch script used for testing below:

#!/bin/bash while true do date >> type-test.out 2>&1 error=`aws ec2 modify-volume --volume-id vol-08ac011213c5b6363 --volume-type=st1 --region us-east-1 2>&1 >/dev/null |awk '{print $2}'` echo $error >> type-test.out 2>&1 if [[ $error =~ "error" ]] then  sleep 21700 else  notfound=true  while ($notfound)  do   modifyState=`aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-08ac011213c5b6363 | jq .VolumesModifications | jq ".[] | .ModificationState"`   if [[ $modifyState =~ "completed" ]]    then     aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-08ac011213c5b6363 >> st1-completion.out 2>&1     notfound=false   else    echo "Waiting for ModificationState to reach completion" >> type-test.out 2>&1    sleep 5  fi done fi date >> type-test.out 2>&1 error=`aws ec2 modify-volume --volume-id vol-08ac011213c5b6363 --volume-type=gp2 --region us-east-1 2>&1 >/dev/null |awk '{print $2}'` echo $error >> type-test.out 2>&1 if [[ $error =~ "error" ]] then  sleep 21700 else  notfound=true  while ($notfound)  do   modifyState=`aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-08ac011213c5b6363 | jq .VolumesModifications | jq ".[] | .ModificationState"`   if [[ $modifyState =~ "completed" ]]    then     aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-08ac011213c5b6363 >> gp2-completion.out 2>&1     notfound=false   else    echo "Waiting for ModificationState to reach completion" >> type-test.out 2>&1    sleep 5  fi done fi date >> type-test.out 2>&1 error=`aws ec2 modify-volume --volume-id vol-08ac011213c5b6363 --volume-type=io1 --region us-east-1 2>&1 >/dev/null |awk '{print $2}'` echo $error >> type-test.out 2>&1 if [[ $error =~ "error" ]] then  sleep 21700 else  notfound=true  while ($notfound)  do   modifyState=`aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-08ac011213c5b6363 | jq .VolumesModifications | jq ".[] | .ModificationState"`   if [[ $modifyState =~ "completed" ]]    then     aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-08ac011213c5b6363 >> io1-completion.out 2>&1     notfound=false   else    echo "Waiting for ModificationState to reach completion" >> type-test.out 2>&1    sleep 5  fi done fi date >> type-test.out 2>&1 error=`aws ec2 modify-volume --volume-id vol-08ac011213c5b6363 --volume-type=sc1 --region us-east-1 2>&1 >/dev/null |awk '{print $2}'` echo $error >> type-test.out 2>&1 if [[ $error =~ "error" ]] then  sleep 21700 else  notfound=true  while ($notfound)  do   modifyState=`aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-08ac011213c5b6363 | jq .VolumesModifications | jq ".[] | .ModificationState"`   if [[ $modifyState =~ "completed" ]]    then     aws ec2 describe-volumes-modifications --region us-east-1 --volume-id vol-08ac011213c5b6363 >> sc1-completion.out 2>&1     notfound=false   else    echo "Waiting for ModificationState to reach completion" >> type-test.out 2>&1    sleep 5  fi done fi done

VOLUME --TYPE MODIFICATION COMPLETION {    "VolumesModifications": [        {            "TargetSize": 3072,            "TargetVolumeType": "sc1",            "ModificationState": "completed",            "VolumeId": "vol-08ac011213c5b6363",            "StartTime": "2017-03-06T11:08:53.840Z",            "Progress": 100,            "OriginalVolumeType": "gp2",            "OriginalIops": 9216,            "EndTime": "2017-03-06T11:14:42.648Z",            "OriginalSize": 3072        }    ] } {    "VolumesModifications": [        {            "TargetSize": 3072,            "TargetVolumeType": "gp2",            "ModificationState": "completed",            "VolumeId": "vol-08ac011213c5b6363",            "TargetIops": 9216,            "StartTime": "2017-03-06T04:56:22.591Z",            "Progress": 100,            "OriginalVolumeType": "st1",            "EndTime": "2017-03-06T05:07:11.440Z",            "OriginalSize": 3072        }    ] } {    "VolumesModifications": [        {            "TargetSize": 3072,            "TargetVolumeType": "gp2",            "ModificationState": "completed",            "VolumeId": "vol-08ac011213c5b6363",            "TargetIops": 9216,            "StartTime": "2017-03-06T17:16:25.168Z",            "Progress": 100,            "OriginalVolumeType": "sc1",            "EndTime": "2017-03-06T17:27:27.422Z",            "OriginalSize": 3072        }    ] }

Elastic EBS Limitations

• Elastic EBS feature is supported only on current generation of aws instances
• There is a limit on one modification request per volume every 6 hours.
• Volume modification change request takes from few seconds to tens of minutes to complete.
• Minimum charge is for 6 hours once volume modification is completed.
• Modify volume option "iops" applies to "io1" (provision iops) EBS volume only.
• Modify Volume option "size" can only be used to increase the volume capacity. Shrinking volume is not supported!
• Both ext4 and xfs support growing file system online.
  • File systems cannot be grown online if EBS device is under Linux MD and RAID volume type is RAID-0. Linux MD RAID supports growing for RAID-1 (mirror) and RAID-5(parity) volumes only.
  • File systems can be grown online if EBS device is under LVM (Linux Volume Manager) control
  • File systems can be grown online on drbd device as long as drbd device is backed up by LVM.
• Modify options: "iops" and "type" (except "size") are transparent to instance and thus can be done to all types of configurations (raw, block, MD, LVM, drbd etc..) . 

• AWS IAM InstanceProfile need additional permission to run ModifyVolume and DescribeVolumesModifications API from the instance to modify volume online. 
  • One can use scripts provided in the script section of this document to modify volume's iops, size and type.
  • One can use AWS lambda function provided by AWS to tag all volumes requiring size changes to set "maintenance" tag and then a script that runs on the instance to resize all targeted volumes.

EBS continue to evolve to match SAN like features in public cloud.  We expect more SAN features to be added into the growing feature list in the future such as: Thin provisioning, deduplication, LUN level snapshots or mirroring and multi-attach volumes for clustering application like Oracle RAC or clustered filesystems.

NOTE: EBS does support snapshot feature, but snapshot is saved to S3. When restored, data is copied to EBS volume as the data is accessed (lazy copy). Due to lazy copy from S3 to EBS , IO latencies jump to 100ms from 2-5 ms until all blocks are copied to EBS from S3. LUN level snapshot copies the data directly to destination LUN.