Skip to main content

Understanding Disk Reservation by Rok

Bogdan Kowalczyk
  • Updated

This document explains how RDM (Rok Disk Manager) manages and reserved the storage on the nodes, and how Rok uses the underlying disks for its internal processing.

 

Disk space reservation by Rok Disk Manager (RDM)

  1. RDM is deployed with a pre-configured script which describes how it uses the underlying disks.
  2. Following this script, it gathers all available disks (both persistent but slow disks, e.g., EBS, and ephemeral but fast disks, e.g., local NVMe) mounted on /dev/sd[f-p] excluding the root disk used by the instance.
  3. RDM then assembles a RAID0 device from all the discovered disks in previous step.
  4. RDM creates a VG (named rokvg) on top of this RAID0 device
  5. RDM creates an LV that Rok uses as ephemeral (scratch) space when taking snapshots for this node. Rok uses this LV to temporarily copy the changed blocks while hashing and de-duplicates them before uploading them to S3. The ephemeral space that Rok reserves for snapshots is by default min(200 * GiB, 0.3 * rokvg.size) per node.
  6. Rok uses the rest of the free space to create the volumes for the users.

Disk space reservation by Rok

  1. Every time rok takes a snapshot, it take a little bit of time to process and upload the new changed data to S3.
  2. In order to avoid avoid freezing the volume and not accept any new writes during the snapshot process, rok creates and uses an ephemeral LV of size 10GiB (for each volume being snapshotted). This volume is used to write live data that the user continue to write on their volume, while rok is processing the snapshots of the volume. 
  3. NOTE: Currently if a volume takes up all of the available space, snapshot process won't be able to reserve space and will fail. We have an issue to fix this and reserve extra space for at least 1 snapshot, so that it is guaranteed that snapshot always succeed

Summary:

There are overall two reservation in available disk space.
  1. min(200 * GiB, 0.3 * rokvg.size) per node for temporarily copying changed block data from previous snapshot while its being hashed, de-duped and uploaded to S3.
  2. 10G per volume snapshot to hold new writes while snapshot is happening.  
 

Example

When using m5d.4xlarge instances which have 2 x 300GB  local NVMe disks:
  • We have total of 600GB available 
  • RDM reserve ~180 GB for Rok ephemeral storage
  • The rest ~420 GB are available to the user volumes
  • Out of that maximum of ~410 GB of volume can be created while keeping ~10 GB reserved for rok snapshot process.
  • If creating more smaller volumes it's important to remember rok will need ~10 GB of reserved space for each volume being snapshot
 
 
When using m5d.8xlarge instances which have 2 x 600GB  local NVMe disks:
  • We have total of 1200GB available 
  • RDM reserve ~200 GB for Rok ephemeral storage
  • The rest ~1000 GB are available to the user volumes
  • Out of that maximum of ~990 GB of volume can be created while keeping ~10 GB reserved for rok snapshot process.
  • If creating more smaller volumes it's important to remember rok will need ~10 GB of reserved space for each volume being snapshot
 

Limitations:

  •  Currently we reserve min(200 * GiB, 0.3 * rokvg.size) per node for holding the change block data while its being hashed, de-duped and uploaded to S3. This means only 200G of max changed data can be snapshotted and trying to snapshot more that 200G of changed data will result in failure.  This means if you trying to take the snapshot of a new large volume which is already more than 200G full will fail. 
  • This limitation can be mitigated by configuring the RDM to reserve more than 200G of reserved spaces based on the requirements.
  • 200G is enough for most of the scenarios, but if there is use case where all of the data needs to be written to the volume before taking first snapshot or it is expected that more than 200G of data can be written between the snapshots, its recommended to increase the reservation amount to desired size. In worst case scenario it could be same a max volume size, which means only 1/2 of storage space can be used to create the volume and other half to hold the temp data.
 
 
 
 
 
 

 

Related articles

Comments

0 comments

Please sign in to leave a comment.