Using Rex-Ray with Docker Swarm on Azure

I try to have a working storage tool for my Docker Swarm cluster on Azure that is performant and use the storage service of my cloud provider. I didn't want to manage a custom cluster with VMs and disk for handling persistence across my cluster.

What is Rex-Ray?

From Rex-Ray website:

REX-Ray is the leading storage orchestration engine providing interoperability with cloud native orchestrators and runtimes to enable persistent applications in containers.

Installing Rex-Ray for Azure

Requirements

• An Azure account
• An Azure subscription
• An Azure storage account
• An Azure resource group
• Any virtual machine with unmanaged OS disk, Azure can't mix managed and unmanaged disk on the same VM.
• Any virtual machine where disks are going to be attached must have the lsscsi utility installed. You can install this with yum install lsscsi on Red Hat based distributions, or with apt-get install lsscsi on Debian based distributions.

Creating a Service Principal

A Service Principal is an application within Azure Active Directory whose authentication tokens can be used as the client_id, client_secret, and tenant_id fields needed by RexRay (subscription_id can be independently recovered from your Azure account details).

If you don't know the subscription id, you can recover it with:

$ az account list
[
  {
    "cloudName": "AzureCloud",
    "id": "sub1-xxx-xxx-xxx-xxx",
    "isDefault": true,
    "name": "Visual Studio Enterprise",
    "state": "Enabled",
    "tenantId": "tenant1-tttt-tttt-tttt-ttttt",
    "user": {
      "name": "user1-uuuu-uuuu-uuuu-uuuuu",
      "type": "servicePrincipal"
    }
  },
  {
    "cloudName": "AzureCloud",
    "id": "sub2-yyy-yyy-yyy-yyy",
    "isDefault": false,
    "name": "Free Trial",
    "state": "Enabled",
    "tenantId": "tenant2-tttt-tttt-tttt-tttt",
    "user": {
      "name": "contact@jmaitrehenry.ca",
      "type": "user"
    }
  }
]

Now we can create the Service Principal application:

$ az account set --subscription="sub1-xxx-xxx-xxx-xxx"
$ az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/sub1-xxx-xxx-xxx-xxx"
{
  "appId": "app1-aaa-aaa-aaa-aaa",
  "displayName": "azure-cli-2017-12-01-20-26-37",
  "name": "http://azure-cli-2017-12-01-20-26-37",
  "password": "secret-sss-sss-sss-ss",
  "tenant": "tenant1-ttt-ttt-ttt-ttt"
}

appId is the client_id defined above.
password is the client_secret defined above.
tenant is the same for the Service Principal and the subscription.

Add require access to our new application

Once the Application exists in Azure Active Directory, we can grant it permissions to modify resources in the Subscription.

Find your newly created app in the Azure Active Directory blade.

Choose Require permissions > Add.

Choose "Windows Azure Service Management API"

Select "Access Azure Service Management as organization users" and save.

Get information from the account storage

To configure the plugin, you need a Storage Account name and access key. You can find them on your Storage Account > Access Keys page:

Installing the docker plugin

On each docker node you need to install the plugin:

$ docker plugin install \
  --alias rexray/azureud \
  rexray/azureud:0.11.1 \
    AZUREUD_CLIENTID=app1-aaa-aaa-aaa-aaa \
    AZUREUD_CLIENTSECRET==secret-sss-sss-sss-ss \
    AZUREUD_SUBSCRIPTIONID=sub1-xxx-xxx-xxx-xxx \
    AZUREUD_TENANTID=tenant1-ttt-ttt-ttt-ttt \
    AZUREUD_RESOURCEGROUP=docker-storage \
    AZUREUD_STORAGEACCESSKEY=mmpwuGgnSKHodND.... \
    AZUREUD_STORAGEACCOUNT=myswarmstorage
    
Plugin "rexray/azureud:0.11.1" is requesting the following privileges:
 - network: [host]
 - mount: [/dev]
 - allow-all-devices: [true]
 - capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
0.11.1: Pulling from rexray/azureud
e433ebe0ac07: Download complete
Digest: sha256:072cd65d4da0cef4fdcd09a7e9df8fecff1eeb3946c50f13c1c4754f391248fd
Status: Downloaded newer image for rexray/azureud:0.11.1
Installed plugin rexray/azureud:0.11.1

Limitations

You can't use a Virtual Machine Scale Set (VMSS) because you can only use Managed Disk with it and Rex-Ray doesn't support it yet.

For more informations:

• Rex-Ray issue for Managed Disk
• Azure virtual machine scale sets and attached data disks

You have a limitation on how many disk you can attach to a single VM.
For more details about virtual machines capacities, read Sizes for Linux virtual machines.

You have global limitation on your storage account and per page blob, see Azure Storage Scalability and Performance Targets for more information.

Using Rex-Ray with your containers

Using Rex-Ray is as simple as using a normal volume. You just need to specify the driver rexray/azureud and a size (in GB).

version: "3"
services:
  mysql:
    image: mysql:latest
    volumes:
      - mysql:/var/lib/mysql

volumes:
  mysql:
    driver: rexray/azureud
    driver_opts:
      size: 20

$ docker volume create -d rexray/azureud myvolume --opt=size=2
$ docker run -ti -v myvolume:/myvolume alpine sh
$ ls /myvolume
$ touch /myvolume/test
$ echo 'test' >/myvolume/test
$ cat /myvolume/test
test

Have fun!

If you find a typo, have a problem when trying what you find on this article, please contact me!

Tags

• Azure
• Swarm
• Docker
• Storage
• Cloud
• open source