Docker on Elastic Beanstalk Tips

AWS Elastic Beanstalk is one of the most used PaaS today, it allows you to deploy your application without provisioning the underlying infrastructure while maintaining the high availability of your application. However, it’s painful to use due to the lack of documentation and real-world scenarios. In this post, I will walk you through how to use Elastic Beanstalk to deploy Docker containers from scratch. Followed by how to automate your deployment process with a Continuous Integration pipeline. At the end of this post, you should be familiar with advanced topics like debugging and monitoring of your applications in EB.

1 – Environment Setup

To get started, create a new Application using the following AWS CLI command:

Create a new environment. Let’s call it “staging” :

Head back to AWS Elastic Beanstalk Console, your new environment should be created:

Point your browser to the environment URL, a sample Docker application should be displayed:

Let’s deploy our application. I wrote a small web application in Go to return a list of Marvel Avengers (I see you Thanos 😉 )

Next, we will create a Dockerfile to build the Docker image. Go is a compiled language, therefore we can use the Docker multi-stage feature to build a lightweight Docker image:

Next, we create a Dockerrun.aws.json that describes how the container will be deployed in Elastic Beanstalk:

Now the application is defined, create an application bundle by creating a ZIP package:

Then, create a S3 bucket to store the different versions of your application bundles:

Issue the following command in order to copy the application into the bucket:

And create a new application version from the application bundle:

Finally, deploy the version to the staging environment:

Give it a few seconds while it’s deploying the new version:

Then, repoint your browser to the environment URL, a list of Avengers will be returned in a JSON format as follows:

Now that our Docker application is deployed, let’s automate this process by setting up a CI/CD pipeline.

2 – CI/CD Pipeline

I opt for CircleCI, but you’re free to use whatever CI server you’re familiar with. The same steps can be applied.

Create a circle.yml file with the following content:

The pipeline will firstly prepare the environment, installing the AWS CLI. Then run unit tests. Next, a Docker image will be built, then pushed to DockerHub. Last step is creating a new application bundle and deploying the bundle to Elastic Beanstalk.

In order to grant Circle CI permissions to call AWS operations, we need to create a new IAM user with following IAM policy:

Generate AWS access & secret keys. Then, head back to Circle CI and click on the project settings and paste the credentials :

Now, everytime you push a change to your code repository, a build will be triggered:

And a new version will be deployed automatically to Elastic Beanstalk:

3 – Monitoring

Monitoring your applications is mandatory. Unfortunately, CloudWatch doesn’t expose useful metrics like Memory usage of your applications in Elastic Beanstalk. Hence, in this part, we will solve this issue by creating our custom metrics.

I will install a data collector agent on the instance. The agent will collect metrics and push them to a time-series database.

To install the agent, we will use .ebextensions folder, on which we will create 3 configuration files:

  • 01-install-telegraf.config: install Telegraf on the instance

  • 02-config-file.config: create a Telegraf configuration file to collect system usage & docker containers metrics.

  • 03-start-telegraf.config: start Telegraf agent.

Once the application version is deployed to Elastic Beanstalk, metrics will be pushed to your timeseries database. In this example, I used InfluxDB as data storage and I created some dynamic Dashboards in Grafana to visualize metrics in real-time:

Containers:

Hosts:

Note: for in-depth explaination on how to configure Telegraf, InfluxDB & Grafana read my previous article.

Full code can be found on my GitHub. Make sure to drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy

AWS CloudWatch Monitoring with Grafana

Hybrid cloud is the new reality. Therefore, you will need a single tool, general purpose dashboard and graph composer for your global infrastructure. That’s where Grafana comes into play. Due to it’s pluggable architecture, you have access to many widgets and plugins to create interactive & user-friendly dashboards. In this post, I will walk you through on how to create dashboards in Grafana to monitor in real-time your EC2 instances based on metrics collected in AWS CloudWatch.

To get started, create an IAM role with the following IAM policy:

Launch an EC2 instance with the user-data script below. Make sure to associate to the instance the role we created earlier:

On the security group section, allow inbound traffic on port 3000 (Grafana Dashboard).

Once created, point your browser to the http://instance_dns_name:3000, you should see Grafana Login page (default credentials: admin/admin) :

Grafana ships with built in support for CloudWatch, so add a new data source:

Note: In case you are using an IAM Role (recommended), keep the other fields empty as above, otherwise, create a new file at ~/.aws/credentials with your own AWS Access Key & Secret key.

Create a new dashboard, and add new graph to the panel, select AWS/EC2 as namespace, CPUUtilization as metric, and the instance id of the instance you want to monitor in the dimension field:

That’s great !

Well, instead of hard-coding the InstanceId in the query, we can use a feature in Grafana called “Query Variables“. Create a new variable to hold list of AWS supported regions :

And, create a second variable to store list of instances ids per selected AWS region:

Now, go back to your graph and update the query as below:

That’s it, go ahead and create other widgets:

Note: You can download the dashboard from GitHub.

Now you’re ready to build interactive & dynamic dashboards for your CloudWatch metrics.

Publish Custom Metrics to AWS CloudWatch

AWS Autoscaling Groups can only scale in response to metrics in CloudWatch and most of the default metrics are not sufficient for predictive scaling. That’s why you need to publish your custom metrics to CloudWatch.

I was surfing the internet as usual, and I couldn’t find any post talking about how to publish custom metrics to AWS CloudWatch, and because I’m a Gopher, I got my hand dirty and I wrote my own script in Go.

You can publish your own metrics to CloudWatch using the AWS Go SDK:

To collect metrics about memory for example,  you can either parse output of command ‘free -m’ or use a third-party library like gopsutil:

The memoryMetrics object expose multiple metrics:

  • Memory used
  • Memory available
  • Buffers
  • Swap cached
  • Page Tables
  • etc

Each metric will be published with an InstanceID dimension. To get the instance id, you can query the meta-data:

Résultat de recherche d'images pour "simple right meme"

What if I’m not a Gopher ? well, don’t freak out, I built a simple CLI which doesn’t require any Go knowledge or dependencies to be installed (AWS CloudWatch Monitoring Scripts requires Perl dependencies) and moreover it’s cross-platform.

The CLI collects the following metrics:

  • Memory: utilization, used, available.
  • Swap: utilization, used, free.
  • Disk: utilization, used, available.
  • Network: packets in/out, bytes in/out, errors in/out.
  • Docker: memory & cpu per container.

The CLI have been tested on instances using the following AMIs (64-bit versions):

  • Amazon Linux
  • Amazon Linux 2
  • Ubuntu 16.04
  • Microsoft Windows Server

To get started, find the appropriate package for your instance and download it. For linux:

After you install the CLI, you may need to add the path to the executable file to your PATH variable. Then, issue the following command:

The command above will collect memory, swap, network & docker containers resource utilization on the current system.

Note: ensure an IAM role is associated with your instance, verify that it grants permission to perform cloudwatch:PutMetricData.

Now that we’ve written custom metrics to CloudWatch. You can view statistical graphs of your published metrics with the AWS Management Console:

You can create your own interactive and dynamic Dashboard based on these metrics:

Hope it helps ! The CLI is still in its early stages, so you are welcome to contribute to the project on GitHub.

Network Infrastructure Weathermap

The main goal of collecting metrics is to store them for long term usage and to create graphs to debug problems or identify trends. However, storing metrics about your system isn’t enough to identity the problem’s & anomalies root cause. It’s necessary to have a high-level overview of your network backbone. Weathermap is perfect for a Network Operations Center (NOC). In this post, I will show you how to build one using Open Source tools only.

Icinga 2 will collect metrics about your backbone, write checks results metrics and performance data to InfluxDB (supported since Icinga 2.5). Visualize these metrics in Grafana in map form.

To get started, add your desired host configuration inside the hosts.conf file:

Note: the city & country attributes will be used to create the weathermap.

To enable the InfluxDBWriter on your Icinga 2 installation, type the following command:

Configure your InfluxDB host and database in /etc/icinga2/features-enabled/influxdb.conf (Learn more about the InfluxDB configuration)

Icinga 2 will forward all your metrics to a icinga2_metrics database. The included host and service templates define a storage, the measurement represents a table by which metrics are grouped with tags certain measurements of certain hosts or services are defined (notice the city & country tags usage).

Don’t forget to restart Icinga 2 after saving your changes:

Once Icinga 2 is up and running it’ll start collecting data and writing them to InfluxDB:

Once our data arrived, it’s time for visualization. Grafana is widely used to generate graphs and dashboards. To create a Weathermap we can use a Grafana plugin called Worldmap Panel. Make sure to install it using grafana-cli tool:

The plugin will be installed into your grafana plugins directory (/var/lib/grafana/plugins):

Restart Grafana, navigate to Grafana web interface and create a new datasource:

Create a new Dashboard:

The Group By clause should be the country code and an alias is needed too. The alias should be in the form $tag_field_name. See the image below for an example of a query:

Under the Worldmap tab, choose the countries option:

Finally, you should see a tile map of the world with circles representing the state of each host.

The field state possible values (0 – OK, 1 – Warning, 2 – Critical, 3 – Unknown/Unreachable)

Note: For lazy people I created a ready to use Dashboard you can import from GitHub.

MySQL Monitoring with Telegraf, InfluxDB & Grafana

This post will walk you through each step of creating interactive, real-time & dynamic dashboard to monitor your MySQL instances using Telegraf, InfluxDB & Grafana.

Start by enabling the MySQL input plugin in /etc/telegraf/telegraf.conf :

Once Telegraf is up and running it’ll start collecting data and writing them to the InfluxDB database:

Finally, point your browser to your Grafana URL, then login as the admin user. Choose ‘Data Sources‘ from the menu. Then, click ‘Add new‘ in the top bar.

Fill in the configuration details for the InfluxDB data source:

You can now import the dashboard.json file by opening the dashboard dropdown menu and click ‘Import‘ :

Note: Check my Github for more interactive & beautiful Grafana dashboards.