Highly Available Docker Registry on AWS with Nexus

Have you ever wondered how you can build a highly available & resilient Docker Repository to store your Docker Images ?

Résultat de recherche d'images pour "you came to the right place meme"

In this post, we will setup an EC2 instance inside a Security Group and create an A record pointing to the server Elastic IP address as follow:

To provision the infrastructure, we will use Terraform as IaC (Infrastructure as Code) tool. The advantage of using this kind of tools is the ability to spin up a new environment quickly in different AWS region (or different IaaS provider) in case of incident (Disaster recovery).

Start by cloning the following Github repository:

Inside docker-registry folder, update the variables.tfvars with your own AWS credentials (make sure you have the right IAM policies).

I specified a shell script to be used as user_data when launching the instance. It will simply install the latest version of Docker CE and turn the instance to Docker Swarm Mode (to benefit from replication & high availability of Nexus container)

Note: Surely, you can use a Configuration Management Tools like Ansible or Chef to provision the server once created.

Then, issue the following command to create the infrastructure:

Once created, you should see the Elastic IP of your instance:

Connect to your instance via SSH:

Verify that the Docker Engine is running in Swarm Mode:

Check if Nexus service is running:

If you go back to your AWS Management Console. Then, navigate to Route53 Dashboard, you should see a new A record has been created which points to the instance IP address.

Point your favorite browser to the Nexus Dashboard URL (registry.slowcoder.com:8081). Login and create a Docker hosted registry as below:

Edit the /etc/docker/daemon.json file, it should have the following content:

Note: For production it’s highly recommended to secure your registry using a TLS certificate issued by a known CA.

Restart Docker for the changes to take effect:

Login to your registry with Nexus Credentials (admin/admin123):

In order to push a new image to the registry:

Verify that the image has been pushed to the remote repository:

To pull the Docker image:

Note: Sometimes you end up with many unused & dangling images that can quickly take significant amount of disk space:

You can either use the Nexus CLI tool or create a Nexus Task to cleanup old Docker Images:

Populate the form as below:

The task above will run everyday at midnight to purge unused docker images from “mlabouardy” registry.

GitLab Performance Monitoring with Grafana

Since GitLab v8.4 you can monitor your own instance with InfluxDB & Grafana stack by using the GitLab application performance measuring system called “Gitlab Performance Monitoring“.

GitLab writes metrics to InfluxDB via UDP. Therefore, this must be enabled in /etc/influxdb/influxdb.conf:

Restart your InfluxDB instance. Then, create a database to store GitLab metrics:

Next, go to Gitlab Setings Dashboard and enable InfluxDB Metrics as shown below:

Then, you need to restart GitLab:

Now your GitLab instance should send data to InfluxDB:

Finally, Point your browser to your Grafana URL, then login as the admin user. Choose ‘Data Sources‘ from the menu. Then, click ‘Add new‘ in the top bar.

Fill in the configuration details for the InfluxDB data source:

You can now import the dashboard.json file by opening the dashboard dropdown menu and click ‘Import‘ :

Note: Check my Github for more interactive & beautiful Grafana dashboards.

Exploring Swarm & Container Overview Dashboard in Grafana

In my previous post, your learnt how to monitor your Swarm Cluster with TICK Stack. In this part, I will show you how to use the same Stack but instead of using Chronograf as our visualization and exploration tool we will use Grafana.

Connect to your manager node via SSH, and clone the following Github repository:

Use the docker-compose.yml below to setup the monitoring stack:

Then, issue the following command to deploy the stack:

Once deployed, you should see the list of services running on the cluster:

Point your browser to http://IP:3000, you should be able to reach the Grafana Dashboard:

The default username & password are admin. Go ahead and log in.

Go to “Data Sources” and create 2 InfluxDB data sources:

  • Vms: pointing to your Cluster Nodes metrics database.
  • Docker: pointing to your Docker Services metrics database.

Finally, import the dashboard by hitting the “import” button:

From here, you can upload the dashboard.json, then pick the data sources you created earlier:

You will end up with an interactive and dynamic dashboard:

Monitor Swarm cluster with TICK stack & Slack

In this article, I will show you how to setup an Open Source time series platform to monitor your Docker Swarm cluster & send notification on Slack in case of anomaly detection.

Components of our monitoring Stack:

Plugin driven server agent for collecting and reporting metrics.

Scalable time series database for metrics, events and real-time analytics.

Real time visualization tool for building graphs on top of data.

Framework for processing, monitoring, and alerting on time series data.

Real-time team messaging application.

Note: all the code used in this post is available on my Github.

1 – Swarm Setup

If you already have an existing Swarm cluster, you can skip this part, if not use the following script to setup a Swarm with 3 nodes (1 manager & 2 workers):

Issue the following commands:

The output of the above command is as follows:

2 – Stack Setup

Once created, connect to your manager node via SSH, and clone the following repository:

To start all of these containers I’m using docker-compose:

Issue the following command to deploy the stack:

Wait for nodes to pull the images from DockerHub:

Once pulled you should see the services running:

Open your browser on http://IP:8888 (Chronograf Dashboard) and properly configure the data source:

3 – System usage Dashboard

Click on “create dashboard“, and assign a name to the dashboard:

Before adding graphs, we will use a concept called Dashboard Template Variable, to create dynamic & interactive graphs. Instead of hard-coding things like node name and container name in our metric queries we will use variables in their place. So click on “Templates Varibles” in top of the dashboard created earlier. And, create a variable called :host: as follows:

Note: currently, there’s no solution to set hostname for services created with swarm global mode (Github). Thats why we got list of IDs instead of names

You can now use the dropdown at the top of the dashboard to select the different options for the :host: template variable:

Now it’s time to create our first graph, so click on “Add Graph” button.

3.1 – Memory usage per Node

To create a query, you can either use the Query Builder or, if you’re already familar with InfluxQL, you can manually enter the query in the text input:

Our query calculates the average of the field keys freeused, and total in the measurement mem_vm, and it groups them by the time and node name.

You can change the graph type, X, and Y axes format by clicking on “Options” tab:

One visualization on a dashboard isn’t spectacularly interesting, so I added a couple more graphs to show you more possibilities:

3.2 – CPU usage per Node

3.3 – Disk usage per Node

We end up with a beautiful dashboard like this:

Let’s create another dashboard to monitor Docker Containers running on the Cluster.

4 – Swarm Services Dashboard

Create a second dashboard called “Services“, and create a template variable to store list of services running on cluster:

You can filter now metrics by service name:

4.1 – Memory usage per Service

4.2 – CPU usage per Service

4.3 – Network Transmit/Receive

4.4 – IO Read/Write per Service

Result:

Note: you can take this further, and filter metrics by the node on which the service is running on by creating another template variable:

Let’s see what happen if we create another service on Swarm:

If you go back to Chronograf, you should see the service has been added automatically to the container dropdown list:

And that’s it! You now have the foundation for building beautiful data visualizations and dashboards with Chronograf.

Kapacitor is the last piece of the puzzle. We now know how to store, get and read metrics, and now you need to elaborate on them to do something like alerting or proactive monitoring.

So on the “Configuration” tab, click on “Add config“:

Add new Kapacitor instance as below, and enable Slack:

Note: update the Slack channel & Webhook URL in case you didn’t update the kapacitor.conf file in the beginning of this tutorial. You can get a Webhook URL by going to this page:

5 – Alerts definition

5.1 – High Memory Utilization Alert

Navigate to the “Rule Configuration” page by visiting the “Alerting” page and click on the “Build Rule” button in the top right corner:

We will trigger an alert if the memory usage is over 60%:

Next, we select Slack as the event handler and configure the alert message:

Note: there’s no need to include a Slack channel in the Alert Message section if you specified a default channel in the initial Slack configuration.

5.2 – High CPU Utilization Alert

Create a second rule to trigger an alert if CPU usage is over 40%:

Alert endpoint:

Save the rule, and you’re all set !

Now our alert rules are defined, lets test them out by creating some load on our cluster.

6 – Stress Testing

I used stress, a tool for generating workload. It can produce CPU, memory, I/O, and disk stress.

6.1 – Stressing the CPU

Note: depending on the type of your CPU, make sure to replace ‘4‘ accordingly.

After few seconds, you should receive a Slack notification:

Kapacitor trigger an alert and also recovered them (Status OK) if the alert is resolved.

6.2 – Stressing the Memory

It will stress memory using three processes, with each about 256 Mb (override with the option –vm-bytes).

Let it run for a couple seconds :

That’s it! You’ve successfully setup a highly scalable, distributed monitoring platform for your Swarm cluster with only Open Source projects.

Résultat de recherche d'images pour "wow amazing meme"

 

Continuous Deployment with AWS CodeDeploy & Github

This post will walk you through how to AutoDeploy your application from Github using AWS CodeDeploy.

Let’s start first by creating 2 IAM roles we will use in this tutorial:

  • IAM role for CodeDeploy to talk to EC2 instances.
  • IAM role for EC2 to access S3.

1 – CodeDeployRole

Go to AWS IAM Console  then navigate to “Roles“, choose “Create New Role“, Select “CodeDeploy” and attach “AWSCodeDeployRole” policy:

2 – EC2S3Role

Create another IAM role, but this time choose EC2 as the trusted entity. Then, attach “AmazonS3ReadOnlyAccess” policy:

Now that we’ve created an IAM roles, let’s launch an EC2 instance which will be used by CodeDeploy to deploy our application.

3 – EC2 Instance

Launch a new EC2 instance with the IAM role we created in last section:

Next to User Data type the following script to install the AWS CodeDeploy Agent at boot time:

Note: make sure to allow HTTP traffic in the security group.

Once created, connect to the instance using the Public IP via SSH, and verify whether the CodeDeploy agent is running:

4 – Application

Add the appspec.yml file to the application to describe to AWS CodeDeploy how to manage the lifecycle of your application:

The BeforeInstall, will install apache server:

The AfterInstall will restart apache server

5 – Setup CodeDeploy

Go to AWS CodeDeploy and create a new application:

Select In-Place deployement (with downtime):

Click on “Skip“, because we already setup our EC2 instance:

The above will take you to the following page where you need to give a name to your application:

Select the EC2 instance and assign a name to the deployment group:

Select the CodeDeployRole we created in the first part of the tutorial:

Then click on “Deploy“:

Create a deployment, select Github as the data source:

Just select “Connect to GitHub“. Doing that will pop up a new browser window, take you to Github login where you will have to enter your username and password

After that come back to this page, and you should see something like below, just enter the remaining details and click “Deploy

This will take you to a page as follows:

If you point your browser to the EC2 public IP, you should see:

Now, let’s automate the deployment using Github Integrations.

6 – Continuous Deployment

Go to IAM Dashboard, and create a new policy which give access to register and create a new deployment from Github.

Next, create a new user and attach the policy we created before:

Note: copy to clipboard the user AWS ACCESS ID & AWS SECRET KEY. Will come handy later.

7 – Github Integration

Generate a new token to invoke CodeDeploy from Github:

Once the token is generated, copy the token and keep it. Then, add AWS CodeDeploy integration:

Fill the fields as below:

Finally, add Github Auto Deployment

Fill the form as below:

To test it out, let’s edit a file or commit a new file. You should see a new deployment on AWS CodeDeploy: