AWS Events Analysis with ELK

Recording your AWS environment activity is a must have. It can help you monitor your environment’s security continuously and detect suspicious or undesirable activity in real-time. Hence, saving thousands of dollars. Luckily, AWS offers a solution called CloudTrail that allow you to achieve that. It records all events in all AWS regions and logs every API calls in a single S3 bucket.

From there, you can setup an analysis pipeline using the popular logging stack ELK (ElasticSearch, Logstash & Kibana) to read those logs, parse, index and visualise them in a single dynamic dashboard and even take actions accordingly:

To get started, create an AMI with the ELK components installed and preconfigured. The AMI will be based on an Ubuntu image:

To provision the AMI, we will use the following shell script:

Now the template is defined, bake a new AMI with Packer:

Once the AMI is created, create a new EC2 instance based on the AMI with Terraform. Make sure to grant S3 permissions to the instance to be able to read CloudTrail logs from the bucket:

Issue the following command to provision the infrastructure:

Head back to AWS Management Console, navigate to CloudTrail, and click on “Create Trail” button:

Give it a name and apply the trail to all AWS regions:

Next, create a new S3 bucket on which the events will be stored on:

Click on “Create“, and the trail should be created as follows:

Next, configure Logstash to read CloudTrail logs on an interval basis. The geoip filter adds information about the geographical location of IP addresses, based on sourceIPAddress field. Then, it stores the logs to Elasticsearch automatically:

In order for the changes to take effect, restart Logstash with the command below:

A new index should be created on Elasticsearch (http://IP:9200/_cat/indices?v)

On Kibana, create a new index pattern that match the index format used to store the logs:

After creating index, we can start exploring our CloudTrail events:

Now that we have processed data inside Elasticsearch, let’s build some graphs. We will use the Map visualization in Kibana to monitor geo access to our AWS environment:

You can now see where the environment is being accessed from:

Next, create more widgets to display information about the identity of the user, the user agent and actions taken by the user. Which will look something like this:

You can take this further and setup alerts based on specific event (someone accesses your environment from an undefined location) to be alerted in near real-time.

Full code can be found on my GitHub. Make sure to drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy.

One-shot containers with Serverless

Have you ever had short lived containers like the following use cases:

  • Batch and ETL (Extract, Transform & Load) Jobs.
  • Database backups and synchronisation.
  • Machine Learning algorithms for generation of learning and training models.
  • Integration & Sanity tests.
  • Web scrapers & crawlers.

And you were wondering how you can deploy your container periodically or in response to an event ? The answer is by using Lambda itself, the idea is by making a Lambda function trigger a deployment of your container from the build server. The following figure illustrates how this process can be implemented:

 

I have wrote a simple application in Go to simulate a short time process using sleep method:

As Go is a complied language, I have used Docker multi-stage build feature to build a lightweight Docker image with the following Dockerfile:

Next, I have a simple CI/CD workflow in Jenkins, the following is the Jenkinsfile used to build the pipeline:

An example of the pipeline execution is given as follows:

Now, all changes to the application will trigger a new build on Jenkins which will build the new Docker image, push the image to a private registry and deploy the new Docker image to the Swarm cluster:

If you issue the “docker service logs APP_NAME” on one of the cluster managers, your application should be working as expected:

Now our application is ready, let’s make execute everyday at 8am using a Lambda function. The following is the entrypoint (handler) that will be executed on each invocation of the function:

It uses the Jenkins API to trigger the deployment process job.

Now the function is defined, use the shell script below to create the following:

  • Build a deployment package (.zip file).
  • Create an IAM role with permissions to push logs to CloudWatch.
  • Create a Go based Lambda function from the deployment package.
  • Create a CloudWatch Event rule that will be executed everyday at 8am.
  • Make the CloudWatch Event invoke the Lambda function.

As a result, a Lambda function will be created as follows:

To test it out, you can invoke it manually either from the Lambda Console or using the following AWS CLI command:

A new deployment should be triggered in Jenkins and your application should be deployed once again:

That’s it, it was a quick example on how you can use Serverless with Containers, you can go further and use Lambda functions to scale out/scale in your services in your Swarm/Kubernetes cluster by using either CloudWatch events for expected increasing traffic (Holidays, Black Friday …) or other AWS managed services like API Gateway in response to incoming client requests.

Full code can be found on my GitHub. Make sure to drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy.

Docker on Elastic Beanstalk Tips

AWS Elastic Beanstalk is one of the most used PaaS today, it allows you to deploy your application without provisioning the underlying infrastructure while maintaining the high availability of your application. However, it’s painful to use due to the lack of documentation and real-world scenarios. In this post, I will walk you through how to use Elastic Beanstalk to deploy Docker containers from scratch. Followed by how to automate your deployment process with a Continuous Integration pipeline. At the end of this post, you should be familiar with advanced topics like debugging and monitoring of your applications in EB.

1 – Environment Setup

To get started, create a new Application using the following AWS CLI command:

Create a new environment. Let’s call it “staging” :

Head back to AWS Elastic Beanstalk Console, your new environment should be created:

Point your browser to the environment URL, a sample Docker application should be displayed:

Let’s deploy our application. I wrote a small web application in Go to return a list of Marvel Avengers (I see you Thanos 😉 )

Next, we will create a Dockerfile to build the Docker image. Go is a compiled language, therefore we can use the Docker multi-stage feature to build a lightweight Docker image:

Next, we create a Dockerrun.aws.json that describes how the container will be deployed in Elastic Beanstalk:

Now the application is defined, create an application bundle by creating a ZIP package:

Then, create a S3 bucket to store the different versions of your application bundles:

Issue the following command in order to copy the application into the bucket:

And create a new application version from the application bundle:

Finally, deploy the version to the staging environment:

Give it a few seconds while it’s deploying the new version:

Then, repoint your browser to the environment URL, a list of Avengers will be returned in a JSON format as follows:

Now that our Docker application is deployed, let’s automate this process by setting up a CI/CD pipeline.

2 – CI/CD Pipeline

I opt for CircleCI, but you’re free to use whatever CI server you’re familiar with. The same steps can be applied.

Create a circle.yml file with the following content:

The pipeline will firstly prepare the environment, installing the AWS CLI. Then run unit tests. Next, a Docker image will be built, then pushed to DockerHub. Last step is creating a new application bundle and deploying the bundle to Elastic Beanstalk.

In order to grant Circle CI permissions to call AWS operations, we need to create a new IAM user with following IAM policy:

Generate AWS access & secret keys. Then, head back to Circle CI and click on the project settings and paste the credentials :

Now, everytime you push a change to your code repository, a build will be triggered:

And a new version will be deployed automatically to Elastic Beanstalk:

3 – Monitoring

Monitoring your applications is mandatory. Unfortunately, CloudWatch doesn’t expose useful metrics like Memory usage of your applications in Elastic Beanstalk. Hence, in this part, we will solve this issue by creating our custom metrics.

I will install a data collector agent on the instance. The agent will collect metrics and push them to a time-series database.

To install the agent, we will use .ebextensions folder, on which we will create 3 configuration files:

  • 01-install-telegraf.config: install Telegraf on the instance

  • 02-config-file.config: create a Telegraf configuration file to collect system usage & docker containers metrics.

  • 03-start-telegraf.config: start Telegraf agent.

Once the application version is deployed to Elastic Beanstalk, metrics will be pushed to your timeseries database. In this example, I used InfluxDB as data storage and I created some dynamic Dashboards in Grafana to visualize metrics in real-time:

Containers:

Hosts:

Note: for in-depth explaination on how to configure Telegraf, InfluxDB & Grafana read my previous article.

Full code can be found on my GitHub. Make sure to drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy

Infrastructure Cost Optimization with Lambda

Having multiple environments is important to build a continuous integration/deployment pipeline and be able to reproduce bugs in production with ease but this comes at price. In order to reduce cost of AWS infrastructure, instances which are running 24/7 unnecessarily (sandbox & staging environments) must be shut down outside of regular business hours. 

The figure below describes an automated process to schedule, stop and start instances to help cutting costs. The solution is a perfect example of using Serverless computing.

Note: full code is available on my GitHub.

2 Lambda functions will be created, they will scan all environments looking for a specific tag. The tag we use is named Environment. Instances without an Environment tag will not be affected:

The StartEnvironment function will query the StartInstances method with the list of instance ids returned by the previous function:

Similarly, the StopEnvironment function will query the StopInstances method:

Finally, both functions will post a message to Slack channel for real-time notification:

Now our functions are defined, let’s build the deployment packages (zip files) using the following Bash script:

The functions require an IAM role to be able to interact with EC2. The StartEnvironment function has to be able to describe and start EC2 instances:

The StopEnvironment function has to be able to describe and stop EC2 instances:

Finally, create an IAM role for each function and attach the above policies:

The script will output the ARN for each IAM role:

Before jumping to deployment part, we need to create a Slack WebHook to be able to post messages to Slack channel:

Next, use the following script to deploy your functions to AWS Lambda (make sure to replace the IAM roles, Slack WebHook token & the target environment):

Once deployed, if you sign in to AWS Management Console, navigate to Lambda Console, you should see both functions has been deployed successfully:

StartEnvironment:

StopEnvironment:

To further automate the process of invoking the Lambda function at the right time. AWS CloudWatch Scheduled Events will be used.

Create a new CloudWatch rule with the below cron expression (It will be invoked everyday at 9 AM):

And another rule to stop the environment at 6 PM:

Note: All times are GMT time.

Testing:

a – Stop Environment

Result:

b – Start Environment

Result:

The solution is easy to deploy and can help reduce operational costs.

Full code can be found on my GitHub. Make sure to drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy.

Deploy a Swarm Cluster with Alexa

Serverless and Containers changed the way we leverage public clouds and how we write, deploy and maintain applications. A great way to combine the two paradigms is to build a voice assistant with Alexa based on Lambda functions – written in Go – to deploy a Docker Swarm cluster on AWS.

The figure below shows all components needed to deploy a production-ready Swarm cluster on AWS with Alexa.

Note: Full code is available on my GitHub.

A user will ask Amazon Echo to deploy a Swarm Cluster:

Echo will intercept the user’s voice command with built-in natural language understanding and speech recognition. Convey them to the Alexa service. A custom Alexa skill will convert the voice commands to intents:

The Alexa skill will trigger a Lambda function for intent fulfilment:

The Lambda Function will use the AWS EC2 API to deploy a fleet of EC2 instances from an AMI with Docker CE preinstalled (I used Packer to bake the AMI to reduce the cold-start of the instances). Then, push the cluster IP addresses to a SQS:

Next, the function will insert a new item to a DynamoDB table with the current state of the cluster:

Once the SQS received the message, a CloudWatch alarm (it monitors the ApproximateNumberOfMessagesVisible parameter) will be triggered and as a result it will publish a message to an SNS topic:

The SNS topic triggers a subscribed Lambda function:

The Lambda function will pull the queue for a new cluster and use the AWS System Manager API to provision a Swarm cluster on the fleet of EC2 instances created earlier:

For debugging, the function will output the Swarm Token to CloudWatch:

Finally, it will update the DynamoDB item state from Pending to Done and delete the message from SQS.

You can test your skill on your Amazon Echo, Echo Dot, or any Alexa device by saying, “Alexa, open Docker

At the end of the workflow described above, a Swarm cluster will be created:

At this point you can see your Swarm status by firing the following command as shown below:

Improvements & Limitations:

  • Lambda execution timeout if the cluster size is huge. You can use a Master Lambda function to spawn child Lambda.
  • CloudWatch & SNS parts can be deleted if SQS is supported as Lambda event source (AWS PLEAAASE !). DynamoDB streams or Kinesis streams cannot be used to notify Lambda as I wanted to create some kind of delay for the instances to be fully created before setting up the Swarm cluster. (maybe Simple Workflow Service ?)
  • Inject SNS before SQS. SNS can add the message to SQS and trigger the Lambda function. We won’t need CloudWatch Alarm.
  • You can improve the Skill by adding new custom intents to deploy Docker containers on the cluster or ask Alexa to deploy the cluster on a VPC

In-depth details about the skill can be found on my GitHub. Make sure to drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy.