CI/CD for Lambda Functions with Jenkins

The following post will walk you through how to build a CI/CD pipeline to automate the deployment process of your Serverless applications and how to use features like code promotion, rollbacks, versions, aliases and blue/green deployment. At the end of this post, you will be able to build a pipeline similar to the following figure:

For the sake of simplicity, I wrote a simple Go based Lambda function that calculates the Fibonacci number:

I implemented also a couple of unit tests for both the Fibonacci recursive and Lambda handler functions:

To create the function in AWS Lambda and all the necessary AWS services, I used Terraform. An S3 bucket is needed to store all the deployment packages generated through the development lifecycle of the Lambda function:

The build server needs to interact with S3 bucket and Lambda functions. Therefore, an IAM instance role must be created with S3 and Lambda permissions:

An IAM role is needed for the Lambda function as well:

Finally, a Go-based Lambda function will be created with the following properties:


Next, build the deployment package with the following commands:

Then, issue the terraform apply command to create the resources:

Sign in to AWS Management Console and navigate to Lambda Console, a new function called “Fibonacci” should be created:

You can test it out, by mocking the input from the “Select a test event” dropdown list:

If you click on “Test” button the Fibonacci number of 7 will be returned:

So far our function is working as expected. However, how can we ensure each changes to our codebase doesn’t break things ? That’s where CI/CD comes into play, the idea is making all code changes and features go through a complex pipeline before integrating them to the master branch and deploying it to production.

You need a Jenkins cluster with at least a single worker (with Go preinstalled), you can follow my previous post for a step by step guide on how to build a Jenkins cluster on AWS from scratch.

Prior to the build, the IAM instance role (created with Terraform) with the write access to S3 and the update operations to Lambda must be configured on the Jenkins workers:

Jump back to Jenkins Dashboard and create new multi-branch project and configure the GitHub repository where the code source is versioned as follows:

Create a new file called Jenkinsfile, it defines a set of steps that will be executed on Jenkins (This definition file must be committed to the Lambda function’s code repository):

The pipeline is divided into 5 stages:

  • Checkout: clone the GitHub repository.
  • Test: check whether our code is well formatted and follows Go best practices and run unit tests.
  • Build: build a binary and create the deployment package.
  • Push: store the deployment package (.zip file) to an S3 bucket.
  • Deploy: update the Lambda function’s code with the new artifact.

Note the usage of the git commit ID as a name for the deployment package to give a meaningful and significant name for each release and be able to roll back to a specific commit if things go wrong.

Once the project is saved, a new pipeline should be created as follows:

Once the pipeline is completed, all stages should be passed, as shown in the next screenshot:

At the end, Jenkins will update the Lambda function’s code with the update-function- code command:

If you open the S3 Console, then click on the bucket used by the pipeline, a new deployment package should be stored with a key name identical to the commit ID:

Finally, to make Jenkins trigger the build when you push to the code repository, click on “Settings” from your GitHub repository, then create a new webhook from “Webhooks”, and fill it in with a URL similar to the following:

In case you’re using Git branching workflows (you should), Jenkins will discover automatically the new branches:

Hence, you must separate your deployment environments to test new changes without impacting your production. Therefore, having multiple versions of your Lambda functions makes sense.

Update the Jenkinsfile to add a new stage to publish a new Lambda function’s version, every-time you push (or merge) to the master branch:


On the master branch, a new stage called “Published” will be added:

As a result, a new version will be published based on the master branch source code:

However, in agile based environment (Extreme programming). The development team needs to release iterative versions of the system often to help the customer to gain confidence in the progress of the project, receive feedback and detect bugs in earlier stage of development. As a result, small releases can be frequent:

AWS services using Lambda functions as downstream resources (API Gatewayas an example) need to be updated every-time a new version is published -> operational overhead and downtime. USE aliases !!!

The alias is a pointer to a specific version, it allows you to promote a function from one environment to another (such as staging to production). Aliases are mutable, unlike versions, which are immutable.

That being said, create an alias for the production environment that points to the latest version published using the AWS command line:

You can now easily promote the latest version published into production by updating the production alias pointer’s value:

Like what you’re read­ing? Check out my book and learn how to build, secure, deploy and manage production-ready Serverless applications in Golang with AWS Lambda.

Deploy a Jenkins Cluster on AWS

Few months ago, I gave a talk at Nexus User Conference 2018 on how to build a fully automated CI/CD platform on AWS using Terraform, Packer & Ansible. I illustrated how concepts like infrastructure as code, immutable infrastructure, serverlesscluster discovery, etc can be used to build a highly available and cost-effective pipeline. The platform I built is given in the following diagram:

The platform has a Jenkins cluster with a dedicated Jenkins master and workers inside an autoscaling group. Each push event to the code repository will trigger the Jenkins master which will schedule a new build on one of the available slaves. The slave will be responsible of running the unit and pre-integration tests, building the Docker image, storing the image to a private registry and deploying a container based on that image to Docker Swarm cluster.

On this post, I will walk through how to deploy the Jenkins cluster on AWS using top trending automation tools.

The cluster will be deployed into a VPC with 2 public and 2 private subnets across 2 availability zones. The stack will consists of an autoscaling group of Jenkins workers in a private subnets and a private instance for the Jenkins master sitting behind an elastic Load balancer. To add or remove Jenkins workers on-demand, the CPU utilisation of the ASG will be used to trigger a scale out (CPU > 80%) or scale in (CPU < 20%) event. (See figure below)

To get started, we will create 2 AMIs (Amazon Machine Image) for our instances. To do so, we will use Packer, which allows you to bake your own image.

The first AMI will be used to create the Jenkins master instance. The AMI uses the Amazon Linux Image as a base image and for provisioning part it uses a simple shell script:

The shell script will be used to install the necessary dependencies, packages and security patches:


It will install the latest stable version of Jenkins and configure its settings:

  • Create a Jenkins admin user.
  • Create a SSH, GitHub and Docker registry credentials.
  • Install all needed plugins (Pipeline, Git plugin, Multi-branch Project, etc).
  • Disable remote CLI, JNLP and unnecessary protocols.
  • Enable CSRF (Cross Site Request Forgery) protection.
  • Install Telegraf agent for collecting resource and Docker metrics.

The second AMI will be used to create the Jenkins workers, similarly to the first AMI, it will be using the Amazon Linux Image as a base image and a script to provision the instance:

A Jenkins worker requires the Java JDK environment and Git to be installed. In addition, the Docker community edition (building Docker images) and a data collector (monitoring) will be installed.

Now our Packer template files are defined, issue the following commands to start baking the AMIs:


Packer will launch a temporary EC2 instance from the base image specified in the template file and provision the instance with the given shell script. Finally, it will create an image from the instance. The following is an example of the output:

Sign in to AWS Management Console, navigate to “EC2 Dashboard” and click on “AMI”, 2 new AMIs should be created as below:

Now our AMIs are ready to use, let’s deploy our Jenkins cluster to AWS. To achieve that, we will use an infrastructure as code tool called Terraform, it allows you to describe your entire infrastructure in templates files.

I have divided each component of my infrastructure to a template file. The following template file is responsible of creating an EC2 instance from the Jenkins master’s AMI built earlier:

Another template file used as a reference to each AMI built with Packer:

The Jenkins workers (aka slaves) will be inside an autoscaling group of a minimum of 3 instances. The instances will be created from a launch configuration based on the Jenkins slave’s AMI:

To leverage the power of automation, we will make the worker instance join the cluster automatically (cluster discovery) using Jenkins RESTful API:

At boot time, the user-data script above will be invoked and the instance private IP address will be retrieved from the instance meta-data and a groovy script will be executed to make the node join the cluster:


Moreover, to be able to scale out and scale in instances on demand, I have defined 2 CloudWatch metric alarms based on the CPU utilisation of the autoscaling group:

Finally, an Elastic Load Balancer will be created in front of the Jenkins master’s instance and a new DNS record pointing to the ELB domain will be added to Route 53:


Once the stack is defined, provision the infrastructure with terraform apply command:

The command takes an additional parameter, a variables file with the AWS credentials and VPC settings (You can create a new VPC with Terraform from here):

Terraform will display an execution plan (list of resources that will be created in advance), type yes to confirm and the stack will be created in few seconds:

Jump back to EC2 dashboards, a list of EC2 instances will created:

 

In the terminal session, under the Outputs section, the Jenkins URL will be displayed:

Point your favorite browser to the URL displayed, the Jenkins login screen will be displayed. Sign in using the credentials provided while baking the Jenkins master’s AMI:

If you click on “Credentials” from the navigation pane, a set of credentials should be created out of the box:

The same goes for “Plugins”, a list of needed packages will be installed also:

Once the Autoscaling group finished creating the EC2 instances, the instances will join the cluster automatically as you can see in the following screenshot:

You should now be ready to create your own CI/CD pipeline !

You can take this further and build a dynamic dashboard in your favorite visualisation tool like Grafana to monitor your cluster resource usage based on the metrics collected by the agent installed on each EC2 instance:

Drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy.

AWS Events Analysis with ELK

Recording your AWS environment activity is a must have. It can help you monitor your environment’s security continuously and detect suspicious or undesirable activity in real-time. Hence, saving thousands of dollars. Luckily, AWS offers a solution called CloudTrail that allow you to achieve that. It records all events in all AWS regions and logs every API calls in a single S3 bucket.

From there, you can setup an analysis pipeline using the popular logging stack ELK (ElasticSearch, Logstash & Kibana) to read those logs, parse, index and visualise them in a single dynamic dashboard and even take actions accordingly:

To get started, create an AMI with the ELK components installed and preconfigured. The AMI will be based on an Ubuntu image:

To provision the AMI, we will use the following shell script:

Now the template is defined, bake a new AMI with Packer:

Once the AMI is created, create a new EC2 instance based on the AMI with Terraform. Make sure to grant S3 permissions to the instance to be able to read CloudTrail logs from the bucket:

Issue the following command to provision the infrastructure:

Head back to AWS Management Console, navigate to CloudTrail, and click on “Create Trail” button:

Give it a name and apply the trail to all AWS regions:

Next, create a new S3 bucket on which the events will be stored on:

Click on “Create“, and the trail should be created as follows:

Next, configure Logstash to read CloudTrail logs on an interval basis. The geoip filter adds information about the geographical location of IP addresses, based on sourceIPAddress field. Then, it stores the logs to Elasticsearch automatically:

In order for the changes to take effect, restart Logstash with the command below:

A new index should be created on Elasticsearch (http://IP:9200/_cat/indices?v)

On Kibana, create a new index pattern that match the index format used to store the logs:

After creating index, we can start exploring our CloudTrail events:

Now that we have processed data inside Elasticsearch, let’s build some graphs. We will use the Map visualization in Kibana to monitor geo access to our AWS environment:

You can now see where the environment is being accessed from:

Next, create more widgets to display information about the identity of the user, the user agent and actions taken by the user. Which will look something like this:

You can take this further and setup alerts based on specific event (someone accesses your environment from an undefined location) to be alerted in near real-time.

Full code can be found on my GitHub. Make sure to drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy.

Docker on Elastic Beanstalk Tips

AWS Elastic Beanstalk is one of the most used PaaS today, it allows you to deploy your application without provisioning the underlying infrastructure while maintaining the high availability of your application. However, it’s painful to use due to the lack of documentation and real-world scenarios. In this post, I will walk you through how to use Elastic Beanstalk to deploy Docker containers from scratch. Followed by how to automate your deployment process with a Continuous Integration pipeline. At the end of this post, you should be familiar with advanced topics like debugging and monitoring of your applications in EB.

1 – Environment Setup

To get started, create a new Application using the following AWS CLI command:

Create a new environment. Let’s call it “staging” :

Head back to AWS Elastic Beanstalk Console, your new environment should be created:

Point your browser to the environment URL, a sample Docker application should be displayed:

Let’s deploy our application. I wrote a small web application in Go to return a list of Marvel Avengers (I see you Thanos 😉 )

Next, we will create a Dockerfile to build the Docker image. Go is a compiled language, therefore we can use the Docker multi-stage feature to build a lightweight Docker image:

Next, we create a Dockerrun.aws.json that describes how the container will be deployed in Elastic Beanstalk:

Now the application is defined, create an application bundle by creating a ZIP package:

Then, create a S3 bucket to store the different versions of your application bundles:

Issue the following command in order to copy the application into the bucket:

And create a new application version from the application bundle:

Finally, deploy the version to the staging environment:

Give it a few seconds while it’s deploying the new version:

Then, repoint your browser to the environment URL, a list of Avengers will be returned in a JSON format as follows:

Now that our Docker application is deployed, let’s automate this process by setting up a CI/CD pipeline.

2 – CI/CD Pipeline

I opt for CircleCI, but you’re free to use whatever CI server you’re familiar with. The same steps can be applied.

Create a circle.yml file with the following content:

The pipeline will firstly prepare the environment, installing the AWS CLI. Then run unit tests. Next, a Docker image will be built, then pushed to DockerHub. Last step is creating a new application bundle and deploying the bundle to Elastic Beanstalk.

In order to grant Circle CI permissions to call AWS operations, we need to create a new IAM user with following IAM policy:

Generate AWS access & secret keys. Then, head back to Circle CI and click on the project settings and paste the credentials :

Now, everytime you push a change to your code repository, a build will be triggered:

And a new version will be deployed automatically to Elastic Beanstalk:

3 – Monitoring

Monitoring your applications is mandatory. Unfortunately, CloudWatch doesn’t expose useful metrics like Memory usage of your applications in Elastic Beanstalk. Hence, in this part, we will solve this issue by creating our custom metrics.

I will install a data collector agent on the instance. The agent will collect metrics and push them to a time-series database.

To install the agent, we will use .ebextensions folder, on which we will create 3 configuration files:

  • 01-install-telegraf.config: install Telegraf on the instance

  • 02-config-file.config: create a Telegraf configuration file to collect system usage & docker containers metrics.

  • 03-start-telegraf.config: start Telegraf agent.

Once the application version is deployed to Elastic Beanstalk, metrics will be pushed to your timeseries database. In this example, I used InfluxDB as data storage and I created some dynamic Dashboards in Grafana to visualize metrics in real-time:

Containers:

Hosts:

Note: for in-depth explaination on how to configure Telegraf, InfluxDB & Grafana read my previous article.

Full code can be found on my GitHub. Make sure to drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy

Infrastructure Cost Optimization with Lambda

Having multiple environments is important to build a continuous integration/deployment pipeline and be able to reproduce bugs in production with ease but this comes at price. In order to reduce cost of AWS infrastructure, instances which are running 24/7 unnecessarily (sandbox & staging environments) must be shut down outside of regular business hours. 

The figure below describes an automated process to schedule, stop and start instances to help cutting costs. The solution is a perfect example of using Serverless computing.

Note: full code is available on my GitHub.

2 Lambda functions will be created, they will scan all environments looking for a specific tag. The tag we use is named Environment. Instances without an Environment tag will not be affected:

The StartEnvironment function will query the StartInstances method with the list of instance ids returned by the previous function:

Similarly, the StopEnvironment function will query the StopInstances method:

Finally, both functions will post a message to Slack channel for real-time notification:

Now our functions are defined, let’s build the deployment packages (zip files) using the following Bash script:

The functions require an IAM role to be able to interact with EC2. The StartEnvironment function has to be able to describe and start EC2 instances:

The StopEnvironment function has to be able to describe and stop EC2 instances:

Finally, create an IAM role for each function and attach the above policies:

The script will output the ARN for each IAM role:

Before jumping to deployment part, we need to create a Slack WebHook to be able to post messages to Slack channel:

Next, use the following script to deploy your functions to AWS Lambda (make sure to replace the IAM roles, Slack WebHook token & the target environment):

Once deployed, if you sign in to AWS Management Console, navigate to Lambda Console, you should see both functions has been deployed successfully:

StartEnvironment:

StopEnvironment:

To further automate the process of invoking the Lambda function at the right time. AWS CloudWatch Scheduled Events will be used.

Create a new CloudWatch rule with the below cron expression (It will be invoked everyday at 9 AM):

And another rule to stop the environment at 6 PM:

Note: All times are GMT time.

Testing:

a – Stop Environment

Result:

b – Start Environment

Result:

The solution is easy to deploy and can help reduce operational costs.

Full code can be found on my GitHub. Make sure to drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy.