Butler CLI: Export/Import Jenkins Plugins & Jobs

Not long ago, I had to migrate Jenkins jobs from an old server to a new one. That’s where StackOverflow comes into the play, below the most voted answers I found:

In spite of their advantages, those solutions comes with their downsides especially if you have a large number of jobs to move or no access root to the server. But, guess what ? I didn’t stop there. I have came up with a CLI to make your life easier and export/import not only Jenkins jobs but also plugins like a boss.

To get started, find the appropriate package for your system and download it. For linux:

Note: For Windows make sure that butler binary is available on the PATHThis page contains instructions for setting the PATH on Windows.

Once done, verify the installation worked, by opening a new terminal session and checking if butler is available :

1 – Plugins Management

To export Jenkins jobs, you need to provide the URL of the source Jenkins instance:

As shown above, butler will dump a list of plugins installed to stdout and a new file plugins.txt will be generated, with list of installed Jenkins plugins with name and version pairs:

Now, to import the plugins to the new Jenkins instance, use the command below with the URL of the Jenkins target instance as an argument:

Butler will install each plugin on the target Jenkins instance by issuing API calls.

2 – Jobs Management

To export Jenkins jobs, just provide the URL of the source Jenkins server:

A new directory jobs/ will be created with every job in Jenkins. Each job will have its own configuration file config.xml.

Now, to import the jobs to the new Jenkins instance, issue the following command:

Butler will use the configuration files created earlier to issue API calls to target Jenkins instance to create jobs.

Once you are done, check Jenkins and you should see your jobs successfully created :

Hope it helps ! The CLI is still in its early stages, so you are welcome to contribute to the project in Github.

Cleanup old Docker images from Nexus Repository

Many of us, are using Nexus as a repository to publish Docker Images. Typically we build images tagged with the commit hash (or using semver ideally) after SCM change automatically in CI and we push them to registry. As result there are many “unneeded” & “old” images that in our case take significant amount of disk space.

I looked around the graphical interface of Nexus and there’s apparently nothing to remove several Docker images at the same time. Or even, a scheduled task  to clean up old hosted Docker images, and to also clean up layers which are no longer used by any hosted images.

So I have come up with a simple bash script which uses Docker Registry API to purge Docker images and keep the last X images and delete all other. But, is there a better solution ? YES ! I built a Nexus CLI

To install Nexus CLI, find the appropriate package for your system and download it. For linux:

After downloading Nexus CLI. Add the execution permission to the binary:

Note: For Windows make sure that nexus-cli binary is available on the PATHThis page contains instructions for setting the PATH on Windows.

After installing, verify the installation worked, by opening a new terminal session and checking if nexus-cli is available :

Once done, configure the Nexus credentials:

Through nexus-cli configure, the Nexus CLI will prompt you for four pieces of information. The Username and Password are your account credentials. Nexus Hostname & Docker repository name.

That should be it. Try out the following command from your cmd prompt and, if you have any images, you should see them listed

Display image tags:

Image description:

To remove a specific image:

To keep only the last X images and delete all other:

That’s it ! Let’s go back to Nexus Dashboard:

As you can see, Nexus kept only the last 4 images and deleted the others.

Résultat de recherche d'images pour "awesome meme"

The CLI is still in its early stages, so you are welcome to contribute to the project in Github.

Continuous Deployment with AWS CodeDeploy & Github

This post will walk you through how to AutoDeploy your application from Github using AWS CodeDeploy.

Let’s start first by creating 2 IAM roles we will use in this tutorial:

  • IAM role for CodeDeploy to talk to EC2 instances.
  • IAM role for EC2 to access S3.

1 – CodeDeployRole

Go to AWS IAM Console  then navigate to “Roles“, choose “Create New Role“, Select “CodeDeploy” and attach “AWSCodeDeployRole” policy:

2 – EC2S3Role

Create another IAM role, but this time choose EC2 as the trusted entity. Then, attach “AmazonS3ReadOnlyAccess” policy:

Now that we’ve created an IAM roles, let’s launch an EC2 instance which will be used by CodeDeploy to deploy our application.

3 – EC2 Instance

Launch a new EC2 instance with the IAM role we created in last section:

Next to User Data type the following script to install the AWS CodeDeploy Agent at boot time:

Note: make sure to allow HTTP traffic in the security group.

Once created, connect to the instance using the Public IP via SSH, and verify whether the CodeDeploy agent is running:

4 – Application

Add the appspec.yml file to the application to describe to AWS CodeDeploy how to manage the lifecycle of your application:

The BeforeInstall, will install apache server:

The AfterInstall will restart apache server

5 – Setup CodeDeploy

Go to AWS CodeDeploy and create a new application:

Select In-Place deployement (with downtime):

Click on “Skip“, because we already setup our EC2 instance:

The above will take you to the following page where you need to give a name to your application:

Select the EC2 instance and assign a name to the deployment group:

Select the CodeDeployRole we created in the first part of the tutorial:

Then click on “Deploy“:

Create a deployment, select Github as the data source:

Just select “Connect to GitHub“. Doing that will pop up a new browser window, take you to Github login where you will have to enter your username and password

After that come back to this page, and you should see something like below, just enter the remaining details and click “Deploy

This will take you to a page as follows:

If you point your browser to the EC2 public IP, you should see:

Now, let’s automate the deployment using Github Integrations.

6 – Continuous Deployment

Go to IAM Dashboard, and create a new policy which give access to register and create a new deployment from Github.

Next, create a new user and attach the policy we created before:

Note: copy to clipboard the user AWS ACCESS ID & AWS SECRET KEY. Will come handy later.

7 – Github Integration

Generate a new token to invoke CodeDeploy from Github:

Once the token is generated, copy the token and keep it. Then, add AWS CodeDeploy integration:

Fill the fields as below:

Finally, add Github Auto Deployment

Fill the form as below:

To test it out, let’s edit a file or commit a new file. You should see a new deployment on AWS CodeDeploy:

How ChatOps can help you DevOps better !

When people hear DevOps, they often relate it to “automation”, “team work” and many “tools”, which is right. DevOps is all about CAMS : a culture of automation, measurement and sharing. The purpose of this article is to show how can ChatOps boost DevOps by bringing CAMS to everyday practice.

What is DevOps ?

Set of practices that emphasize the collaboration and communication of both Software Engineers and IT & Infrastructure Operations to reduce the time to market of a Product. One main goal of DevOps is to deploy features into production quickly and to detect and correct problems when they occur, without disrupting other services (Self-healing, Blue/Green deployment, Canary updates …)

There are several guidelines in order to achieve DevOps maturity, here are a few you need to know:

  • Continuous Integration is the practice of integrating, building and testing code within the development environment. It requires developers team to integrate code into a shared repository (Version Control System). Checks out the code and runs all the pre-deployment tests (Do not require code to be deployed to a server). In case it passed compile and package the code into an Artifact (JAR, Docker Image, gem …). Push it into an Artifact Repository Manager. The Artifact is deployed then inside an immutable container to a test environment (Quality Assurance). Once deployed post-deployment tests (Functional, integration & performance tests) are executed.
  • Continuous Delivery is an extension to continuous integration pipeline. It aims to make any change to the system releasable, it requires a person or business rule to decide when the final push to production should occur.
  • Continuous Deployment is an advanced evolution of continuous delivery. It’s the practice of deploying all the way into production without any human intervention.

In addition to those practices discussed above. Today, most DevOps teams embrace collaborative messaging platforms, such as Slack, to communicate with each other. And get real time updates about the system through online chat. And that’s certainly the spirit behind ChatOps.

What is ChatOps ?

“Placing tools directly in the middle of the conversation” — Jesse Newland, GitHub

Collaboration and conversation are a force that let work and learn together to produce new things. And this is happening in an exponential way that accelerates every year.

ChatOps (an amalgamation of chat and operations) is an emerging movement that led to ease the integration between teams and various tools/platforms of DevOps and others. It is about conversation driven development by bringing the tools into conversations. Robots are today members of your team to whom you can send a request and get an instant response.

ChatOps is a model where people, tools, process and automation are connected in a transparent flow. It also helps collaborate and control pipelines in one window.

Today, DevOps practices toolchain puts in many grades of tools, including : development software, networks and servers management, tests, monitoring, etc. Collaborating and controlling the practices’ pipelines in one window has helped developers teams work in more efficient and agile way.

Three main components in Chatops:

Collaboration tool: It’s the Chat Client where stakeholders and teams are connected between them and to the systems they work on. The are several chat platforms :

  • Slack: a leading chat platform for teams which accumulated more that 4 million daily active users. It is also one of the first platforms that integrated bots into its system.
  • HipChat by Atlassian is a group chat, file sharing, video chat & screen sharing built for teams & business.

Bot: It is the core of chatops methodology. The Bot comes in the middle of the collaboration tool and the Devops tools.
The Bot receives requests from team members, retrieves information from integrated systems by executing set of commands (scripts).

  • Hubot, a leading bot tool for ChatOps. It is a valuable open source robot (CoffeeScript) for automating chat rooms by Github made back in 2013. Hubot is useful and powerful via Scripts. They define the skills of your Hubot instance. Hundreds of them are written and maintained by the community and you can create your own. It mainly helps to automate most of ops-related tasks.
  • Lita is a framework for bots dedicated to company chat rooms written in Ruby. It is heavily inspired from Hubot. This framework can be used to build operational task automations and has a very comprehensive list of plugins which means that it can be integrated in many chat platforms as : Slack, Facebook Messenger and others.
  • Cog, made by Operable, is another chatbot framework to help automate DevOps workflows. It’s designed to be chat platform and language agnostic, and uses a Unix-style pipeline to activate complex functionality.
  • ErrBot is a chatbot daemon that generates bots that are in the middle of a chat platform and DevOps tools. It is written in Python and aims to make integrating and tool that provides an API, easily to a chat platform via commands.

System integration: Third key element in ChatOps. Simply put, it is the DevOps tools allowing more productivity. Such as :

  • Issue tracking: JIRA, OTRS, TeamForge …
  • Version Control Systems: Github, Gitlab, Bitbucket …
  • Infrastructure as Code (IaC): Terraform, Vagrant, Packer, Swarm, Kubernetes, Docker, AWS CloudFormation …
  • Configuration Management tools (Provisioning): Ansible, Salt, Chef, Puppet …
  • Continuous Integration Servers: Jenkins, Travis CI, Bamboo …
  • Monitoring: Grafana, Kibana, Prometheus …

Nowadays ChatOps are operational. Several teams around the world already connected their chat platforms to their build systems to get notifications, query and execute processes on their continuous integration servers. Same thing happens in QA (Quality Assurance) teams, support teams and the rest.

With ChatOps, trust is build among the team especially that work is shared and brought into the foreground by putting it all in one place. Your chat platform is your new command line.

Conversation-driven collaboration is not new, but with ChatOps, we observed a combination of the oldest form of collaboration and the newest techies. We are not surprised that this combination have changed the way staff members work. This should make people often think of making software that allows this collaboration to be more contributory, soft and secure.

For further Information on DevOps and ChatOps approaches, check out our DevOps Wiki on Github.

Thanks for reading,
With ❤ by Rania Zyane and Mohamed Labouardy.