Komiser: AWS Environment Inspector

In order to build HA & Resilient applications in AWS, you need to assume that everything will fail. Therefore, you always design and deploy your application in multiple AZ & regions. So you end up with many unused AWS resources (Snapshots, ELB, EC2, Elastic IP, etc) that could cost you a fortune.

One pillar of AWS Well-Architected Framework is Cost optimization. That’s why you need to have a global overview of your AWS Infrastructure. Fortunately, AWS offers many fully-managed services like CloudWatch, CloudTrail, Trusted Advisor & AWS Config to help you achieve that. But, they require a deep understanding of AWS Platform and they are not straighforward.

That’s why I came up with Komiser a tool that simplifies the process by querying the AWS API to fetch information about almost all critical services of AWS like EC2, RDS, ELB, S3, Lambda … in real-time in a single Dashboard.

Note: To prevent excedding AWS API rate limit for requests, the response is cached in in-memory cache by default for 30 minutes.

Komiser supported AWS Services:

  • Compute:
    • Running/Stopped/Terminated EC2 instances
    • Current EC2 instances per region
    • EC2 instances per family type
    • Lambda Functions per runtime environment
    • Disassociated Elastic IP addresses
    • Total number of Key Pairs
    • Total number of Auto Scaling Groups
  • Network & Content Delivery:
    • Total number of VPCs
    • Total number of Network Access Control Lists
    • Total number of Security Groups
    • Total number of Route Tables
    • Total number of Internet Gateways
    • Total number of Nat Gateways
    • Elastic Load Balancers per family type (ELB, ALB, NLB)
  • Management Tools:
    • CloudWatch Alarms State
    • Billing Report (Up to 6 months)
  • Database:
    • DynamoDB Tables
    • DynamoDB Provisionned Throughput
    • RDS DB instances
  • Messaging:
    • SQS Queues
    • SNS Topics
  • Storage:
    • S3 Buckets
    • EBS Volumes
    • EBS Snapshots
  • Security Identity & Compliance:
    • IAM Roles
    • IAM Policies
    • IAM Groups
    • IAM Users

1 – Configuring Credentials

Komiser needs your AWS credentials to authenticate with AWS services. The CLI supports multiple methods of supporting these credentials. By default the CLI will source credentials automatically from its default credential chain. The common items in the credentials chain are the following:

  • Environment Credentials
  • Shared Credentials file (~/.aws/credentials)
  • EC2 Instance Role Credentials

To get started, create a new IAM user, and assign to it this following IAM policy:

Next, generate a new AWS Access Key & Secret Key, then update ~/.aws/credentials file as below:

2 – Installation

2.1 – CLI

Find the appropriate package for your system and download it. For linux:

Note: The Komiser CLI is updated frequently with support for new AWS services. To see if you have the latest version, see the project Github repository.

After you install the Komiser CLI, you may need to add the path to the executable file to your PATH variable.

2.2 – Docker Image

Use the official Komiser Docker Image:

3 – Overview

Once installed, start the Komiser server:

If you point your favorite browser to http://localhost:3000, you should see Komiser Dashboard:

Hope it helps ! The CLI is still in its early stages, so you are welcome to contribute to the project on Github.

Serverless Application with Flutter & Lambda

Few days ago, Google has announced the beta release of Flutter at Mobile World Congress 2018. A mobile UI framework to build native apps for both iOS and Android. It uses Dart to write application. The code is compiled using the standard Android and iOS toolchains for the specifc mobile platform, hence, better performance and startup times.

Flutter has a lot of benefits such as:

  • Open Source.
  • Hot reload for quicker development cycle.
  • Native ARM code runtime.
  • Rich widget set & growing community of plugins backed by Google.
  • Excellent editor integretation: Android Studio & Visual Studio Code.
  • Single codebase for iOS and Android, full native performance (does not use JavaScript as a bridge or WebViews) .
  • React Native competitor.
  • Dart feels more Java, easy for Java developers to jump on it.
  • It use Widgets, so for people coming from web developement background everything should seem very familiar.
  • It might end the Google vs Oracle Java wars.

So it was a great opportunity to get my hands dirty and create a Flutter application based on Serverless Golang API I built in my previous post “Serverless Golang API with AWS Lambda

The Flutter application will call API Gateway which will invoke a Lambda Function that will use TMDB API to get a list of movies airing this week in theatres in real-time. The application will consume the JSON response and display results in a ListView.

Note: All code can be found on my GitHub.

To get started, follow my previous tutorial on how to create a Serverless API, once deployed, copy to clipboard the API Gateway Invoke URL.

Next, get the Flutter SDK by cloning the following GitHub repository:

Note: Make sure to add flutter/bin folder to your PATH environment variable.

Next, install the missing dependencies and SDK files:

Start Android Studio, and install Flutter plugin from File>Settings>Plugins :

Create a new Flutter project:

Note: Flutter comes with a CLI that you can use to create a new project “flutter create PROJECT_NAME

Android Studio will generate the files for a basic Flutter sample app, we will work in lib/main.dart file:

Run the app. You should see the following screen:

Create a simple POJO class Movie with a set of attributes and getters:

Create a widget, TopMovies, which creates it’s State, TopMoviesState. The state class will maintain a list of movies.

Add the stateful TopMovies widget to main.dart:

Add the TopMoviesState class. Most of the app’s logic will resides in this class.

Let’s initialize our _movies variable with a list of movies by invoking API Gateway, we will iterate through the JSON response, and add the movies using the _addMovie function:

The _addMovie() function will simply add the movie passed as an argument to list of _movies:

Now we just need to display movies in a scrolling ListView. Flutter comes with a suit of powerful basic widgets. In the code below I used the Text, Row, Image widgets. In addition to Padding & Align components to display properly a Movie item:

Finally, update the build method for MyApp to call the TopMovies widget instead:

Restart the app. You should see a list of movies airing today in cinema !

That’s it ! we have successfully created a Serverless application in just 143 lines of code and it works like a charm on both Android and iOS.

Flutter is still in womb so of course it has some cons:

  • Steep learning curve compared to React Native which uses JavaScript.
  • Unpopular comparing to Kotlin or Java.
  • Does not support 32-bit iOS devices.
  • Due to autogenerated code, the build artifact is huge (APK for Android is almost 25 Mb, While I built the same app in Java for 3 Mb).

But for a beta release it look very stable and well designed. I can’t wait to see what you can build with it !

Setup Raspberry PI 3 as AWS VPN Customer Gateway

In my previous article, I showed you how to use a VPN Software Solution like OpenVPN to create a secure tunnel to your AWS private resources. In this post, I will walk you through step by step on how to setup a secure bridge to your remote AWS VPC subnets from your home network with a Raspberry PI as a Customer Gateway.

To get started, find your Home Router public-facing IP address:

Next, sign in to AWS Management Console, navigate to VPC Dashboard and create a new VPN Customer Gateway:

Next, create a Virtual Private Gateway:

And attach it to the target VPC:

Then, create a VPN Connection with the Customer Gateway and the Virtual Private Gateway:

Note: Make sure to add your Home CIDR subnet to the Static IP Prefixes section.

Once the VPN Connection is created, click on “Tunnel Details” tab, you should see two tunnels for redundancy:

It may take a few minutes to create the VPN connection. When it’s ready, select the connection and choose “Download Configuration“, and open the configuration file and write down your Pre-shared-key and Tunnel IP:

I used a Raspberry PI 3 (Quand Core CPU 1.2 GHz, 1 GB RAM) with Raspbian, with SSH server enabled (default username & password: pi/raspberry), you can login and start manipulating the PI:

IPsec kernel support must be installed. Therefore, you must install openswan on your PI:

Update the /etc/ipsec.conf file as below:

Create a new IPsec Connection in /etc/ipsec.d/home-to-aws.conf :

  • left: Your Raspberry PI private IP.
  • leftid: Your Home Router public-facing IP.
  • leftsubnet: CIDR of your Home Subnet.
  • right: Virtual Private Gateway Tunnel IP.
  • rightsubnet: CIDR of your VPC.

Add the tunnel pre-shared key to /var/lib/openswan/ipsec.secrets.inc:

To enable the IPv4 forwarding, edit /etc/sysctl.conf, and ensure the following lines are uncommented:

Run sysctl -p to reload it. Then, restart IPsec service:

Verify if the service is running correctly:

If you go back to your AWS Dashboard, you should see the 1st tunnel status changed to UP:

Add a new route entry that forwards traffic to your home subnet through the VPN Gateway:

Note: Follow the same steps above to setup the 2nd tunnel for resiliency & high availablity of VPN connectivity.

Launch an EC2 instance in the private subnet to verify the VPN connection:

Allow SSH only from your Home Gateway CIDR:

Connect via SSH using the instance private ip address:


Congratulations ! you can now connect securely to your private EC2 instances.

To take it further and connect from other machines in the same Home Network, add a static route as described below:

1 – Windows

2 – Linux

3 – Mac OS X

Test it out:


AWS OpenVPN Access Server

Being able to access AWS resources directly in secure way can be very useful. To achieve this you can:

  • Setup a dedicated connection with AWS Direct Connect
  • Use a Network Appliance
  • Software Defined Private Network like OpenVPN

In this post, I will walk you through how to create an OpenVPN server on AWS, to connect securely to your VPC, Private Network resources and applications from any device anywhere.

To get started, sign in to your AWS Management Console and launch an EC2 instance from the OpenVPN Access Server AWS Marketplace offering:

For demo purpose, choose t2.micro:

Use the default settings with the exception of “Enable termination protection” as we dont want our VPN being terminated on accident:

Assign a new Security Group as below:

  • TCP – 22 : Remote access to the instance.
  • TCP – 443 : HTTPS, this is the interface used by users to log on to the VPN server and retrieve their keying and installation information.
  • TCP – 943 : OpenVPN Admin Web Dashboard.
  • UDP – 1194 : OpenVPN UDP Port.

To ensure our VPN instance Public IP address doesnt change if it’s stopped, assign to it an Elastic IP:

For simplicity, I added an A record in Route 53 which points to the instance Elastic IP:

Once the AMI is successfully launched, you will need to connect to the server via SSH using the DNS record:

On first time connecting, you will be prompted and asked to setup the OpenVPN server:

Setup a new password for the openvpn admin user:

Point your browser to https://openvpn.slowcoder.com, and login using openvpn credentials

Download the OpenVPN Connect Client, after your installation is complete, click on “Import” then “From server” :

Then type the OpenVN DNS name:

Enter your openvpn as the username and enter the same password as before and click on “connect“:

After you are connected, you should see a green check mark:

To verify the client is connected, login to OpenVPN Admin Dashboard on https://openvpn.slowcoder.com/admin :

Finally, create a simple web server instance in a private subnet to verify the VPN is working:

If you point your browser to the webserver private address, you should see a simple HTML page

Network Infrastructure Weathermap

The main goal of collecting metrics is to store them for long term usage and to create graphs to debug problems or identify trends. However, storing metrics about your system isn’t enough to identity the problem’s & anomalies root cause. It’s necessary to have a high-level overview of your network backbone. Weathermap is perfect for a Network Operations Center (NOC). In this post, I will show you how to build one using Open Source tools only.

Icinga 2 will collect metrics about your backbone, write checks results metrics and performance data to InfluxDB (supported since Icinga 2.5). Visualize these metrics in Grafana in map form.

To get started, add your desired host configuration inside the hosts.conf file:

Note: the city & country attributes will be used to create the weathermap.

To enable the InfluxDBWriter on your Icinga 2 installation, type the following command:

Configure your InfluxDB host and database in /etc/icinga2/features-enabled/influxdb.conf (Learn more about the InfluxDB configuration)

Icinga 2 will forward all your metrics to a icinga2_metrics database. The included host and service templates define a storage, the measurement represents a table by which metrics are grouped with tags certain measurements of certain hosts or services are defined (notice the city & country tags usage).

Don’t forget to restart Icinga 2 after saving your changes:

Once Icinga 2 is up and running it’ll start collecting data and writing them to InfluxDB:

Once our data arrived, it’s time for visualization. Grafana is widely used to generate graphs and dashboards. To create a Weathermap we can use a Grafana plugin called Worldmap Panel. Make sure to install it using grafana-cli tool:

The plugin will be installed into your grafana plugins directory (/var/lib/grafana/plugins):

Restart Grafana, navigate to Grafana web interface and create a new datasource:

Create a new Dashboard:

The Group By clause should be the country code and an alias is needed too. The alias should be in the form $tag_field_name. See the image below for an example of a query:

Under the Worldmap tab, choose the countries option:

Finally, you should see a tile map of the world with circles representing the state of each host.

The field state possible values (0 – OK, 1 – Warning, 2 – Critical, 3 – Unknown/Unreachable)

Note: For lazy people I created a ready to use Dashboard you can import from GitHub.