Preventing race conditions in Docker

It’s easy to get race conditions with Compose & Docker. Take for example, if you have a common pattern when you have the application server depends on the database, but since the database server didn’t have time to configure itself and application has already started it would just failed connecting for it.

A race condition example with NodeJS app & MySQL:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

var MySQL = require('mysql'),
    express = require('express'),
    app = express();

var connection = MySQL.createConnection({
  host : process.env.MYSQL_HOST || 'localhost',
  user : process.env.MYSQL_USER || '',
  password  : process.env.MYSQL_PASSWORD || ''
});

connection.connect(function(err){
  if(err){
    console.log('error connecting:', err.stack);
    process.exit(1);
  }
  console.log('connected as id:', connection.threadId);
})

app.get('/', function(req, res){
  res.send('Hello world :)');
})

app.listen(3000, function(){
  console.log('Server started ....');
})

To build the application container, I used the following Dockerfile:

1
2
3
4
5
6
7
8
9
10
11
12

FROM node:8.7.0
MAINTAINER mlabouardy <mohamed@labouardy.com>

WORKDIR /app

RUN npm install mysql express

COPY server.js .

EXPOSE 3000

CMD node server.js

To deploy the stack, I used docker-compose:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

version: "3.0"

services:
  mysql:
    image: mysql:5.6
    environment:
      - MYSQL_ROOT_PASSWORD=root
    networks:
      - db-net

  app:
    build: .
    ports:
      - 3000:3000
    environment:
      - MYSQL_HOST=mysql
      - MYSQL_USER=root
      - MYSQL_PASSWORD=root
    networks:
      - db-net

networks:
  db-net:
    driver: bridge

Let’s build the image:

1

docker-compose build

Then, create the containers:

1

docker-compose up -d 

Let’s see the status:

1

docker-compose ps

The application failed to start, lets see why ?

1

docker-compose logs -f app

RACE CONDITION ! The application container come up before the DB and tried to connect to MySQL database and fail with a database connection error. To avoid that, There are many solutions:

• Adding a mechanism in the code to wait for DB to be up and setup before starting to connect to it
• Using restart policy – Docker Docs
• Holding the container until the database is up and running

I will go with the 3rd solution, an open source tool called Dockerize, the advantage of this tool is that’s its pretty fast to just look over the opening the socket until it’s getting open and then launch the web app.

Note: Dockerize gives you the ability to wait for services on a specified protocol (file, tcp, tcp4, tcp6, http, https and unix)

So just update the Dockerfile to install Dockerize:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

FROM node:8.7.0
MAINTAINER mlabouardy <mohamed@labouardy.com>

RUN apt-get update && apt-get install -y wget

ENV DOCKERIZE_VERSION v0.5.0
RUN wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
    && tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
    && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz

WORKDIR /app

RUN npm install mysql express

COPY server.js .

EXPOSE 3000

CMD dockerize -wait tcp://mysql:3306 -timeout 1m && node server.js

Then, build the new image:

1
2

docker-compose up -d
docker-compose ps

1

docker-compose logs -f app

Its working !

Drop your comments, feedback, or suggestions below — or connect with me directly on Twitter @mlabouardy.