breaking into the unknown…

multiple project using sidekiq on same machine


In this post I have explained how to install , configure, start and stop sidekiq . You can start sidekiq from terminal with below command on your project root

$ bundle exec sidekiq -e staging -d -L log/sidekiq.log # -e option set the environment (staging in this case), -d option make the sidekiq to start as daemon i,e it will keep running even after you close the terminal, -L option set the log file path, you can get the list of all available option with -h as below

$ sidekiq -h
-c, –concurrency INT                           processor threads to use
-d, –daemon                                           Daemonize process
-e, –environment ENV                          Application environment
-g, –tag TAG                                            Process tag for procline
-i, –index INT                                         unique process index on this machine
-p, –profile                                              Profile all code run by Sidekiq
-q, –queue QUEUE[,WEIGHT]…       Queues to process with optional weights
-r, –require [PATH|DIR]                     Location of Rails application with workers or file to require
-t, –timeout NUM                                  Shutdown timeout
-v, –verbose                                            Print more verbose output
-C, –config PATH                                  path to YAML config file
-L, –logfile PATH                                 path to writable logfile
-P, –pidfile PATH                                 path to pidfile
-V, –version                                           Print version and exit
-h, –help                                                 Show help

O.K , so we know how to start sidekiq with all the possible option. But what happen if you start the sidekiq on two or more project on the same machine. In my case, My staging server host two projects, and all of them need sidekiq as background worker, so I moved to project folder and started sidekiq on both

ThirdPillar]# bundle exec sidekiq -e staging -d -L log/sidekiq.log
Barcelona]# bundle exec sidekiq -e staging -d -L log/sidekiq.log

Now I checked the running instance of sidekiq

ps -ef | grep sidekiq
root      3823     1  1 03:03 ?  00:00:38 sidekiq 2.12.0 ThirdPillar [0 of 25 busy]
root      4116     1  1 03:04 ?  00:00:35 sidekiq 2.12.0 Barcelona [0 of 25 busy]

So, I can see that, sidekiq have started on both the project and running on different port. But when I tried to load my application, all the pages which involve background processing start breaking. So the sidekiq not working . If I kill one sidekiq, the other start working. It is obvious that, the multiple sidekiq daemon conflicting with each other.

Now understand why is the conflict :

=> sidekiq work on redis as database server. basically it store the background processes on redis

=> One machine can have only single redis server, but redis can have multiple database on it

=> Unless you provide separate redis configuration for sidekiq, it will work with default i,e try to use redis on the local machine itself and the  database number 0 on it

So the problem is that, both sidekiq instance you have started, using the same redis server and the database number 0, so causing the conflict and none of them working. So now we have two solutions:

solution 1: configure sidekiq to point to different redis server

solution 2: configure sidekiq to use the same redis server available locally, but make them to use different database and also separate them with namespace

We will go for solution 2 . Let us try something with redis on console before changing sidekiq configuration.

> # no options is passed
=> #<Redis client v3.0.4 for redis://> # o number database being used
> 1) # database 1 is paased
=> #<Redis client v3.0.4 for redis://> # 1 number database being used
> 4)
=> #<Redis client v3.0.4 for redis://> 4 number database being used

Here, you can see that, your redis server is running at and number after / is the database number

So now we can introduce below line in config/initializer/sidekiq.rb file of both the project

config.redis = { :namespace => ‘ThirdPillar’, :url => ‘redis://’ }

config.redis = { :namespace => ‘Barcelona’, :url => ‘redis://’ }

The namespace can have any value, but as convention I have keep the name same as the folder name of the project. So the Barcelona will use datbase 1 of the redis server at redis:// . The modified configuration files for Barcelona look as below.


require 'sidekiq'
require 'sidekiq-status'

Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware
  config.redis = { :namespace => 'Barcelona', :url => 'redis://' }

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes # default
  config.redis = { :namespace => 'Barcelona', :url => 'redis://' }

Make similar change in sidekiq.rb of the other project. Restart the sidekiq on both the project, both will strat working  🙂



Author: arunyadav4u

over 7 years experience in web development with Ruby on Rails.Involved in all stage of development lifecycle : requirement gathering, planing, coding, deployment & Knowledge transfer. I can adept to any situation, mixup very easily with people & can be a great friend.

2 thoughts on “multiple project using sidekiq on same machine

  1. Was looking for the same things. Thanks for posting this 🙂

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s