Skip to content

Background Job Processing with Sidekiq

Introduction

Sidekiq is a background job processor for Ruby that leverages Redis for managing job queues. It enables the efficient execution of tasks asynchronously, which helps offload long-running operations from the main application thread, thereby improving performance and responsiveness.

Sidekiq is widely used for handling background tasks such as sending emails, processing uploads, interacting with external APIs, and executing scheduled jobs.

Integrating Sidekiq with a Rails Application

To use Sidekiq, we need to add the sidekiq gem to our application. Additionally, we include sidekiq-scheduler for managing recurring jobs.

gem 'sidekiq'
gem 'sidekiq-scheduler'

Configuration

The configuration file is typically located in config/sidekiq.yml and defines job queues and priorities.

Example configuration:

:queues:
- [internal, 100]
- [sync, 2]
- [publish, 1]
- [delete_orphaned_resources, 1]
- [default, 1]

Warning

The numbers next to queue names indicate their priority. A higher number means a higher priority.

We also need to configure Sidekiq options in config/initializers/sidekiq.rb. This file contains Redis configuration, logging settings, and additional options that facilitate the use of Sidekiq.

config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1') }
...
clients_list = ENV.fetch('SIDEKIQ_CLIENTS_LIST', '').downcase.split(',')

Warning

The SIDEKIQ_CLIENTS_LIST parameter is crucial, as it defines the list of clients supported on the server. Based on this list, cyclic jobs are retrieved for execution. Jobs for each client are stored in separate files under config/sidekiq/{client_name}.yml.

Defining a Sidekiq Job

Job files are located in app/jobs/api/v3/{job_name}_job.rb. To create a job, define a class inheriting from API::V3::BaseJob.

Scheduling Recurring Jobs

For recurring jobs, we use the sidekiq-scheduler gem, which allows tasks to be executed at specified intervals.

Jobs for each client are stored in separate files under config/sidekiq/{client_name}.yml.

Example:

ESA-publish_articles:
  cron: '0 * * * * *'
  class: API::V3::ResourcePublicationJob
  args: [{ client_identifier: 'EKSTRAKLASA', resource_name: 'Article' }]

The schedule is loaded in an initializer:

schedule.merge!(YAML.load_file(File.expand_path("../../sidekiq/#{client}.yml", __FILE__)))

Recurring Jobs:

Below are some examples of common recurring jobs that Sidekiq executes in our API:

  • COMMON-delete_orphaned_* - Runs daily to remove orphaned records and optimize the database.
  • {Client}-publish_* – Runs every minute to publish a resource.
  • {Client}-sync_* - Runs daily to synchronize data from external services.

Summary

Sidekiq is an essential tool in our API for:

  • Asynchronous execution of background jobs, reducing load on the main application.
  • Efficient management of job queues with Redis.
  • Scheduling and handling recurring tasks.

By utilizing Sidekiq, we enhance the performance, scalability, and efficiency of our API.