Laravel9 Queue System Part-2

In this tutorial we will learn about different types of configuration for laravel queue system. If you have not read about first part of Laravel queue system I would recommend you read the first part:

Laravel9 Queue System Part-1

Laravel queue connections configuration

Laravel's queue configuration options are stored in your application's config/queue.php configuration file. Here is the sample file look like:

<?php

return [

    /*
    |--------------------------------------------------------------------------
    | Default Queue Connection Name
    |--------------------------------------------------------------------------
    |
    | Laravel's queue API supports an assortment of back-ends via a single
    | API, giving you convenient access to each back-end using the same
    | syntax for every one. Here you may define a default connection.
    |
    */

    'default' => env('QUEUE_CONNECTION', 'sync'),

    /*
    |--------------------------------------------------------------------------
    | Queue Connections
    |--------------------------------------------------------------------------
    |
    | Here you may configure the connection information for each server that
    | is used by your application. A default configuration has been added
    | for each back-end shipped with Laravel. You are free to add more.
    |
    | Drivers: "sync", "database", "beanstalkd", "sqs", "redis", "null"
    |
    */

    'connections' => [

        'sync' => [
            'driver' => 'sync',
        ],

        'database' => [
            'driver' => 'database',
            'table' => 'jobs',
            'queue' => 'default',
            'retry_after' => 90,
            'after_commit' => false,
        ],

        'beanstalkd' => [
            'driver' => 'beanstalkd',
            'host' => 'localhost',
            'queue' => 'default',
            'retry_after' => 90,
            'block_for' => 0,
            'after_commit' => false,
        ],

        'sqs' => [
            'driver' => 'sqs',
            'key' => env('AWS_ACCESS_KEY_ID'),
            'secret' => env('AWS_SECRET_ACCESS_KEY'),
            'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
            'queue' => env('SQS_QUEUE', 'default'),
            'suffix' => env('SQS_SUFFIX'),
            'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
            'after_commit' => false,
        ],

        'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => env('REDIS_QUEUE', 'default'),
            'retry_after' => 90,
            'block_for' => null,
            'after_commit' => false,
        ],

    ],

    /*
    |--------------------------------------------------------------------------
    | Failed Queue Jobs
    |--------------------------------------------------------------------------
    |
    | These options configure the behavior of failed queue job logging so you
    | can control which database and table are used to store the jobs that
    | have failed. You may change them to any database / table you wish.
    |
    */

    'failed' => [
        'driver' => env('QUEUE_FAILED_DRIVER', 'database-uuids'),
        'database' => env('DB_CONNECTION', 'mysql'),
        'table' => 'failed_jobs',
    ],

];

In this file you can define different types of connection drivers that you can use for your queue. You can have multiple drivers defined in this file:

  • sync: file queue driver
  • database: using database queue driver
  • redis: using redis queue driver
  • sqs: using amazon sqs queue driver
  • beanstalkd: using beanstalkd queue driver

Even though you have configured multiple driver does not mean laravel will use all of them. Laravel will look at your .env file for this setting:

QUEUE_CONNECTION=sync

It will use defined connection set using QUEUE_CONNECTION env variable.

Connection vs Queues

Before getting started with Laravel queues, it is important to understand the distinction between "connections" and "queues". As we learned above you can confugure different connections for supported queue drivers in laravel in config/queue.php file.

Queues are different consider queues as a bucket. You can name a bucket that contains different types of jobs. For example: you can create different buckets like:

  • payments
  • orders

you can tell laravel to push your jobs to specific bucket. You can configure your worker to process specific buckets. Let's take a look at some examples below:

ProcessOrderJob::dispatch()->onQueue('orders');
ProcessPaymentJob::dispatch()->onQueue('payments');

Alternatively, you can set a $queue public property in the job class:

namespace App\Jobs;

class ProcessPaymentJob implements ShouldQueue
{
    public $queue = 'payments';
}

You can configure your worker to process specific bucket or queue:

# this worker only deals with orders queue or bucket
php artisan queue:work --queue=orders

# this worker only deals with payments queue or bucket
php artisan queue:work --queue=payments

High Priority Jobs

By default laravel sends job to default queue. However, you can dispatch jobs to multiple queues.

# sends job to default queue
ProcessShipmentJob::dispatch();

# sends jobs to orders queue
ProcessOrderJob::dispatch()->onQueue('orders');

# sends jobs to payments queue
ProcessPaymentJob::dispatch()->onQueue('payments');

You might be thinking why do we need high priority jobs. Let say that your application is sending all jobs to default queue. All jobs are stacked if you want to process your customer payment you would have to wait until other jobs in the queues are done.

This may take a while considering different jobs are being pushed to same default queue. I would rather create a new queue called payments and push payment jobs to this queue.

Later you can run dedicated workers that only process this payments queue as high priority. This way your worker is only responsible for handling single type of queue.

Let's instruct our workers to process jobs from the payments queue:

php artisan queue:work --queue=payments,default

This will instruct the worker to pop—or dequeue—jobs from the payments and default queues with the payments queue having a higher priority.

You can configure a separate worker for each one:

# this worker only deals with orders queue or bucket
php artisan queue:work --queue=orders

# this worker only deals with payments queue or bucket
php artisan queue:work --queue=payments

How to tune workers to get best out of them

Let say that you configure a worker processing jobs from the payments queue while another worker is processing jobs from the default queue.

php artisan queue:work --queue=payments
php artisan queue:work --queue=default

This ensures that there's always a worker processing jobs from payments or default queue. Consider this scenario, let say that your payment queue is full while your default queue is empty.

Your default queue worker is empty and sitting idle while your payment queue has been full and it has jobs pending to process.

It's better if we instruct the workers to process jobs from the other queue if their main queue is empty. To do this, we'll start our workers with these configurations:

php artisan queue:work --queue=payments,default
php artisan queue:work --queue=default,payments

Now, the first worker is going to consume jobs from the payments queue as long as it has jobs; if not, it'll consume jobs from the default queue.

The second worker will consume jobs from the default queue if it's busy; otherwise, it'll start looking at the payments queue.

In next tutorial we will learn more advance queue configurations.