In this tutorial we will learn about different types of configuration for laravel queue system. If you have not read about first part of Laravel queue system I would recommend you read the first part:
Laravel queue connections configuration
Laravel's queue configuration options are stored in your application's config/queue.php configuration file. Here is the sample file look like:
<?php
return [
/*
|--------------------------------------------------------------------------
| Default Queue Connection Name
|--------------------------------------------------------------------------
|
| Laravel's queue API supports an assortment of back-ends via a single
| API, giving you convenient access to each back-end using the same
| syntax for every one. Here you may define a default connection.
|
*/
'default' => env('QUEUE_CONNECTION', 'sync'),
/*
|--------------------------------------------------------------------------
| Queue Connections
|--------------------------------------------------------------------------
|
| Here you may configure the connection information for each server that
| is used by your application. A default configuration has been added
| for each back-end shipped with Laravel. You are free to add more.
|
| Drivers: "sync", "database", "beanstalkd", "sqs", "redis", "null"
|
*/
'connections' => [
'sync' => [
'driver' => 'sync',
],
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
'after_commit' => false,
],
'beanstalkd' => [
'driver' => 'beanstalkd',
'host' => 'localhost',
'queue' => 'default',
'retry_after' => 90,
'block_for' => 0,
'after_commit' => false,
],
'sqs' => [
'driver' => 'sqs',
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
'queue' => env('SQS_QUEUE', 'default'),
'suffix' => env('SQS_SUFFIX'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
'after_commit' => false,
],
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
'after_commit' => false,
],
],
/*
|--------------------------------------------------------------------------
| Failed Queue Jobs
|--------------------------------------------------------------------------
|
| These options configure the behavior of failed queue job logging so you
| can control which database and table are used to store the jobs that
| have failed. You may change them to any database / table you wish.
|
*/
'failed' => [
'driver' => env('QUEUE_FAILED_DRIVER', 'database-uuids'),
'database' => env('DB_CONNECTION', 'mysql'),
'table' => 'failed_jobs',
],
];
In this file you can define different types of connection drivers that you can use for your queue. You can have multiple drivers defined in this file:
- sync: file queue driver
- database: using database queue driver
- redis: using redis queue driver
- sqs: using amazon sqs queue driver
- beanstalkd: using beanstalkd queue driver
Even though you have configured multiple driver does not mean laravel will use all of them. Laravel will look at your .env file for this setting:
QUEUE_CONNECTION=sync
It will use defined connection set using QUEUE_CONNECTION env variable.
Connection vs Queues
Before getting started with Laravel queues, it is important to understand the distinction between "connections" and "queues". As we learned above you can confugure different connections for supported queue drivers in laravel in config/queue.php file.
Queues are different consider queues as a bucket. You can name a bucket that contains different types of jobs. For example: you can create different buckets like:
- payments
- orders
you can tell laravel to push your jobs to specific bucket. You can configure your worker to process specific buckets. Let's take a look at some examples below:
ProcessOrderJob::dispatch()->onQueue('orders');
ProcessPaymentJob::dispatch()->onQueue('payments');
Alternatively, you can set a $queue public property in the job class:
namespace App\Jobs;
class ProcessPaymentJob implements ShouldQueue
{
public $queue = 'payments';
}
You can configure your worker to process specific bucket or queue:
# this worker only deals with orders queue or bucket php artisan queue:work --queue=orders # this worker only deals with payments queue or bucket php artisan queue:work --queue=payments
High Priority Jobs
By default laravel sends job to default queue. However, you can dispatch jobs to multiple queues.
# sends job to default queue
ProcessShipmentJob::dispatch();
# sends jobs to orders queue
ProcessOrderJob::dispatch()->onQueue('orders');
# sends jobs to payments queue
ProcessPaymentJob::dispatch()->onQueue('payments');
You might be thinking why do we need high priority jobs. Let say that your application is sending all jobs to default queue. All jobs are stacked if you want to process your customer payment you would have to wait until other jobs in the queues are done.
This may take a while considering different jobs are being pushed to same default queue. I would rather create a new queue called payments and push payment jobs to this queue.
Later you can run dedicated workers that only process this payments queue as high priority. This way your worker is only responsible for handling single type of queue.
Let's instruct our workers to process jobs from the payments queue:
php artisan queue:work --queue=payments,default
This will instruct the worker to pop—or dequeue—jobs from the payments and default queues with the payments queue having a higher priority.
You can configure a separate worker for each one:
# this worker only deals with orders queue or bucket php artisan queue:work --queue=orders # this worker only deals with payments queue or bucket php artisan queue:work --queue=payments
How to tune workers to get best out of them
Let say that you configure a worker processing jobs from the payments queue while another worker is processing jobs from the default queue.
php artisan queue:work --queue=payments php artisan queue:work --queue=default
This ensures that there's always a worker processing jobs from payments or default queue. Consider this scenario, let say that your payment queue is full while your default queue is empty.
Your default queue worker is empty and sitting idle while your payment queue has been full and it has jobs pending to process.
It's better if we instruct the workers to process jobs from the other queue if their main queue is empty. To do this, we'll start our workers with these configurations:
php artisan queue:work --queue=payments,default php artisan queue:work --queue=default,payments
Now, the first worker is going to consume jobs from the payments queue as long as it has jobs; if not, it'll consume jobs from the default queue.
The second worker will consume jobs from the default queue if it's busy; otherwise, it'll start looking at the payments queue.
In next tutorial we will learn more advance queue configurations.