Laravel9 Queue System Part-1

While building a large scale laravel application sometime you come across scenarios where you have to perform a task but that task may take too long to finish.

For better user experience you do not want your users to be waiting until your task is completed. For example: a large csv file upload that may take an hour or more.

You do not want user to wait for an hour rather you want to run this process in the background and let your user know that it will take time and update progress about import in real time.

By moving time intensive tasks to a queue, your application can respond to web requests with blazing speed and provide a better user experience to your customers.

Laravel queue system has three main components: messages, queues, and workers.

What are queues in laravel 9?

A queue is a linear data structure in which elements are added to the end and can be removed from the top. Queues works on a principle of first-in-first-out (FIFO).

What it means, when element is pushed to queue it will be added to the queue it will be stacked in the same order in which they added. When queue is processed first message that was added will be processed first.

What are messages in queue?

A message is basically text or string. An application can send a message from one part of the application and can be received from other part of the application.

In queue a sender sends a message and on the other end receiver recieves the message. When a message is sent sender does not need to worry about what happens to the message after it is sent.

Same as when receiver receives a message it does not need to know who the sender is.

What are workers in queue system?

A worker is a long-running process that watches if messages are arrived. When it sees a message it will dequeue the message and process it.

Here's an example worker code:

while(true) {
   $message = dequeue();
   process($message);
}

Once you start a worker, it'll keep dequeuing messages from your queue. You can start multiple workers, more workers you have more faster your messages will be processed.

Queues in Laravel 9

Laravel ships with a powerful queue system right out of the box. It supports multiple drivers for storing messages:

• Amazon SQS
• Redis
• Datbase
• Beanstalkd

In laravel you have to create a job class. Let say that you have a job that is responsible for processing large csv file. In laravel 9 you can use following command to create a new job.

php artisan make:job ProcessCsvFile

Once you run this command it will create a file in app/Jobs directory. Let's look at the sample file content:

<?php
 
namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Http\UploadedFile;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Contracts\Queue\ShouldQueue;
 
class ProcesCsvFile implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
 
    /**
     * Create a new job instance.
     */
    public function __construct(
        public UploadedFile $file,
    ) {}

    /**
     * Execute the job.
     */
    public function handle(): void
    {
        // Process uploaded podcast...
    }
}

Job class contains handle method in which you can write your logic related to performing csv upload. You can pass an argument to this class in constructor.

In above class we will pass uploaded file as an argument to this class. Later your handle method can use this passed argument to perform some tasks.

Purpose of creating job class is to push this job into laravel queues. Now, we need to learn how we can push our jobs to different queues.

How to push laravel job to queue?

Let say that you create a job that process a large csv file. In order to process this file in background we would have to dispatch this job to a queue.

There are different methods you can use in order to queue your job in laravel:

use App\Jobs\ProcesCsvFile;
use Illuminate\Support\Facades\Bus;

// sends job to default queue
// using static dispatch function
ProcesCsvFile::dispatch($file);

// sends job to default queue
// using laravel bus facade
Bus::dispatch(new ProcesCsvFile($file));

// sends job to default queue
// using dispatch function
dispatch(new ProcesCsvFile($file));

// sends job to csv-queue queue
// using static dispatch function
ProcesCsvFile::dispatch($file)->onQueue('csv-queue');

When you enqueue a job, the queue manager will serialize it and build a string payload for the message body. When a worker dequeues this job, it will execute the handle() method. Inside that method, you should put all your business logic.

Setting up a laravel worker

Without worker your job has no meaning. You have to have worker setup in order for your job to execute. In laravel there are two ways you can run your worker.

# using queue:work
php artisan queue:work

# using queue:listen
php artisan queue:listen

The main difference between queue:work and queue:listen is shown below:

• queue:listen reflects your changes right way without restarting the worker
• queue:work needs to be restarted before reflecting the changes you made

queue:work command is more efficient than the queue:listen version because it has to bootstrap laravel app only single time and then it does not need to boot it again.

Why Use a Message Queue?

There are several benefits of using a message queue:

• Better user experience: user does not need to wait while running longer tasks
• Fault tolerance: Queue systems are built to persist jobs until they are processed. In the case of failure, you can configure your jobs to be retried several times.
• Scalability:  For applications with unpredictable load, it's a waste of money to allocate more resources all the time even when there's no load. With queues, you can control the rate at which your workers consume jobs.
• Batching: Batching a large task into several smaller tasks is more efficient.
• Ordering: you can ensure that certain jobs will be processed in a specific order.
• Rate limiting: you can also limit the number of jobs running concurrently.

In next tutorial, we will learn more about laravel queue system in depth.