If you are working on big data intensive application in Laravel you must know how to process time-consuming process in the background.
Laravel queues are great way to run background jobs. By default laravel supports following types of queues:
- Database
- Redis
- Beanstalk
- Amazon SQS
In this tutorial I will teach you how we can integrate pub/sub driver to process background jobs. You should have basic knowledge of queue and google cloud console in order to start pub/sub integration.
I will walk you through step by step.
- Login to Google Cloud Console
-
Now, create your first topic by clicking this link: Create Topic in PubSub
- If you do not wish to click above link you can login to google cloud console
- then in left menu item find Pub/Sub and click on that menu item
- You will see following screen:
Click on create topic in my case I will create a topic named test-queue. Once you have created a topic in pub/sub next thing is to create credentials that we can use in our laravel project.
Create Pub/Sub credentials
Login to Google Cloud Console. Go to Left menu and find APIs & Services menu item, once you find it you will see following screen. Hit on create credentials button.
Choose Service account from the dropdown you will be presented to new screen as shown below fill out name for your service account.
Once you click on continue button after entering name of the service account you would be presented with following screen where you would have to select the role.
Select Pub/Sub Editor role which allow us to manage pub/sub topics and subscriptions.
Click on continue button once role has been selected and click on done button. Now, open your newly created service account you will see following screen.
Click on Keys tab, and then click on ADD KEY button, it will take you to following screen where you can select json from the radio option and hit create.
This action will download service credentials json file on your computer with proper credentials. We will use this json file later in our tutorial with our pub/sub integration.
Back to Laravel Project and adding pub/sub library
I have created a small library that integrates pub/sub in your laravel project go ahead and download this gitlab project using below command:
composer require l2t/laravel-pubsub-queue
So far we added dependency in our laravel project. Let's add required configurations in config/queue.php file as shown below under connections key:
<?php
return [
/*
|--------------------------------------------------------------------------
| Default QueueProcessor Connection Name
|--------------------------------------------------------------------------
|
| Laravel's queue API supports an assortment of back-ends via a single
| API, giving you convenient access to each back-end using the same
| syntax for every one. Here you may define a default connection.
|
*/
'default' => env('QUEUE_CONNECTION', 'sync'),
/*
|--------------------------------------------------------------------------
| QueueProcessor Connections
|--------------------------------------------------------------------------
|
| Here you may configure the connection information for each server that
| is used by your application. A default configuration has been added
| for each back-end shipped with Laravel. You are free to add more.
|
| Drivers: "sync", "database", "beanstalkd", "sqs", "redis", "null"
|
*/
'connections' => [
'sync' => [
'driver' => 'sync',
],
// OTHER DRIVERS GOES HERE
'pubsub' => [
'retries' => 3,
'driver' => 'pubsub',
'request_timeout' => 60,
'queue' => env('PUBSUB_DEFAULT_QUEUE'),
'project_id' => env('GOOGLE_CLOUD_PROJECT'),
'queue_prefix' => env('PUBSUB_QUEUE_PREFIX', ''),
'keyFilePath' => storage_path(env('GOOGLE_APPLICATION_CREDENTIALS')),
],
],
/*
|--------------------------------------------------------------------------
| Failed QueueProcessor Jobs
|--------------------------------------------------------------------------
|
| These options configure the behavior of failed queue job logging so you
| can control which database and table are used to store the jobs that
| have failed. You may change them to any database / table you wish.
|
*/
'failed' => [
'driver' => env('QUEUE_FAILED_DRIVER', 'database-uuids'),
'database' => env('DB_CONNECTION', 'mysql'),
'table' => 'failed_jobs',
],
];
We would store our credentials json file in laravel storage directory. Let's create a new directory under storage folder to store our json file:
# go to laravel project on your local cd laravel-app # create a new secrets folder in storage directory # make sure in your config/queue file keyFilePath points to this storage dir correctly mkdir -p storage/secrets
So far we added pubsub configs in config/queue.php file. Let's now create necessary environment variables that we would need in our laravel project.
Open .env file and add following variables:
# pub/sub driver QUEUE_CONNECTION=pubsub # google credentials PUBSUB_DEFAULT_QUEUE=test-queue GOOGLE_CLOUD_PROJECT=test-project GOOGLE_APPLICATION_CREDENTIALS=secrets/test-project-b70a851167ff.json
Make sure to change above values, in first part of our tutorial we created topic name and generated credentials json file if you remember we need to input this data here.
Create and run laravel job in pub/sub
Now, that we installed the library and added all necessary config and env variables. We can now test our integration to see if it is working correctly.
Let's create a new job in laravel using following command:
php artisan make:job PubSubJob
Above command will generate a new PubSubJob class in app/Jobs directory. Let's open this file and add single line that logs string in laravel log file.
<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Support\Facades\Log;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
class PubSubJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
/**
* Create a new job instance.
*
* @return void
*/
public function __construct()
{
$this->onQueue(env('PUBSUB_DEFAULT_QUEUE'));
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
Log::info("this job was processed by pub/sub.");
}
}
Now that we have created this new job what are we looking here?
- We want this job to run in background using pub/sub
- By default all jobs will go to default queue mentioned in our config/queue.php file
How to trigger background job to run in pub/sub?
Now, you might be thinking how I can trigger above job to run in pub/sub right. Well you can dispatch this job in your laravel controller.
There are few ways you can dispatch your jobs:
# following code dispatch job to default queue
dispatch(new App\Jobs\PubSubJob());
# following code dispatch job to specified queue
dispatch(new App\Jobs\PubSubJob())->onQueue(env('PUBSUB_DEFAULT_QUEUE'));
Once your controller hit above line you will see following message when you run below command:
php artisan queue:work pubsub
Above command will pickup your dispatched job and will show a message like this:
Once job is processed you can check your laravel log file stored in storage/log/laravel.log you will see your new job logged following line.
[2022-05-13 19:58:06] local.INFO: this job was processed by pub/sub.
Common Problems [FAQs]
-
My job is not picked by pub/sub?
- make sure your env variable set correctly to QUEUE_CONNECTION=pubsub
-
I have correct queue connection however my job is not picked by laravel?
- make sure you have put your google credential file in storage/secrets directory and name of the file matches with env variable GOOGLE_APPLICATION_CREDENTIALS
If you still have any issues with installation let me know by using contact us page. My github repo is hosted below: