Prevent Job Overlapping with - WithoutOverlapping

WithoutOverlapping middleware of Laravel Jobs will add your duplicate job to the queue and prevent that duplicate job being executed while another job of same id is still processing. Useful only if you have multiple queue workers that is able to run multiple queued jobs at once.

WithoutOverlapping

Does not work without a key passed to the WithoutOverlapping.
Key passed to WithoutOverlapping will create the lock by combining the job class name.
- Lock only applies to jobs of same class not with other classes even with the same id in the WithoutOverlapping middleware.
Does not work if you have only one queue worker that execute one job at a time.
Only check job of same id is processing or not. Does not check queue for duplicate jobs.
If you have multiple queue workers
- As soon as duplicate queue start to process while original job is still processing it will throw an exception.
Prevent only simultaneous execution of the same job.

WithoutOverlapping with dontRelease

Make sure to have multiple queue workers to test this.
Does not add duplicate job to the queue if original job is still processing.
- Therefore it does not throw exceptions.
- Free queue workers will not be able to process a duplicate job since duplicate jobs are not added to the queue since original one is still processing.

WithoutOverlapping with releaseAfter

Use if you want to execute those overlapping jobs.
Make sure to run queue worker with retry option. (ex: queue:work –tries=3)
It will throw exceptions when duplicate jobs are trying to execute while original job is running.
It will throw exceptions even with dontRelease()
If there is many duplicate jobs some may not run after maximum retry count exceeded.

WithoutOverlapping with ExpireAfter

Laravel uses a locking mechanism called atomic lock feature.
This locking feature prevent other duplicate jobs running concurrently with WithoutOverlapping middleware.
If a job failed, php execution time expired, etc. can cause this lock to stay forever without releasing.
If lock does not released you won’t be able to run jobs ever.
Calculate time taken to execute the job. Then add extra minutes top of that for the lock expiration. (ex: expireAfter(180) )

Example

Let test an example of using WithoutOverlapping middleware and test how it works with an example.

Task:

Imagine you have a Laravel web application for a company. This web application uses multiple queue workers to process jobs. As a result multiple jobs are executed simultaneously. This Company has 3 departments. Report generation is the most resource hungry and time consuming task of the application. Imagine report generation process takes around 1 hour to complete. You should block requests send by the department 1 to create the reports if queue worker is creating the report for department 1. It should allow report generation request for department 2 since there is no processing on report generation for department 2.

Setup and create a sample Laravel Job

First setup a Laravel project. Then create a job class using artisan command. Let’s create CreateReports job class.

php artisan make:job CreateReports

Open recently created job class and modify handle() method.

Log information when handle method executed.
Add sleep() to simulate time consuming process.
Modify construct method to accept department id as a string.

namespace App\Jobs;

use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Log;

class CreateReports implements ShouldQueue
{
    use Queueable;

    /**
     * Create a new job instance.
     */
    public function __construct(public string $departmentId) {}

    /**
     * Execute the job.
     */
    public function handle(): void
    {
        sleep(10);
        Log::info("Report generated for the department: $this->departmentId");
    }
}

Dispatch the created job

For the simplicity of this example, we are going to use web.php file to dispatch above created job.

<?php

use App\Jobs\CreateReports;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Route;

Route::get('/', function () {
    $departmentId = 201;
    Log::info("Received get request $departmentId");
    CreateReports::dispatch($departmentId);
    
});

Generating the Job Overlapping issue

Run 3 queue workers by running below command on three terminal windows of your local environment.

php artisan queue:work

Now send few get requests to root route of the application. You will see same job executed on multiple queue workers simultaneously (check terminal windows where you ran queue:work command). We do not want to process many jobs that return same results at the same time. This is called Job Overlapping. Lets prevent job overlapping issue.

Prevent Job Overlapping

Here we are using WithoutOverlapping middleware on the job class. WithoutOverlapping middleware will prevent execution of the duplicate jobs while original job is executing.

<?php

namespace App\Jobs;

use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Log;

class CreateReports implements ShouldQueue
{
    use Queueable;

    /**
     * Create a new job instance.
     */
    public function __construct(public string $departmentId) {}

    /**
     * Execute the job.
     */
    public function handle(): void
    {
        sleep(10);
        Log::info("Report generated for the department: $this->departmentId");
    }

    public function middleware():array
    {
        return [new WithoutOverlapping($this->departmentId)];
    }

}

Testing the application

Run your web application using below command.

php artisan serve

Run 3 queue workers using 3 terminal windows.

php artisan queue:work

Now send 2 quests to root route and check Laravel logs file.

[2024-12-17 13:48:54] local.INFO: Received get request 201  
[2024-12-17 13:48:58] local.INFO: Received get request 201  
[2024-12-17 13:49:00] local.ERROR: App\Jobs\CreateReports has been attempted too many times. {"exception": ....
[2024-12-17 13:49:04] local.INFO: Report generated for the department: 201

According to log information only one report is generated for the department 201. Second request to generate the report has thrown a exception.

Prevent Job Overlapping without exceptions

You can use dontRelease in order to prevent exceptions while processing the duplicate jobs. dontRelease method prevent adding duplicate jobs to the queue. As a result there is no job to execute and throw exceptions.

<?php

namespace App\Jobs;

use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Queue\Middleware\RateLimited;
use Illuminate\Queue\Middleware\WithoutOverlapping;
use Illuminate\Support\Facades\Log;

class CreateReports implements ShouldQueue
{
    use Queueable;

    /**
     * Create a new job instance.
     */
    public function __construct(public string $departmentId) {}

    /**
     * Execute the job.
     */
    public function handle(): void
    {
        sleep(10);
        Log::info("Report generated for the department: $this->departmentId");
    }

    public function middleware():array
    {
        return [(new WithoutOverlapping($this->departmentId))->dontRelease()];
    }

}

If you check the log information again by sending few requests to create reports, you will see there is no exceptions.