Laravel 10 Import Large CSV File Using Queue - TechvBlogs

Laravel 10 Import Large CSV File Using Queue

In this article, You will learn How to import Large CSV file using Queue in Laravel 10.


Smit Pipaliya - Author - TechvBlogs
Smit Pipaliya
 

10 months ago

TechvBlogs - Google News

The time has come to place all of it together. We’ve slowly built on the previous articles, and the feature accelerated our pace. With batch placing, we should insert a hundred rows (even though you can select any number). With chunk analyzing, we eliminated the reminiscence hassle of loading the whole Excel sheet into reminiscence. Alternatively, we packed a hundred rows well worth of statistics (even though we may want to have extended or reduced that variety to whatever we desired).

But there’s still one more difficulty. What if the file is so massive that it times PHP out? How do you import a file that takes 5 mins to import? Even if PHP didn’t time out, could you want to make your consumer wait whilst the import becomes going for walks? Of direction no longer. That’s where queued imports shine.

How to Import with Queue?

The first component to notice is that queued imports most straightforward paintings with bite reading. That approach that the WithChunkReading difficulty has to be a gift. After chew analysis is carried out into the import, the handiest other problem left to implement is ShouldQueue. Could it be that easy? Yes.
 

<?php

namespace App\Imports;

use App\Models\User;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Support\Facades\Hash;
use Maatwebsite\Excel\Concerns\ToModel;
use Maatwebsite\Excel\Concerns\WithChunkReading;

class QueuedImport implements ToModel, WithChunkReading, ShouldQueue
{
    /**
    * @param array $row
    *
    * @return \Illuminate\Database\Eloquent\Model|null
    */
    public function model(array $row)
    {
        return new User([
            'name'     => $row['first_name'] . " " . $row['last_name'],
            'email'    => $row['email'],
            'password' => Hash::make($row['password']),
        ]);
    }

    public function chunkSize(): int
    {
        return 20;
    }
}

Let’s create our UserController method and add our route and test. I’ll spoil it slightly. It won’t work right away unless you have something running.

<?php

namespace App\Http\Controllers;

//...
use App\Imports\QueuedImport;
use App\Models\User;
use Illuminate\Http\Request;
use Maatwebsite\Excel\Facades\Excel;

class UserController extends Controller
{
    //...

    public function import_with_queue() {
        Excel::import(
            new QueuedImport,
            'data/data.csv'
        );

        return redirect('/')->with('success', 'Users Imported Successfully!');
    }
}
<?php

use Illuminate\Support\Facades\Route;
use App\Http\Controllers\UserController;
use Illuminate\Support\Facades\Artisan;

//...

Route::get(
    '/import-with-queue',
    [UserController::class, 'import_with_queue']
);

Let’s call the route and see what happens. Well, it worked. The import was a success, but it didn’t get sent to the queue and did it in the background. It just imported everything else so far. You can see that the user was locked on the screen until the import finished.

Why didn’t the queue import work?

If it didn’t work, it’s because of how you have Laravel configured. Open your .env file, and you’ll notice that the QUEUE_CONNECTION is set to sync. We need to prepare Laravel to be able to use queues. Let’s modify it so that we can use queues. Run the following commands:

php artisan queue:table

This generates the jobs migration, where jobs will be inserted.

php artisan migrate

Creates the jobs table. All left is to modify the QUEUE_CONNECTION to the database and clear the config. Open your .env file and change the QUEUE_CONNECTION from sync to the database.

QUEUE_CONNECTION=database

Finally clear the config cache.

php artisan config:clear

Let’s try our example again. Do you think it’s going to work?

Well, it was lightning fast this time. It looked like the job was created, but nothing happened. Where’s the import? One last step. We need to make sure that our queue is working.

php artisan queue:work

Cool! We even get an excellent chunk message display when each chunk is complete and how long it took the queue to import it.

2023-05-08 02:06:19 Maatwebsite\Excel\Jobs\QueueImport ....................... 526.52ms DONE
2023-05-08 02:06:19 Maatwebsite\Excel\Jobs\ReadChunk ....................... 654.62ms DONE
2023-05-08 02:06:20 Maatwebsite\Excel\Jobs\ReadChunk ....................... 256.7ms DONE
2023-05-08 02:07:00 Maatwebsite\Excel\Jobs\ReadChunk ....................... 630.79ms DONE
2023-05-08 02:07:01 Maatwebsite\Excel\Jobs\ReadChunk ....................... 759.34ms DONE
2023-05-08 02:07:02 Maatwebsite\Excel\Jobs\ReadChunk ....................... 123.56ms DONE
2023-05-08 02:07:03 Maatwebsite\Excel\Jobs\AfterImportJob ....................... 6.36ms DONE

Explicitly Stating Queue Import

The queue is implicitly called when calling Excel::import with the ShouldQueue concern implemented. If you want to be explicit, you could use Excel::queueImport(). Either way, the ShouldQueue concern is always required.

Modifying the Queued Import

If you try modifying the details inside the queued import class, you’ll notice that the changes are not reflected. Each time the import is limited, you must restart the queue.

Thank you for reading this article.

If you have any queries or doubts about this topic please feel free to contact us. We will try to reach you.

Comments (0)

Comment


Note: All Input Fields are required.