Excel Import with chunks become slower and slower

[bertog]

Hi there,

I'm currently import on a production App around 700K records via Excel Import. I use Queues and chuncks. The import is becoming slower and slower every 10k I import. From the initial 24s we are now around 150s at 120K records. The import code is quite simple.

`
class ProductsImport implements ShouldQueue, ToModel, WithChunkReading, WithHeadingRow
{
public function chunkSize(): int
{
return config('bee-too.excel_import_chunk_size');
}

public function model(array $row)
{
    $payload = [
        'product_brand_id' => $row['product_brand_id'],
        'product_category_id' => $row['product_category_id'],
        'product_subcategory_id' => array_key_exists('product_subcategory_id', $row) ? $row['product_subcategory_id'] : null,
        'measurement_unit_id' => $row['measurement_unit_id'],
        'vat_id' => $row['vat_id'],
        'sku' => $row['sku'],
        'supplier_sku' => array_key_exists('supplier_sku', $row) ? $row['supplier_sku'] : null,
        'erp_product_code' => array_key_exists('erp_product_code', $row) ? $row['erp_product_code'] : null,
        'ean13' => array_key_exists('ean13', $row) ? $row['ean13'] : null,
        'isbn' => array_key_exists('isbn', $row) ? $row['isbn'] : null,
        'upc' => array_key_exists('upc', $row) ? $row['upc'] : null,
        'description' => $row['description'],
        'extended_description' => $row['extended_description'],
        'low_stock_threshold' => array_key_exists('available_for_orders', $row) ? $row['low_stock_threshold'] : 1,
        'available_for_orders' => array_key_exists('available_for_orders', $row) ? $row['available_for_orders'] : true,
        'available_for_preorders' => array_key_exists('available_for_preorders', $row) ? $row['available_for_preorders'] : false,
        'available_on_request' => array_key_exists('available_on_request', $row) ? $row['available_on_request'] : false,
        'min_order_qty_limit' => array_key_exists('min_order_qty_limit', $row) ? $row['min_order_qty_limit'] : 1,
        'max_order_qty_limit' => array_key_exists('max_order_qty_limit', $row) ? $row['max_order_qty_limit'] : null,
        'shipping_price' => array_key_exists('shipping_price', $row) ? $row['shipping_price'] : null,
        'price' => $row['price'],
        'width' => array_key_exists('width', $row) ? $row['width'] : null,
        'height' => array_key_exists('height', $row) ? $row['height'] : null,
        'weight' => array_key_exists('weight', $row) ? $row['weight'] : null,
        'depth' => array_key_exists('depth', $row) ? $row['depth'] : null,
        'available_from' => array_key_exists('available_from', $row) ? $row['available_from'] : Carbon::now()->toDateString(),
    ];

    return new Product($payload);

`

On the Product model creation raise an event that populate a couple of other tables, but without any searching on the database they are simple insert. Every insert update also the index on Meilisearch.

I'm trying to troubleshoot the issue but I'm run out of ideas. Why it become more and more slower? Any suggestion?

Guido Bertolino