Queue just stops working after x jobs

[Ic3m4n34]

I have a expressJS server where I use queues to make API calls and store the data I receive in a database. I usually add 10000 - 15000 jobs. All works great until it doesn't. After 1000 - 3000 jobs the queue just stop working.

I don't get an error or anything.

This is what the Dashboard currently looks like. Does anyone have an idea why this is happening or how I can debug this?

[manast]

So it looks like you have a lot of active jobs that do not complete. This is normally a bug in the processor code where the jobs are just being processed for-ever, this can be for multiple reasons, like not returning a promise that resolves, etc.
Also make sure you have at least one QueueScheduler running to clean for stalled jobs.

[Ic3m4n34]

Thanks or your answer @manast . This was my best guess also. Maybe you can take a look at my code? This is what my job looks like:

module.exports = async (job) => {
  const { data: url } = job;
  // make api call
  try {
    const { data } = await axios.get(url);

    // do stuff with data
    const dataToUpsert = data;

    console.info('before addTokensToDB');

    // inserts data in supabase (async)
    return addToDB(dataToUpsert);
  } catch (error) {
    console.error('===================');
    console.error('error adding to DB', error);
    console.error('===================');
    return Promise.reject(new Error('error adding to DB'));
  }
};

Something else is noteworthy here: I'm calling two API's. One where I don't get much errors and one which fails occasionally. If it fails, my jobs move to the Delayed state and get processed again, which is fine.

But this is the one that gets stuck over time. Is there a thing as "too much errors"? 😅

As for the QueueScheduler, thats the only thing I'm doing with it:

const queueScheduler = new QueueScheduler('Tokens');

Do I need to do something else?
Your help is really appreciated :-)

[manast]

First of all you do not need to wrap the code in a try/catch, thats what the job processor does for you and fail the job accordingly. Secondly, this call return addToDB(dataToUpsert); we do not know what it does, it probably hangs the promise in some situations.

[Ic3m4n34]

Here is addToDB

const addToDB = async (arrayOfTokens) => {
  const { error } = await supabase
    .from('tokens')
    .upsert(arrayOfTokens, { onConflict: 'id' });

  if (error) {
    console.error('Error in addToDB', error);
    return error;
  }
  console.info(`${arrayOfTokens[0].slug}: #${arrayOfTokens[0].id} added to db`);
  return arrayOfTokens[0].id;
};

But I never saw that error. If something fails, it's always the axios.get call.

[manast]

You could put more console.logs after every await call to see on which one it gets stucked.