requestHandlerTimeoutSecs is not working properly

[teammakdi]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/cheerio (CheerioCrawler)

Issue description

It appears that the requestHandler continues processing the same crawl request beyond the designated requestHandler timeout. Although I set requestHandlerTimeoutSecs to 30 seconds, the requestHandler persists in processing the request even after the timeout, as evident in the logs.

INFO  CheerioCrawler: Initializing the crawler.
Processing https://crawlee.dev...
ERROR CheerioCrawler: Request failed and reached maximum retries. requestHandler timed out after 30 seconds. {"id":"3FULW9cbbkMrV4R","url":"https://crawlee.dev","method":"GET","uniqueKey":"https://crawlee.dev"}
Request https://crawlee.dev failed.
finished
{
  accessedAt: 2023-09-22T13:53:27.222Z,
  createdAt: 2023-09-22T13:52:35.867Z,
  hadMultipleClients: false,
  handledRequestCount: 1,
  id: '68f61bd1-a340-4484-b85a-2478e7a12c2a',
  modifiedAt: 2023-09-22T13:53:07.225Z,
  name: 'default',
  pendingRequestCount: 0,
  stats: {},
  totalRequestCount: 1,
  userId: '1'
}
Processing Request
INFO  Statistics: CheerioCrawler request statistics: {"requestAvgFailedDurationMillis":31277,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":0,"requestTotalDurationMillis":31277,"requestsTotal":1,"crawlerRuntimeMillis":60016,"retryHistogram":[1]}
INFO  CheerioCrawler:AutoscaledPool: state {"currentConcurrency":0,"desiredConcurrency":10,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":0},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.7,"actualRatio":0},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":0},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":0}}}
Request Finished

Code sample

const { CheerioCrawler, Configuration, RequestQueue } = require('crawlee')

async function main () {
  const config = Configuration.getGlobalConfig()
  config.set('persistStorage', false)
  const requestQueue = await RequestQueue.open()

  const crawler = new CheerioCrawler({
    requestQueue,
    minConcurrency: 10,
    maxConcurrency: 50,

    maxRequestRetries: 0,

    requestHandlerTimeoutSecs: 30,

    maxRequestsPerCrawl: 10,
    keepAlive: true,

    async requestHandler ({ request, $ }) {
      console.log(`Processing ${request.url}...`)
      await new Promise(resolve => setTimeout(resolve, 50000))
      console.log('finished')
      response = await requestQueue.getInfo()
      // pendingRequestCount: 0
      // totalRequestCount: 1
      // No Request in request queue but request Handler still working
      console.log(response)
      console.log('Processing Request')
      await new Promise(resolve => setTimeout(resolve, 60000))
      console.log('Request Finished')
    },

    failedRequestHandler ({ request }) {
      console.log(`Request ${request.url} failed.`)
    }
  })

  // Run the crawler and wait for it to finish.
  crawler.run()

  requestQueue.addRequest({ url: 'https://crawlee.dev' })

}

main()

Package version

3.5.4

Node.js version

18.17.0

Operating system

No response

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

The code is just a small POC.

[B4nan]

Technically, you can't cancel a running promise, this is how JS works, once it starts, it will finish eventually, the limit is only about not waiting for it (and canceling everything that happens afterward, as that is what we can do). That timeout is about failing the requestHandler, not canceling what's inside it.