15692: Introduce background jobs #16927
Conversation
A new abstract class can be used to implement job function classes. It handles the necessary logic for starting and stopping jobs, including exception handling and rescheduling of recurring jobs. This commit also includes the migration of data source jobs to the new framework.
Using the 'import_string()' utility from Django allows the job script class to be simplified, as module imports no longer need to avoid loops. This should make it easier to queue and maintain jobs.
Instead of maintaining two separate job execution logics, the same job is now used for both background and interactive execution.
The independent implementations of interactive and background script execution have been merged into a single BackgroundJob implementation.
A new abstract class can be used to implement job function classes that specialize in scheduling. These use the same logic as regular BackgroundJobs, but ensure that they are only scheduled once at any given time.
A new abstract class can be used to implement job function classes that specialize in system background tasks (e.g. synchronization or housekeeping). In addition to the features of the BackgroundJob and ScheduledJob classes, these implement additional logic to not need to be bound to an existing NetBox object and to setup job schedules on plugin load instead of an interactive request.
There was a problem hiding this comment.
@alehaa thank you for such a thorough and well-documented PR! Apologies again for the delay in review. At a high level, this all looks doable. I just have a few questions about the implementation details.
This partially reverts commit db591d4. The 'run_now' parameter of 'enqueue()' remains, as its being used by following commits.
Because scripts are already linked through the Job Instance field, the name is displayed twice. Removing this reduces redundancy and opens up the possibility of simplifying the BackgroundJob framework in future commits.
Instead of using separate classes, the logic of ScheduledJob is now merged into the generic BackgroundJob class. This allows reusing the same logic, but dynamically deciding whether to enqueue the same job once or multiple times.
Instead of defining individual names on enqueue, BackgroundJob classes can now set a job name in their meta class. This is equivalent to other Django classes and NetBox scripts.
|
@jeremystretch I‘m a bit confused about your commits. You requested changes, I‘ve worked a day on them and just before uploading commits apear in this PR. 🧐 |
|
@alehaa apologies for the confusion; it had been several days with no response so I assumed you did not have time to make the changes. I've undertaken the work we discussed above as well as some general cleanup. We need to have all the v4.1 work merged by end of week. |
|
@alehaa do you want to handle renaming ScheduledJob from here, or should I just wrap it up? |
|
@jeremystretch no worries, but I’d like to keep it up from here. I see some of your code handles parts I‘ve not finished yet. Please give me till tomorrow to integrate your changes into mine and I‘ll upload the combination from both of our sources. |
ChoiceSet was moved to core in 40572b5.
System jobs usually perform low-priority background tasks and therefore can use a different queue than 'default', which is used for regular jobs related to specific objects.
As the job's name is set by enqueue(), it must not be passed in handle() to avoid duplicate kwargs with the same name.
Not only can a job's interval change, but so can the time at which it is scheduled to run. If a specific scheduled time is set, it will also be checked against the current job schedule. If there are any changes, the job is rescheduled with the new time.
Instead of using a class method for run(), a regular method is used for this purpose. This gives the possibility to add more convenience methods in the future, e.g. for interacting with the job object or for logging, as implemented for scripts.
alehaa
force-pushed
the
15692-background-jobs
branch
from
575aec1
to
c047bf4
Compare
|
@jeremystretch I've finished my work and added everything to this PR and its ready for review now. Please note that I did need to do a force push, as some of our works did conflict and I didn't want to open a third PR for this issue. Therefore you might need to to a Despite the open tasks and some bug fixing, I've
Also note the open comment on the |
This reverts commit b17b205.
This commit reverts commits 4880d81 and 0b15ecf. Using the database 'connection_created' signal for job registration feels a little wrong at this point, as it would trigger registration very often. However, the background job framework is prepared for this use case and can be used by plugins once the auto-registration of jobs is solved.
| # your logic goes here | ||
| ``` | ||
|
|
||
| You can schedule the background job from within your code (e.g. from a model's `save()` method or a view) by calling `MyTestJob.enqueue()`. This method passes through all arguments to `Job.enqueue()`. However, no `name` argument must be passed, as the background job name will be used instead. |
There was a problem hiding this comment.
might be good to give an example here of calling enqueue
| object_type = ObjectType.objects.get_for_model(instance, for_concrete_model=False) | ||
| rq_queue_name = get_queue_for_model(object_type.model) | ||
| if schedule_at and immediate: | ||
| raise ValueError("enqueue() cannot be called with values for both schedule_at and immediate.") |
There was a problem hiding this comment.
Should be internationalized with _(
|
BTW @alehaa IMHO excellent work on this, both the writeup in the issue and the PR itself is very high quality and impressive. Nice work! |
|
Seems like we're just about there, however I still have one concern: As it stands, I expect the average user will have difficulty distinguishing between the Job database model and the BackgroundJob runner class. I'd like to rename |
|
Thanks again for all your work on this @alehaa! |
Fixes: #15692
This PR includes three new background job class to be used in NetBox and by NetBox plugins. They handle the necessary logic for starting and stopping jobs, including exception handling and rescheduling of recurring jobs.
This PR also implements the use of this framework for existing data source and script jobs, eliminating duplicate code and ensuring consistency.