Improve Exception Handling in Pre and Post Run Methods SCMSUITE-10131 SO107

[dormrod]

Description

The behaviour of exceptions raised in prerun and postrun methods of jobs was inconsistent and had issues, for example:

• SingleJob with a serial runner would raise the exception in the main thread and so kill the script entirely (not just fail a job)
• With a parallel runner, any exceptions would only get raised on the spawned thread and not be surfaced on the main thread, so there is a discrepancy in behaviour
• A multi-job which had a job which raised an exception in a child job would deadlock. This is because the parent job would never be notified that the job had finished

This PR makes this behaviour consistent, by catching any exception raised in these job steps, marking the job as complete, and storing the error message.

Changes

To achieve this, the following changes were made:

• The method get_errormsg which was already implemented on the AMSJob has been added to the Job base class with some basic implementation
• Add the decorator _fail_on_exception which handles exceptions for functions, marks the job and complete and notifies the parent multi-job
• Apply this decorator to _prepare, _execute and _finalize
• Store the exception trace in _error_msg which can be used by get_errormsg

The following changes were also made:

• Fix some mypy issues that were flagged on the latest version
• Convert the Job to an actual abstract base class instead of just raising exceptions when the methods were called

[robertrueger]

I did not run tests yet, but code looks good to me. We should think a bit about which exceptions in job execution should be caught, and which ones should not. It seems that we are very liberally catching a lot now ...