Suitability of APScheduler for HA, especially v4

[lee-jian-hui]

I am trying to examine the suitability of apscheduler if it checks these boxes regarding HA for this scenario:

• spinning up multiple instances of worker and schedulers
• schedule one task that will run at every 10 pm GMT+8 on all instances
• using redis as message broker
• i only expect the task to only run ONCE

QUESTIONS:

• could i understand the new event broker as the jobstore in previous versions with more capabilities?

Does ver4 provide HA (high availability) features out of the box and are there sample codes for this?
In my testing I have deployed multiple app instances and the runs were duplicated which is not desired.
In addition, maybe anyone could point out to me if my deployment steps do not fall in recommended guidelines?

Here is the sample code i have modified and used following the latest docs on master
Some lines are commented out because I were facing issues with those imports on apschedler v4.0.0a5

from datetime import datetime
from apscheduler import Scheduler
# from apscheduler import Scheduler, TaskDefaults, task
from apscheduler.triggers.cron import CronTrigger
from apscheduler.triggers.interval import IntervalTrigger
from apscheduler.eventbrokers.redis import RedisEventBroker
from dotenv import load_dotenv
# from redis import Redis
from redis.asyncio import Redis as RedisAsync
import os
from redis.exceptions import ConnectionError

load_dotenv(".env", verbose=True, override=True)

def setup_redis(ssl=False):
    redis_ = None
    if ssl == True:
        redis_ = RedisAsync(
            db=os.getenv("SCHEDULER_RQ_REDIS_DB"), 
            host=os.getenv("SCHEDULER_RQ_REDIS_HOST"), 
            port=os.getenv("SCHEDULER_RQ_REDIS_PORT"),
            password=os.getenv("SCHEDULER_RQ_REDIS_PASSWORD"),
            ssl=True,
            ssl_ca_certs=os.getenv("REQUESTS_CA_BUNDLE")
        )    
    else:   
        redis_ = RedisAsync(
            db=os.getenv("SCHEDULER_RQ_REDIS_DB"), 
            host=os.getenv("SCHEDULER_RQ_REDIS_HOST"), 
            port=os.getenv("SCHEDULER_RQ_REDIS_PORT"),
            password=os.getenv("SCHEDULER_RQ_REDIS_PASSWORD"),
        )
    return redis_


# @task(max_running_jobs=3, metadata={"foo": ["taskfunc"]})
def mytaskfunc():
    print("running stuff at {}".format(datetime.now()))

# task_defaults = TaskDefaults(
#     misfire_grace_time=15,
#     job_executor="processpool",
#     metadata={"global": 3, "foo": ["bar"]}
# )


def main():
    redis_ = setup_redis()
    event_broker = RedisEventBroker(redis_)

    with Scheduler(event_broker=event_broker) as scheduler:
        task = scheduler.configure_task(
            "sometask",
            func=mytaskfunc,
            job_executor="threadpool",
            # metadata={"direct": True}
        )
        trigger = IntervalTrigger(seconds=10)
        scheduler.add_schedule(func_or_task_id=task, trigger=trigger)

        scheduler.run_until_stopped()


if __name__ == "__main__":  
    main()

logs:

run on app1 with 3 instances

2024-10-21T14:22:08.234+08:00 [APP/PROC/WEB/1] [OUT] running stuff at 2024-10-21 06:22:08.234691
2024-10-21T14:22:08.355+08:00 [APP/PROC/WEB/2] [OUT] running stuff at 2024-10-21 06:22:08.355508
2024-10-21T14:22:08.459+08:00 [APP/PROC/WEB/0] [OUT] running stuff at 2024-10-21 06:22:08.459599
2024-10-21T14:22:18.232+08:00 [APP/PROC/WEB/1] [OUT] running stuff at 2024-10-21 06:22:18.232721
2024-10-21T14:22:18.353+08:00 [APP/PROC/WEB/2] [OUT] running stuff at 2024-10-21 06:22:18.353694
2024-10-21T14:22:18.458+08:00 [APP/PROC/WEB/0] [OUT] running stuff at 2024-10-21 06:22:18.458419
2024-10-21T14:22:28.231+08:00 [APP/PROC/WEB/1] [OUT] running stuff at 2024-10-21 06:22:28.231574
2024-10-21T14:22:28.353+08:00 [APP/PROC/WEB/2] [OUT] running stuff at 2024-10-21 06:22:28.353322
2024-10-21T14:22:28.459+08:00 [APP/PROC/WEB/0] [OUT] running stuff at 2024-10-21 06:22:28.459454
2024-10-21T14:22:38.231+08:00 [APP/PROC/WEB/1] [OUT] running stuff at 2024-10-21 06:22:38.231379
2024-10-21T14:22:38.353+08:00 [APP/PROC/WEB/2] [OUT] running stuff at 2024-10-21 06:22:38.353043
2024-10-21T14:22:38.459+08:00 [APP/PROC/WEB/0] [OUT] running stuff at 2024-10-21 06:22:38.458904
2024-10-21T14:22:48.232+08:00 [APP/PROC/WEB/1] [OUT] running stuff at 2024-10-21 06:22:48.232044
2024-10-21T14:22:48.353+08:00 [APP/PROC/WEB/2] [OUT] running stuff at 2024-10-21 06:22:48.353308
2024-10-21T14:22:48.458+08:00 [APP/PROC/WEB/0] [OUT] running stuff at 2024-10-21 06:22:48.458533
2024-10-21T14:22:58.232+08:00 [APP/PROC/WEB/1] [OUT] running stuff at 2024-10-21 06:22:58.231848
2024-10-21T14:22:58.352+08:00 [APP/PROC/WEB/2] [OUT] running stuff at 2024-10-21 06:22:58.352355
2024-10-21T14:22:58.458+08:00 [APP/PROC/WEB/0] [OUT] running stuff at 2024-10-21 06:22:58.457938

run on app2 with 3 instances

2024-10-21T14:22:58.472+08:00 [APP/PROC/WEB/2] [OUT] running stuff at 2024-10-21 06:22:58.472781
2024-10-21T14:22:58.571+08:00 [APP/PROC/WEB/1] [OUT] running stuff at 2024-10-21 06:22:58.571584
2024-10-21T14:22:58.730+08:00 [APP/PROC/WEB/0] [OUT] running stuff at 2024-10-21 06:22:58.730290
2024-10-21T14:23:08.471+08:00 [APP/PROC/WEB/2] [OUT] running stuff at 2024-10-21 06:23:08.471524
2024-10-21T14:23:08.572+08:00 [APP/PROC/WEB/1] [OUT] running stuff at 2024-10-21 06:23:08.571997
2024-10-21T14:23:08.730+08:00 [APP/PROC/WEB/0] [OUT] running stuff at 2024-10-21 06:23:08.730043
2024-10-21T14:23:18.470+08:00 [APP/PROC/WEB/2] [OUT] running stuff at 2024-10-21 06:23:18.470679
2024-10-21T14:23:18.573+08:00 [APP/PROC/WEB/1] [OUT] running stuff at 2024-10-21 06:23:18.572163
2024-10-21T14:23:18.729+08:00 [APP/PROC/WEB/0] [OUT] running stuff at 2024-10-21 06:23:18.729756
2024-10-21T14:23:28.471+08:00 [APP/PROC/WEB/2] [OUT] running stuff at 2024-10-21 06:23:28.471203
2024-10-21T14:23:28.571+08:00 [APP/PROC/WEB/1] [OUT] running stuff at 2024-10-21 06:23:28.571680
2024-10-21T14:23:28.730+08:00 [APP/PROC/WEB/0] [OUT] running stuff at 2024-10-21 06:23:28.730049

[legout]

I think you also need a data store beside the event broker.

Might be related to #959

[lee-jian-hui]

what is the data store's function exactly? to my knowledge i have thought it as a persistent data store for scheduler results etc.

[agronholm]

Yes, task data, schedules and pending jobs are stored there in addition to job results.

[agronholm]

The event broker is only used for notifications across schedulers, so when a new schedule or job is added, all the schedulers will be notified so they can be woken up from their sleep.