You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are in a bit of a bind, we have 2 identical Composer environments on GCP (composer-2.9.4-airflow-2.9.3), one of them suddenly stopped working and the scheduler is now in a crash loop. No amount of DAG run or Task instance deletion helped. Both environments run the same DAGS with the same scheduling. Here is the stack trace:
2024-09-27 09:20:33.228 CEST airflow-scheduler Traceback (most recent call last): File "/opt/python3.11/bin/airflow", line 8, in <module> sys.exit(main()) ^^^^^^
2024-09-27 09:20:33.229 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/__main__.py", line 58, in main
2024-09-27 09:20:33.230 CEST airflow-scheduler args.func(args)
2024-09-27 09:20:33.230 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/cli/cli_config.py", line 49, in command
2024-09-27 09:20:33.231 CEST airflow-scheduler return func(*args, **kwargs)
2024-09-27 09:20:33.231 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.231 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/cli.py", line 114, in wrapper
2024-09-27 09:20:33.232 CEST airflow-scheduler return f(*args, **kwargs)
2024-09-27 09:20:33.233 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.233 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
2024-09-27 09:20:33.233 CEST airflow-scheduler return func(*args, **kwargs)
2024-09-27 09:20:33.235 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.235 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 58, in scheduler
2024-09-27 09:20:33.237 CEST airflow-scheduler run_command_with_daemon_option(
2024-09-27 09:20:33.237 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/cli/commands/daemon_utils.py", line 85, in run_command_with_daemon_option
2024-09-27 09:20:33.239 CEST airflow-scheduler callback()
2024-09-27 09:20:33.239 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 61, in <lambda>
2024-09-27 09:20:33.240 CEST airflow-scheduler callback=lambda: _run_scheduler_job(args),
2024-09-27 09:20:33.246 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.246 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py", line 49, in _run_scheduler_job
2024-09-27 09:20:33.247 CEST airflow-scheduler run_job(job=job_runner.job, execute_callable=job_runner._execute)
2024-09-27 09:20:33.247 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/session.py", line 79, in wrapper
2024-09-27 09:20:33.249 CEST airflow-scheduler return func(*args, session=session, **kwargs)
2024-09-27 09:20:33.250 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.250 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/job.py", line 402, in run_job
2024-09-27 09:20:33.252 CEST airflow-scheduler return execute_job(job, execute_callable=execute_callable)
2024-09-27 09:20:33.253 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.253 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/job.py", line 431, in execute_job
2024-09-27 09:20:33.254 CEST airflow-scheduler ret = execute_callable()
2024-09-27 09:20:33.255 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.255 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 846, in _execute
2024-09-27 09:20:33.325 CEST airflow-scheduler self._run_scheduler_loop()
2024-09-27 09:20:33.326 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 978, in _run_scheduler_loop
2024-09-27 09:20:33.328 CEST airflow-scheduler num_queued_tis = self._do_scheduling(session)
2024-09-27 09:20:33.330 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.330 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1052, in _do_scheduling
2024-09-27 09:20:33.335 CEST airflow-scheduler self._create_dagruns_for_dags(guard, session)
2024-09-27 09:20:33.335 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/retries.py", line 89, in wrapped_function
2024-09-27 09:20:33.336 CEST airflow-scheduler for attempt in run_with_db_retries(max_retries=retries, logger=logger, **retry_kwargs):
2024-09-27 09:20:33.336 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 443, in __iter__
2024-09-27 09:20:33.338 CEST airflow-scheduler do = self.iter(retry_state=retry_state)
2024-09-27 09:20:33.342 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.342 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 376, in iter
2024-09-27 09:20:33.343 CEST airflow-scheduler result = action(retry_state)
2024-09-27 09:20:33.345 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.345 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda>
2024-09-27 09:20:33.346 CEST airflow-scheduler self._add_action_func(lambda rs: rs.outcome.result())
2024-09-27 09:20:33.347 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.347 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/concurrent/futures/_base.py", line 449, in result
2024-09-27 09:20:33.348 CEST airflow-scheduler return self.__get_result()
2024-09-27 09:20:33.348 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.348 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
2024-09-27 09:20:33.349 CEST airflow-scheduler raise self._exception
2024-09-27 09:20:33.349 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/retries.py", line 98, in wrapped_function
2024-09-27 09:20:33.350 CEST airflow-scheduler return func(*args, **kwargs)
2024-09-27 09:20:33.355 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.355 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1126, in _create_dagruns_for_dags
2024-09-27 09:20:33.356 CEST airflow-scheduler self._create_dag_runs_dataset_triggered(
2024-09-27 09:20:33.356 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py", line 1279, in _create_dag_runs_dataset_triggered
2024-09-27 09:20:33.424 CEST airflow-scheduler data_interval = dag.timetable.data_interval_for_events(exec_date, dataset_events)
2024-09-27 09:20:33.426 CEST airflow-scheduler ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.426 CEST airflow-scheduler File "/opt/python3.11/lib/python3.11/site-packages/airflow/timetables/simple.py", line 194, in data_interval_for_events
2024-09-27 09:20:33.427 CEST airflow-scheduler start = min(start_dates)
2024-09-27 09:20:33.427 CEST airflow-scheduler ^^^^^^^^^^^^^^^^
2024-09-27 09:20:33.427 CEST airflow-scheduler TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'
DAG start dates are mostly days_ago(1) with the occasional hardcoded date.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hi all,
We are in a bit of a bind, we have 2 identical Composer environments on GCP (composer-2.9.4-airflow-2.9.3), one of them suddenly stopped working and the scheduler is now in a crash loop. No amount of DAG run or Task instance deletion helped. Both environments run the same DAGS with the same scheduling. Here is the stack trace:
DAG start dates are mostly
days_ago(1)
with the occasional hardcoded date.Any idea?
Beta Was this translation helpful? Give feedback.
All reactions