data generation for crop segmentation workflow failed

[mursaleen-bkk]

I run the detagen_crop_segmentation to download the data of the same given area by farmvibes from 2020/1/1 to 2020/1/5. To download data for only five days. But the workflow failed as show in the given figure.

[rafaspadilha]

Hi, @mursaleen-bkk.

To help us understand the isse, whenever you have a failed workflow, please make sure to share the outputs of wf_run.reason and wf_run.task_details.

Considering what you described, SpaceEye requires a window of time of at least 3 months in order to remove the clouds.

If you are interested in the Sentinel-2 images, you can adapt the notebooks/crop_segmentation/wf_viz_crop_seg.yaml custom workflow to ignore the SpaceEye task and get the S-2 images as sinks. An example would be:

name: visualize_crop_segmentation
sources:
  user_input:
    - s2.user_input
    - cdl.user_input
sinks:
  s2_rasters: s2.raster ### NEW SINK FOR VISUALIZATION
  s2_masks: s2.mask ### NEW SINK FOR VISUALIZATION
  cdl: cdl.raster
parameters:
  pc_key:
tasks:
  s2: ### NEW TASK FOR VISUALIZATION
    workflow: data_ingestion/sentinel2/preprocess_s2_improved_masks
  cdl:
    workflow: data_ingestion/cdl/download_cdl
edges:

Please, let me know if that solves your issue

[mursaleen-bkk]

Hello @rafaspadilha!
Here is the output of task_details

{'spaceeye.preprocess.s2.list': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 11, 58, 249347), submission_time=datetime.datetime(2023, 3, 20, 11, 11, 58, 182679), end_time=datetime.datetime(2023, 3, 20, 11, 12, 3, 106783), reason=None, status='done'),
 'cdl.list_cdl': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 12, 3, 123658), submission_time=datetime.datetime(2023, 3, 20, 11, 11, 58, 196230), end_time=datetime.datetime(2023, 3, 20, 11, 12, 4, 773467), reason=None, status='done'),
 'spaceeye.preprocess.s2.filter': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 12, 4, 832297), submission_time=datetime.datetime(2023, 3, 20, 11, 12, 4, 788383), end_time=datetime.datetime(2023, 3, 20, 11, 12, 4, 871035), reason=None, status='done'),
 'cdl.download_cdl': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 12, 4, 897465), submission_time=datetime.datetime(2023, 3, 20, 11, 12, 4, 801232), end_time=datetime.datetime(2023, 3, 20, 11, 15, 10, 961043), reason=None, status='done'),
 'spaceeye.preprocess.s2.download': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 15, 11, 6596), submission_time=datetime.datetime(2023, 3, 20, 11, 15, 10, 977077), end_time=datetime.datetime(2023, 3, 20, 11, 18, 30, 657279), reason=None, status='done'),
 'spaceeye.preprocess.s2.preprocess': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 18, 30, 699080), submission_time=datetime.datetime(2023, 3, 20, 11, 18, 30, 672352), end_time=datetime.datetime(2023, 3, 20, 11, 21, 5, 822302), reason=None, status='done'),
 'spaceeye.preprocess.s2.group': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 21, 6, 441645), submission_time=datetime.datetime(2023, 3, 20, 11, 21, 5, 840119), end_time=datetime.datetime(2023, 3, 20, 11, 21, 6, 480196), reason=None, status='done'),
 'spaceeye.preprocess.s2.merge': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 21, 6, 539512), submission_time=datetime.datetime(2023, 3, 20, 11, 21, 6, 505767), end_time=datetime.datetime(2023, 3, 20, 11, 21, 6, 798772), reason=None, status='done'),
 'spaceeye.preprocess.cloud.cloud': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 21, 6, 864816), submission_time=datetime.datetime(2023, 3, 20, 11, 21, 6, 814423), end_time=datetime.datetime(2023, 3, 20, 11, 25, 30, 145595), reason=None, status='done'),
 'spaceeye.preprocess.cloud.shadow': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 25, 30, 163618), submission_time=datetime.datetime(2023, 3, 20, 11, 21, 6, 827879), end_time=datetime.datetime(2023, 3, 20, 11, 29, 50, 228081), reason=None, status='done'),
 'spaceeye.spaceeye.group_s2': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 29, 50, 256172), submission_time=datetime.datetime(2023, 3, 20, 11, 21, 6, 844685), end_time=datetime.datetime(2023, 3, 20, 11, 29, 58, 937085), reason=None, status='done'),
 'spaceeye.preprocess.cloud.merge': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 29, 58, 985963), submission_time=datetime.datetime(2023, 3, 20, 11, 29, 58, 954153), end_time=datetime.datetime(2023, 3, 20, 11, 30, 6, 947855), reason=None, status='done'),
 'spaceeye.spaceeye.group_mask': RunDetails(start_time=datetime.datetime(2023, 3, 20, 11, 30, 6, 994585), submission_time=datetime.datetime(2023, 3, 20, 11, 30, 6, 965724), end_time=datetime.datetime(2023, 3, 20, 11, 30, 15, 481324), reason=None, status='done'),
 'spaceeye.spaceeye.spaceeye': RunDetails(start_time=None, submission_time=datetime.datetime(2023, 3, 20, 11, 30, 15, 501066), end_time=None, reason=None, status='queued')}

And also I want to download the crop data layer (CDL Data) download

[rafaspadilha]

Hi, @mursaleen-bkk. The workflow that I've shared in my previous comment allows you to download CDL as well.

For SpaceEye, you will need to run for a longer time period (at least 3 months).

Could you share the output of wf_run.reason for that workflow that failed?

[mursaleen-bkk]

Hello, @rafaspadilha
Here is the output of wf_run.reason is

[rafaspadilha]

It may seem that message to execute the spaceeye workflow was lost between worker and orchestrator, and spaceeye never started. We are working on a few fixes that should reduce this behavior. I recommend resubmitting the workflow when that happens.

However, in this particular case, note that SpaceEye will not work with 5 days, as it requires a larger window of time (at least 3 months) to reconstruct the pixels occluded by clouds.

[rafaspadilha]

@mursaleen-bkk I've release a new version of FarmVibes, with fixes that may solve the issue that you were seeing of tasks being in the queued state indefinitely. May I ask you to update you cluster and retry submitting your jobs?

I'll close this discussion for now. Feel free to raise an issue if your problem persists.