creating coco formatted result.json with sahi predict CLI command for coco evaluation

[isspid]

I thought a question rather than an issue was more fitting for this.

I am a little bit confused by the CLI commands. I have a model which I use to run inference on a directory of images. After running it I can get the new images which have the predictions visualized. Could you please explain how to get a .json file of these predictions so that I can run the coco_evaluate command afterwards?

Edit: Basically I want to know the procedure of getting the results.json file.

[fcakyon]

@isspid thanks for bringing this up. This has been answered before: https://github.com/obss/sahi/issues/273#issuecomment-971402591

But I will briefly explain again 🚀

1. You have to prepare a COCO formatted dataset (image folder and COCO formatted dataset.json).
2. sahi predict --source coco/image/folder --dataset_json_path coco_dataset.json + other arguments will export a results.json.
3. using dataset.json and result.json you run sahi coco evaluate --dataset_json_path dataset.json --result_json_path result.json.
4. eval.json containing coco evaluation results will be exported.
5. (bonus) you can generate coco error analysis plots by sahi coco analyse --dataset_json_path dataset.json --result_json_path result.json.

[isspid]

Fatih, you have been amazing once again. Many thanks.

[dimpolitik]

@isspid I have two questions. Which format do you use in CVAT to create the dataset.json and then which snippet code to produce the result.json file?
In my case I saved the annotations in COCO fromat and then I used this to produce the result.json.

sahi predict --slice_width 640 --slice_height 640 --overlap_height_ratio 0.2 --overlap_width_ratio 0.2 --model_confidence_threshold 0.5 --source /home/politikos/datasets/pets/images/test/ --dataset_json_path dataset_coco.json --model_path /home/politikos/pets_detection/runs/detect/pets_xl/weights/best.pt --model_type yolov8 --model_config_path /home/politikos/.local/lib/python3.11/site-packages/ultralytics/cfg/datasets/pets.yaml.

But the dataset_coco.json and the resulting result.json file don't have the same "content format" when I open them.
Thank you for your time!

[isspid]

Hi @dimpolitik. It has been quite some time but I will try to help as much as I can.

From CVAT you can export the COCO formatted json file by selecting the setting below:

In order to generate the results.json file please run sahi predict --source coco/image/folder --dataset_json_path coco_dataset.json + other arguments

If they don't have the same format, please upload them here, I will try to take a look at them and help in anyway I can.

[dimpolitik]

Hi @isspid!
Attached the two files.
The result json was produced with:

sahi predict --source /home/politikos/datasets/pets/images/test/ --dataset_json_path dataset.json --model_path /home/politikos/pets_detection/runs/detect/pets_x16/weights/best.pt --model_type yolov8 --model_config_path /home/politikos/.local/lib/python3.11/site-packages/ultralytics/cfg/datasets/pets.yaml
dataset.json
result.json

When I run: sahi coco evaluate --type bbox --classwise --iou_thrs 0.5 --dataset_json_path dataset.json --result_json_path result.json
I get zero values and -1.0 for APs.

Important notice. I trained the YOLO model with the yolo txt annotation files, not coco if this matters.

Thank you so much for your willingness to help. Really appreciated!

[isspid]

I believe you are experiencing the same problem as me, which is discussed in this thread. Please take a look at this answer:
#336 (reply in thread).

I will simply explain here what @fcakyon explained to me in the answer.

Basically "your category ids start from 1 while model is performing inference on starting with category 0."

You can update the category ids:

from sahi.utils.coco import Coco
from sahi.utils.file import save_json

# init Coco objects by specifying coco dataset paths and image folder directories
coco = Coco.from_coco_dict_or_path("dataset.json")

# select window categories; and map them to id 0
desired_name2id = {
  "window": 0,
}
coco.update_categories(desired_name2id)

# export updated/filtered COCO dataset
save_json(coco.json, "updated_dataset.json")

[dimpolitik]

I was aware of this code. The corrected json is the attached. I get the same -1 values.
correct_dataset.json

Did your corrected json have the same structure?

[isspid]

Your corrected json doesn't have any annotations as far as I can see.

I think you need to edit the code in order for it to work for you. Please try this and let me know how it goes.

from sahi.utils.coco import Coco
from sahi.utils.file import save_json

# init Coco objects by specifying coco dataset paths and image folder directories
coco = Coco.from_coco_dict_or_path("dataset.json")

# select window categories; and map them to id 0
desired_name2id = {
  "shark": 0,
  "skate": 1,
}
coco.update_categories(desired_name2id)

# export updated/filtered COCO dataset
save_json(coco.json, "updated_dataset.json")

[dimpolitik]

That's true!!! I thought that "window" : 0 was the default way to correct json. Now I get values other than -1. Great!! Thank you so much! I was struggling many days with that.

"bbox_mAP":0.242,
"bbox_mAP50":0.479,
"bbox_mAP75":0.209,
"bbox_mAP_s":-1.0,
"bbox_mAP_m":-1.0,
"bbox_mAP_l":0.242,
"bbox_mAP50_s":-1.0,
"bbox_mAP50_m":-1.0,
"bbox_mAP50_l":0.479,
"bbox_mAP_copypaste":"0.242 0.209 0.479 -1.000 -1.000 0.242 -1.000 -1.000 0.479",
"results_per_category":{
"bbox_shark_mAP":"0.174",
"bbox_shark_mAP_s":"-1.000",
"bbox_shark_mAP_m":"-1.000",
"bbox_shark_mAP_l":"0.174",
"bbox_shark_mAP50":"0.354",
"bbox_shark_mAP50_s":"-1.000",
"bbox_shark_mAP50_m":"-1.000",
"bbox_shark_mAP50_l":"0.354",
"bbox_skate_mAP":"0.310",
"bbox_skate_mAP_s":"-1.000",
"bbox_skate_mAP_m":"-1.000",
"bbox_skate_mAP_l":"0.310",
"bbox_skate_mAP50":"0.604",
"bbox_skate_mAP50_s":"-1.000",
"bbox_skate_mAP50_m":"-1.000",

[isspid]

Happy to read it worked. Good luck.