URGENT!!! TEDS score different from the one reported in the paper.

[dimitri009]

I already asked this but I didnt received any response.

As a report for a work I wanted to test the TEDS of various models but the TEDS score of SLANet on PubTabNet is different from the one reported in the documentations.

I downloaded SLANet from here and tested with PubTabNet but I ended with a TEDS of 0.468942475598164

code:
python3 ppstructure/table/eval_table.py --det_model_dir=output2/en_ppocr_mobile_v2.0_table_det_infer --rec_model_dir=output2/en_ppocr_mobile_v2.0_table_rec_infer --table_model_dir=output2/en_ppstructure_mobile_v2.0_SLANet_infer --image_dir=/data/Datasets/pubtabnet/pubtabnet/val_/ --rec_char_dict_path=ppocr/utils/dict/table_dict.txt --table_char_dict_path=ppocr/utils/dict/table_structure_dict.txt --det_limit_side_len=736 --det_limit_type=min --rec_image_shape=3,32,320 --gt_path=output2/gt_.txt

I would like to know if someone has tested the TEDS score and how he did it please it is urgent.

[GreatV]

It appears that your observed TEDS score of 0.468942475598164 for SLANet on the PubTabNet dataset is significantly lower than the reported official TEDS score of 95.89%. Below are potential causes and solutions for this discrepancy:

---

1. Mismatch in Input Data or GT Labels

• Problem: The evaluation process requires ground truth (gt_path) files in a specific format. Ensure that your gt_path is correctly formatted as a tab-separated file with the image name and its corresponding HTML string.
• Solution: Use the provided script to convert your ground truth annotations into the required format:

  python3 ppstructure/table/convert_label2html.py --ori_gt_path /path/to/your_label_file --save_path /path/to/save_file

  Confirm that the generated gt_path contains valid and correctly formatted HTML strings for each image.

---

2. Model Configuration Issues

• Problem: Using incorrect or mismatched model files (e.g., detection, recognition, or SLANet models) can lead to suboptimal performance.

• Solution: Ensure you are using the correct pre-trained models for English table recognition. Use the following links to download the appropriate models:

  • Detection model: en_ppocr_mobile_v2.0_table_det_infer
  • Recognition model: en_ppocr_mobile_v2.0_table_rec_infer
  • SLANet model: en_ppstructure_mobile_v2.0_SLANet_infer

  Ensure these models are properly unzipped and referenced in your evaluation command.

---

3. Command Line Parameters

• Problem: Incorrect parameter settings for evaluation can lead to inaccurate results.
• Solution: Verify that your command matches the official example:

  python3 table/eval_table.py \
      --det_model_dir=inference/en_ppocr_mobile_v2.0_table_det_infer \
      --rec_model_dir=inference/en_ppocr_mobile_v2.0_table_rec_infer \
      --table_model_dir=inference/en_ppstructure_mobile_v2.0_SLANet_infer \
      --image_dir=/data/Datasets/pubtabnet/pubtabnet/val_/ \
      --rec_char_dict_path=ppocr/utils/dict/table_dict.txt \
      --table_char_dict_path=ppocr/utils/dict/table_structure_dict.txt \
      --gt_path=output2/gt_.txt \
      --det_limit_side_len=736 \
      --det_limit_type=min \
      --rec_image_shape=3,32,320

  Pay special attention to:
  • --det_limit_side_len: Ensure the image resizing is consistent with the documentation.
  • --rec_image_shape: Ensure it matches the shape used during model training.

---

4. Ground Truth Dataset

• Problem: Differences in the dataset used for evaluation can lead to variations in TEDS scores.
• Solution: Confirm that you're using the PubTabNet validation set as specified in the documentation. The dataset should match the one used to report the official results.

---

5. Environment Differences

• Problem: Model performance can vary based on the hardware/software environment.
• Solution: Ensure your environment is consistent with the official setup:
  • Python version: >= 3.7
  • PaddleOCR version: Latest
  • Ensure dependencies are correctly installed.

---

6. Debugging Tips

To debug further:

1. Intermediate Outputs: Check the intermediate outputs of each model (detection, recognition, and SLANet) to ensure they align with expectations.
2. Simplify Test Case: Run the evaluation on a small subset of images and manually verify the output against the ground truth.
3. Log File: Enable logging in your script to identify potential issues in the pipeline.

---

Final Recommendation

If the above steps do not resolve the issue, consider:

• Comparing the output HTML strings to the ground truth to identify where the discrepancies occur.
• Opening a GitHub issue in the PaddleOCR repository with detailed logs and your experiment setup for further assistance.

---

Let me know if you need help with any of the steps!

---

Response generated by feifei-bot | chatgpt-4o-latest