Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Verify tensorrt error on lung_nodule_ct_detection #488

Closed
yiheng-wang-nv opened this issue Aug 17, 2023 · 1 comment · Fixed by #489
Closed

Verify tensorrt error on lung_nodule_ct_detection #488

yiheng-wang-nv opened this issue Aug 17, 2023 · 1 comment · Fixed by #489

Comments

@yiheng-wang-nv
Copy link
Collaborator

environment:
projectmonai/monai:1.2.0 image
error message:

export bundle lung_nodule_ct_detection with precision fp32 failed.
Traceback (most recent call last):
  File "ci/verify_tensorrt.py", line 110, in <module>
    verify_all_onnx_tensorrt_bundles()
  File "ci/verify_tensorrt.py", line 100, in verify_all_onnx_tensorrt_bundles
    verify_tensorrt(export_context)
  File "ci/verify_tensorrt.py", line 32, in verify_tensorrt
    trt_export(**export_context)
  File "/opt/monai/monai/bundle/scripts.py", line 1404, in trt_export
    _export(
  File "/opt/monai/monai/bundle/scripts.py", line 997, in _export
    net = converter(model=net, **kwargs)
  File "/opt/monai/monai/networks/utils.py", line 941, in convert_to_trt
    ir_model = convert_to_torchscript(model, device=target_device, inputs=inputs, use_trace=use_trace)
  File "/opt/monai/monai/networks/utils.py", line 744, in convert_to_torchscript
    script_module = torch.jit.trace(model, example_inputs=inputs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 792, in trace
    return trace_module(
  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 1049, in trace_module
    module._c._create_method_from_trace(
RuntimeError: Encountering a dict at the output of the tracer might cause the trace to be incorrect, this is only valid if the container structure does not change based on the module's inputs. Consider using a constant container instead (e.g. for `list`, use a `tuple` instead. for `dict`, use a `NamedTuple` instead). If you absolutely need this and know the side effects, pass strict=False to trace() to allow this behavior.

Hi @binliunls , could you please help to take a look at it?
@binliunls binliunls mentioned this issue Aug 17, 2023
Merged
8 tasks
@binliunls
Copy link
Contributor

Hi @yiheng-wang-nv ,
I think this is caused by the trt_export API lacking some extra parameters which are specific to the detection bundle. I added them in this PR to try to fix it. Please help me run this test again and to see if this issue is fixed.

Thanks,
Bin

yiheng-wang-nv pushed a commit that referenced this issue Aug 17, 2023
@binliunls
488 fix detection ci test [skip-gpu] (#489)
99b8be7 
Fixes #488.

### Description
Add some extra parameters to the detection bundle to cover the ci test
issue.

### Status
**Ready/Work in progress/Hold**

### Please ensure all the checkboxes:
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Codeformat tests passed locally by running `./runtests.sh
--codeformat`.
- [ ] In-line docstrings updated.
- [ ] Update `version` and `changelog` in `metadata.json` if changing an
existing bundle.
- [ ] Please ensure the naming rules in config files meet our
requirements (please refer to: `CONTRIBUTING.md`).
- [ ] Ensure versions of packages such as `monai`, `pytorch` and `numpy`
are correct in `metadata.json`.
- [ ] Descriptions should be consistent with the content, such as
`eval_metrics` of the provided weights and TorchScript modules.
- [ ] Files larger than 25MB are excluded and replaced by providing
download links in `large_file.yml`.
- [ ] Avoid using path that contains personal information within config
files (such as use `/home/your_name/` for `"bundle_root"`).

---------

Signed-off-by: binliu <binliu@nvidia.com>
yiheng-wang-nv pushed a commit to yiheng-wang-nv/model-zoo that referenced this issue Jul 29, 2024
@binliunls
488 fix detection ci test [skip-gpu] (Project-MONAI#489)
7117a85 
Fixes Project-MONAI#488.

### Description
Add some extra parameters to the detection bundle to cover the ci test
issue.

### Status
**Ready/Work in progress/Hold**

### Please ensure all the checkboxes:
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Codeformat tests passed locally by running `./runtests.sh
--codeformat`.
- [ ] In-line docstrings updated.
- [ ] Update `version` and `changelog` in `metadata.json` if changing an
existing bundle.
- [ ] Please ensure the naming rules in config files meet our
requirements (please refer to: `CONTRIBUTING.md`).
- [ ] Ensure versions of packages such as `monai`, `pytorch` and `numpy`
are correct in `metadata.json`.
- [ ] Descriptions should be consistent with the content, such as
`eval_metrics` of the provided weights and TorchScript modules.
- [ ] Files larger than 25MB are excluded and replaced by providing
download links in `large_file.yml`.
- [ ] Avoid using path that contains personal information within config
files (such as use `/home/your_name/` for `"bundle_root"`).

---------

Signed-off-by: binliu <binliu@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

488 fix detection ci test [skip-gpu] binliunls/model-zoo
2 participants
@yiheng-wang-nv @binliunls