AI ideas?

[Mazhivee]

Automated Image Enhancement: Implement AI-based algorithms for automatic image enhancement, including denoising, color correction, and contrast adjustment. This would improve the quality of input images, leading to better 3D reconstructions. There are pre-trained AI models and libraries available that can be integrated for this purpose.

Feature Detection and Matching: Enhance the feature detection and matching process using AI-based methods. These methods can help improve the accuracy and speed of this crucial step in photogrammetry. Libraries like OpenCV offer AI-powered feature detection algorithms.

Depth Estimation: Implement AI-based depth estimation algorithms to generate initial depth maps from 2D images. Depth maps can serve as a valuable starting point for the photogrammetry process, and AI-driven depth estimation can be faster and more accurate than traditional methods.

Quality Control: Develop AI modules to automatically identify common issues in the reconstruction process, such as outliers, artifacts, or incomplete reconstructions. The AI could then provide feedback or recommendations to the user.

Real-time Guidance: Implement AI-powered real-time guidance during the image capture process. For example, the software could suggest optimal camera positions or angles to ensure a successful reconstruction.

Texture Mapping Assistance: Utilize AI to automate the texture mapping process and address common issues like texture seams. This would make the final 3D models look more polished.

User-Friendly Tutorials: Develop AI-driven tutorials or tooltips that guide users through the photogrammetry process step by step. These can provide immediate assistance to users, especially beginners.

Hardware Profiling: Use AI to profile the user's hardware and optimize the processing parameters accordingly. This can ensure that Meshroom runs efficiently on various systems.

these are some ideas we think would be very nice...

[natowi]

1 Automated Image Enhancement: yes, already possible. Just write a node for suitable model. I did some tests some time ago. It does not help with the input images, quite the opposite. It worsens the result as AI "invents" data that is not there, which leads to inaccuracy between shots. Photogrammetry relies on the accuracy of each feature. Image enhancement could be useful on the texturing side.
Delighting would be interesting.

2 Feature Detection and Matching: already wip see: https://github.com/mirkosprojects/MeshroomNodes #2130 https://github.com/PIX3LFLUX/MeshroomDFM also some recent work being done by other users as mentioned here #2179 (comment) (wip)

3 Depth Estimation: already some internal tests, here is a public project by another user: https://github.com/joreeves/mr2nerf some (older discussion) here: #528

4 Quality Control: interesting but difficult. Have not seen any project in that direction yet. Maybe artefact detection, but nothing specific

5 Real-time Guidance: nothing I would fix with AI. real time rough reconstruction would do the trick, too. Something like a capturing app going in this direction https://www.reddit.com/r/photogrammetry/comments/mginkw/boofcv_runs_on_android_if_you_want_to_play_with/

The other ideas require data and documentation on the Meshroom side so a model could be trained on that.

Other Ideas:

• Models for image masking: wip
• Image and point cloud segmentation
• 3d completion
• ...

There is no lack of ideas, there is a lack of people (having the time) to implement it ;)
Also some design choices need to be considered, as deep learning models can be quite large. Having a 25GB Meshroom executable does not sound so appealing :D Probably something like a plugin manager needs to be implemented first.
Additionally, many of the really good models do not allow commercial use.

Easy to implement Meshroom nodes for:

• https://github.com/nihui/realsr-ncnn-vulkan
• https://github.com/nihui/waifu2x-ncnn-vulkan
• https://github.com/nihui/rife-ncnn-vulkan
• https://github.com/nihui/realcugan-ncnn-vulkan

[Mazhivee]

thank you for the clarifications...
Lack of people with projects of this greatness is i think very difficult to get they have to or realy experts in both meshroom and programming which is not easy to find, if so , they need to be willing to put there time and effort into the project, really i can imagine that.

I am bit of new but know this project for at least 5 maybe 6 years now and lately very busy and not actively following the changes but yet i wanted to give "some" ideas that could inspire or swing a conversation on a some sort of ai. form that maybe could be implemented.. so this reply of you makes me think about refined solutions that maybe i can provide , but yeah i have some things to think about for the next maybe good idea to express.. i am not anovice programmer i like writing bash scripts but yeah cpp or other stuff like python its incredible powerfull stuff i try but yeah i try and find it very difficult to understand my mind is very simpel :)... i use to program on a c64 or actually revers engineer but that was ages ago,,, like assembler though not fully as pro but as hobbiest.
any way thanx for the very detailed additional information which i intend to look in further and do some research on it..

[natowi]

Your ideas are great, I also like to look into different directions.
If you have programs that can be addressed from cli, it is relatively easy to add them to Meshroom (like the ai upscale tools listed before)
Here is some documentation https://github.com/natowi/meshroom_external_plugins/wiki/2-DevDoc
You are most welcome to contribute in one form or the other to Meshroom.