Is there an issue with coordinate mapping in fine_matching?

[guanfang12]

In the ’get_fine_ds_match function‘ of fine_matching.py, there is a - W // 2 operation when calculating mkpts0_c, which seems to assume that the initial mkpts0_c is at the center of the current cropped patch. However, when I tested the following code in fine_preprocess.py, it appears that the crop uses the result of mkpts0_c as the top-left element of the crop:

from kornia.utils.grid import create_meshgrid
feat_arange = create_meshgrid(1152,1152, False,'cuda').permute(0,3,1,2).repeat(2,1,1,1) # [2, 2, 1152, 1152]
feat_arange = F.unfold(feat_arange, kernel_size=(W, W), stride=stride, padding=0) # [2, 128, 20736] | (1152/8)^2 = 20736
feat_arange = rearrange(feat_arange, 'n (c ww) l -> n l ww c', ww=W**2) # [2, 20736, 64, 2] | W*W = 64

feat_arange = create_meshgrid(1152,1152, False,'cuda').permute(0,3,1,2).repeat(2,1,1,1) # [2, 2, 1152, 1152]
feat_arange = F.unfold(feat_arange, kernel_size=(W+2, W+2), stride=stride, padding=1) # [2, 200, 20736] | (1152/8)^2 = 20736
feat_arange = rearrange(feat_arange, 'n (c ww) l -> n l ww c', ww=(W+2)**2) # [2, 20736, 100, 2] | W*W = 64

# for test LoFTR orig crop method
feat_arange = create_meshgrid(1152,1152, False,'cuda').permute(0,3,1,2).repeat(2,1,1,1) # [2, 2, 1152, 1152]
feat_arange = F.unfold(feat_arange, kernel_size=(W, W), stride=stride, padding=W//2)
feat_arange = rearrange(feat_arange, 'n (c ww) l -> n l ww c', ww=W**2) # [2, 20736, 64, 2] | W*W = 64

• Using ELoFTR crop code shows that the top-left corner of the patch corresponds to the coarse matching result:

• Using LoFTR crop code shows that the center of the patch corresponds to the coarse matching result:
  

Furthermore, in fine_matching, when updating mkpts0_c, it seems that the mapping formula with align_corner=False is not used correctly; it only applies a +0.5 operation without executing -0.5 after scaling:

• v = (i + 0.5)*scale - 0.5
• u = (j + 0.5)*scale - 0.5

[guanfang12]

Perhaps changing the calculation method of mkpts0_c from align_corner = True to align_corner = False in coarse_matching seems to solve the above problem.

        # orig: align_corner = True; v = i*scale
        mkpts0_c = torch.stack(
            [i_ids % data['hw0_c'][1], i_ids // data['hw0_c'][1]],
            dim=1) * scale0

        # suggestion: align_corner = False; v = (i + 0.5)*scale - 0.5
        mkpts0_c = (torch.stack(
            [i_ids % data['hw0_c'][1], i_ids // data['hw0_c'][1]],
            dim=1) + 0.5) * scale0 - 0.5

For example, as for i_ids = 0, the corresponding pixel will be [(0 + 0.5) * 8 - 0.5, (0 + 0.5) * 8 - 0.5] = [3.5, 3.5]. At this time, mkpts0_c is located at the center of croped patches.

[boyagesmile]

{'auc@10': 0.7030638823809927,
'auc@20': 0.8219782096445133,
'auc@5': 0.5396609255151746,
'num_matches': 1827.398,
'prec@5e-04': 0.9611862659919719}
When I use this method to test, I get worse results, why is this?

Perhaps changing the calculation method of mkpts0_c from align_corner = True to align_corner = False in coarse_matching seems to solve the above problem.

        # orig: align_corner = True; v = i*scale
        mkpts0_c = torch.stack(
            [i_ids % data['hw0_c'][1], i_ids // data['hw0_c'][1]],
            dim=1) * scale0

        # suggestion: align_corner = False; v = (i + 0.5)*scale - 0.5
        mkpts0_c = (torch.stack(
            [i_ids % data['hw0_c'][1], i_ids // data['hw0_c'][1]],
            dim=1) + 0.5) * scale0 - 0.5

For example, as for i_ids = 0, the corresponding pixel will be [(0 + 0.5) * 8 - 0.5, (0 + 0.5) * 8 - 0.5] = [3.5, 3.5]. At this time, mkpts0_c is located at the center of croped patches.

[guanfang12]

{'auc@10': 0.7030638823809927, 'auc@20': 0.8219782096445133, 'auc@5': 0.5396609255151746, 'num_matches': 1827.398, 'prec@5e-04': 0.9611862659919719} When I use this method to test, I get worse results, why is this?

Perhaps changing the calculation method of mkpts0_c from align_corner = True to align_corner = False in coarse_matching seems to solve the above problem.

        # orig: align_corner = True; v = i*scale
        mkpts0_c = torch.stack(
            [i_ids % data['hw0_c'][1], i_ids // data['hw0_c'][1]],
            dim=1) * scale0

        # suggestion: align_corner = False; v = (i + 0.5)*scale - 0.5
        mkpts0_c = (torch.stack(
            [i_ids % data['hw0_c'][1], i_ids // data['hw0_c'][1]],
            dim=1) + 0.5) * scale0 - 0.5

For example, as for i_ids = 0, the corresponding pixel will be [(0 + 0.5) * 8 - 0.5, (0 + 0.5) * 8 - 0.5] = [3.5, 3.5]. At this time, mkpts0_c is located at the center of croped patches.

It may be because training uses biased settings. If you use the officially provided checkpoint, directly modifying coarse matching will definitely reduce the results. If you want to modify coarse matching for training, the corresponding supervision theoretically needs to be further modified.

[boyagesmile]

{'auc@10': 0.7030638823809927, 'auc@20': 0.8219782096445133, 'auc@5': 0.5396609255151746, 'num_matches': 1827.398, 'prec@5e-04': 0.9611862659919719} When I use this method to test, I get worse results, why is this?

Perhaps changing the calculation method of mkpts0_c from align_corner = True to align_corner = False in coarse_matching seems to solve the above problem.

        # orig: align_corner = True; v = i*scale
        mkpts0_c = torch.stack(
            [i_ids % data['hw0_c'][1], i_ids // data['hw0_c'][1]],
            dim=1) * scale0

        # suggestion: align_corner = False; v = (i + 0.5)*scale - 0.5
        mkpts0_c = (torch.stack(
            [i_ids % data['hw0_c'][1], i_ids // data['hw0_c'][1]],
            dim=1) + 0.5) * scale0 - 0.5

For example, as for i_ids = 0, the corresponding pixel will be [(0 + 0.5) * 8 - 0.5, (0 + 0.5) * 8 - 0.5] = [3.5, 3.5]. At this time, mkpts0_c is located at the center of croped patches.

It may be because training uses biased settings. If you use the officially provided checkpoint, directly modifying coarse matching will definitely reduce the results. If you want to modify coarse matching for training, the corresponding supervision theoretically needs to be further modified.

so，I should change the spvs_coarse() in the supervision.py in the same time?

    grid_pt0_c = create_meshgrid(h0, w0, False, device).reshape(1, h0 * w0, 2).repeat(N, 1, 1)  # [N, hw, 2]
    grid_pt0_c += 0.5  
    grid_pt0_i = scale0 * grid_pt0_c - 0.5
    grid_pt1_c = create_meshgrid(h1, w1, False, device).reshape(1, h1 * w1, 2).repeat(N, 1, 1)
    grid_pt1_c += 0.5  
    grid_pt1_i = scale1 * grid_pt1_c - 0.5 

and

    w_pt0_c = (w_pt0_i +0.5) / scale1 - 0.5
    w_pt1_c = (w_pt1_i+0.5) / scale0 - 0.5

I wonder to know that the monitoring method before the transformation seems to get the coordinates of the point in the top left corner and then adjust it, but the adjustment position according to get_fine_ds_match() of fine_matching.py is based on the shift of the center point, which is not corresponding. But still got a good result.