Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to use eval_tool? #10

Open
ftlong6666 opened this issue Mar 4, 2020 · 43 comments
Open

How to use eval_tool? #10

ftlong6666 opened this issue Mar 4, 2020 · 43 comments

Comments

@ftlong6666
Copy link

Hello @fyangneil could you explain the step how to do evaluation?,

  1. should we convert all the predicted .png file into .mat?
  2. How to setup your toolbox? I always got this error.
Error in crack_nms (line 11)
    load(fullfile(path_to_input, iids(i).name));
  1. What should we do first? I tried to compile the toolbox successfully and modify the code into this.
mkdir(path_to_output);
iids = dir(fullfile(path_to_input, '*.png'));
for i = 1:length(iids)
    %edge = imread(fullfile(path_to_input, iids(i).name));\
    %load(fullfile(path_to_input, iids(i).name));
    %predmap=predmap;
    %disp(fullfile(path_to_input, iids(i).name))
    filename = fullfile(path_to_input, iids(i).name);
    predmap = imread(filename);
    predmap = rgb2gray(predmap);
    [Ox, Oy] = gradient2(convTri(predmap, 4));
    [Oxx, ~] = gradient2(Ox);
    [Oxy, Oyy] = gradient2(Oy);
    O = mod(atan(Oyy .* sign(-Oxy) ./ (Oxx + 1e-5)), pi);
    predmap = edgesNmsMex(predmap, O, 2, 5, 1.01, 8);
    imwrite(predmap, fullfile(path_to_output, [iids(i).name(1:end-4) '.png']));
    save(fullfile(path_to_output, [iids(i).name(1:end-4) '.mat']),'predmap');
end

However I got this error. Suppose that I have the predicted crack in .png file in the input_folder and I want to get the evaluation result only.

Error using convConst
A must be a 4x4 or bigger 2D or 3D float array.

Error in convTri (line 87)
    J = convConst('convTri',I,r,s);

Error in crack_nms (line 17)
    [Ox, Oy] = gradient2(convTri(predmap, 4));
@fyangneil
Copy link
Owner

@ftlong6666

  1. you should convert to 'mat' file and refer to the results we uploaded to Google Drive or Baidu Yunpan.
  2. You can use our dataset and provided results to test if the tool can work or not. If everything goes well, you should get the results reported in our paper.
  3. Before using the tool, please make sure that in predicted crack map the bright regions are crack, background is black. To get ODS and OIS, run 'crack_nms.m' first then 'crack_eval.m'. To get AIU, run 'crack_AIU.m'..

@ftlong6666
Copy link
Author

ftlong6666 commented Mar 5, 2020 •
edited
Loading

Hi I have tried it and I still got this error.

Error using convConst
A must be a 4x4 or bigger 2D or 3D float array.

Error in convTri (line 87)
    J = convConst('convTri',I,r,s);

Error in crack_nms (line 14)
    [Ox, Oy] = gradient2(convTri(predmap, 4));

Also how to change the path of Ground truth?, where path_to_input is the path of result.

@fyangneil
Copy link
Owner

fyangneil commented Mar 5, 2020 •
edited
Loading

@ftlong6666
Could you provide more details of how you use the tool.
path_to_input is the directory to the crack prediction results (mat files).

@ftlong6666
Copy link
Author

@fyangneil After I check I realized that the testdata is not in mat file. Here is the step to produce the error.

  1. Extract your test data and fphb result
  2. Change the path_to_input to the fphb location (the result in mat)
  3. run the crack_nms.
    However here is the things: I don't know how to point the program to ground-truh location, and your testing groundtruth has no .mat file, so I need to convert to mat first. Do you have a code to convert all of the .png files into mat?
    Thank you

@ftlong6666
Copy link
Author

Also I confuse with your code in file crack_nms.m, in line 9 predmap=predmap; where the predmap is not assign first.

@fyangneil
Copy link
Owner

@to run crack_nms, you do not need the ground truth files, only crack prediction files are needed.

@fyangneil
Copy link
Owner

@ftlong6666 the predmap is from line 8 "load(fullfile(path_to_input, iids(i).name))"

@ftlong6666
Copy link
Author

@fyangneil Thank you for quick reply it helps a lot. I have changed it to
predmap = load(fullfile(path_to_input, iids(i).name));
but I got this error.

Error using double
Conversion to double from struct is not possible.

Error in convn (line 27)
    A = double(A);

Error in convTri (line 93)
  J = convn(convn(J,f,'valid'),f','valid');

Error in crack_nms (line 15)
    [Ox, Oy] = gradient2(convTri(predmap, 4));

@fyangneil
Copy link
Owner

@ftlong6666 you do not need to change the line 9, it should work.

@ftlong6666
Copy link
Author

ftlong6666 commented Mar 5, 2020 •
edited
Loading

@fyangneil , I used exactly the same code here is the full code.

clear; clc;
path_to_input='../fphb_testresult/hed_fuse_fpn_ada_python_v1_iter_12000';
path_to_output='../fphb_testresult/hed_fuse_fpn_ada_python_v1_iter_12000/nms';
mkdir(path_to_output);
iids = dir(fullfile(path_to_input, '*.mat'));
for i = 1:length(iids)
    %edge = imread(fullfile(path_to_input, iids(i).name));\
    load(fullfile(path_to_input, iids(i).name));
    predmap=predmap;
    [Ox, Oy] = gradient2(convTri(predmap, 4));
    [Oxx, ~] = gradient2(Ox);
    [Oxy, Oyy] = gradient2(Oy);
    O = mod(atan(Oyy .* sign(-Oxy) ./ (Oxx + 1e-5)), pi);
    predmap = edgesNmsMex(predmap, O, 2, 5, 1.01, 8);
    imwrite(predmap, fullfile(path_to_output, [iids(i).name(1:end-4) '.png']));
    save(fullfile(path_to_output, [iids(i).name(1:end-4) '.mat']),'predmap');
end

But I still got this error message.

Error using convConst
A must be a 4x4 or bigger 2D or 3D float array.

Error in convTri (line 87)
    J = convConst('convTri',I,r,s);

Error in crack_nms (line 10)
    [Ox, Oy] = gradient2(convTri(predmap, 4));

/fphb_testresult/hed_fuse_fpn_ada_python_v1_iter_12000 is the folder which contains your result.

@ftlong6666
Copy link
Author

Here is the workspace.
image

@fyangneil
Copy link
Owner

@ftlong6666 what is the version of your matlab

@ftlong6666
Copy link
Author

Matlab R2019b Windows 10

@fyangneil
Copy link
Owner

@ftlong6666 could you show the predmap?

@ftlong6666
Copy link
Author

@fyangneil Here is the predmap variables.
image

@fyangneil
Copy link
Owner

@ftlong6666 looks good, it should be fine.Could you set a breakpoint in " J = convConst('convTri',I,r,s);"

@ftlong6666
Copy link
Author

Sure here is the result.
image
Also the content of variables.
image
Here is the l.
image

@fyangneil
Copy link
Owner

@ftlong6666 have compiled the edges toolbox?

@ftlong6666
Copy link
Author

Sure I have compiled it before.
image

@ftlong6666
Copy link
Author

I do this one to compile.

Simply unzip, then add all directories to the Matlab path:
  >> addpath(genpath('/edges/toolbox/')); savepath;

If needed, run the compile script for the mex files:
  >> toolboxCompile;

@fyangneil
Copy link
Owner

fyangneil commented Mar 5, 2020 via email

@ftlong6666
Copy link
Author

  1. What is your Matlab version?
  2. Do you have a code to convert .png result into .mat?
  3. Why we don't need the ground truth?, we should compare the result against the ground truth right?

@fyangneil
Copy link
Owner

fyangneil commented Mar 5, 2020 via email

@ftlong6666
Copy link
Author

@fyangneil You mentioned that to run nms_crack.m we don't need ground-truth. So if we need it, in which part should I change the path? Also what is the dimension of the mat file?, is it in HxWx3 or HxWx1 or HxW?

@fyangneil
Copy link
Owner

@ftlong6666 for the deimension of mat file, you can check our provided results. When you run crack_eval.m and crack_AIU.m, you need ground truth files.

@ftlong6666
Copy link
Author

@fyangneil If you don't have the code to convert .png to .mat, do you have the .mat of the groundtruth?, as long as you have the result, you should have the .mat of the groundtruth right?, in case you have, could you also upload it?
Thank you

@ftlong6666
Copy link
Author

Hello @fyangneil its just working, I don't know why. Anyway after I run the nms_crack.m , I try to run crack_eval.m . But I got this error, do you know why?

Index exceeds the number of array elements (0).

Error in edgesEvalDir_crack (line 68)
for i=1:n, id=ids{i}(1:end-4);idRes=idsRes{i}(1:end-4);

Error in try_old2 (line 5)
edgesEvalDir_crack('resDir',resDir,'gtDir',gtDir, 'thin', 1, 'pDistr',{{'type','parfor'}},'maxDist',0.0075);

Also in your code I see that you point it to directory testcrop folder, but I also see testdata folder, so which one is used? Also I realized that both of them don't have .mat files.

@fyangneil
Copy link
Owner

@ftlong6666 regarding crack_eval, only png image are needed. After nms, you can get png files, which used for evaluation. The ground truth file used for evaluation is also png files.
The images in testcrop folder are the cropped chips from images in testdata. For details, you are referred to our paper (FPHBN).

@ftlong6666
Copy link
Author

@fyangneil yes got it, but the problem is your code still looking for ".mat" files. First I put this line in edgesEvalImg.m in line 54 so the full code will be like this.

% load edges (E) and ground truth (G)
disp(E);
if(all(ischar(E))), E=double(imread(E))/255; end
G=load(G); G=G.groundTruth; n=length(G);
for g=1:n, G{g}=double(G{g}.Boundaries); end

Then after I run it, the result is still an error message like this.

FPHB
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_081839_1281_1.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1281_1081.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1_361.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_163940_1_1081.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_164000_1281_1081.mat
Error using imread>get_format_info (line 541)
Unable to determine the file format.

Error in imread (line 389)
        fmt_s = get_format_info(fullname);

Error in edgesEvalImg (line 55)
if(all(ischar(E))), E=double(imread(E))/255; end

Error in fevalDistr>fedParfor (line 99)
parfor i=1:nJob, r=feval(funNm,jobs{i}{:});

Error in fevalDistr (line 80)
  case 'parfor',    [out,res]=fedParfor(funNm,jobs,store);

Error in edgesEvalDir_crack (line 74)
fevalDistr('edgesEvalImg',jobs(do),p.pDistr{:});

Error in try_old2 (line 5)
edgesEvalDir_crack('resDir',resDir,'gtDir',gtDir, 'thin', 1, 'pDistr',{{'type','parfor'}},'maxDist',0.0075);

Thus the error said that your code crack_eval.m is still looking for mat file. I am using your result that you provided in Google drive. Do you know what I am missing in here?, from the error I can see that your code try to load in image, but the path is still .mat .

@ftlong6666
Copy link
Author

ftlong6666 commented Mar 8, 2020 •
edited
Loading

@fyangneil Are you sure use this code to provide the result?, because I just realize that the path of the result needs .png but the groundtruth should be in .mat. Let me show you.
In the code edgesEvalDir.m line 70 you type this one.

im1=fullfile(resDir,[idRes '.mat']); gt1=fullfile(gtDir,[id '.png']);

but in edgesEvalImg.m line 53 if we print out the path, the E is the mat and G is the .png.

% load edges (E) and ground truth (G)
disp(E);
disp(G);
if(all(ischar(E))), E=double(imread(E))/255; end
G=load(G); G=G.groundTruth; n=length(G);
for g=1:n, G{g}=double(G{g}.Boundaries); end

so here is the result.

FPHB
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_081839_1281_1.mat
..\testcrop2\20160222_081839_1281_1.png
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1281_1081.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1_361.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_115305_1921_361.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_163940_1_1081.mat
..\testcrop2\20160222_163940_1_1081.png
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_164000_641_1.mat
Error using imread>get_format_info (line 541)
Unable to determine the file format.

Error in imread (line 389)
        fmt_s = get_format_info(fullname);

Error in edgesEvalImg (line 56)
if(all(ischar(E))), E=double(imread(E))/255; end

Error in fevalDistr>fedParfor (line 99)
parfor i=1:nJob, r=feval(funNm,jobs{i}{:});

Error in fevalDistr (line 80)
  case 'parfor',    [out,res]=fedParfor(funNm,jobs,store);

Error in edgesEvalDir_crack (line 75)
fevalDistr('edgesEvalImg',jobs(do),p.pDistr{:});

Error in try_old2 (line 6)
edgesEvalDir_crack('resDir',resDir,'gtDir',gtDir, 'thin', 1, 'pDistr',{{'type','parfor'}},'maxDist',0.0075);

From here we can see that you try to read .mat files with imread, and read groundtruth .png with load which is wrong. If I change all of the GT to .mat files, there is no fields groundtruthand Boundaries. Yes this make sense that this code originally used to evaluate the edge detection from BSDS but since you have ground truth by your self, you should format it in the same field as in BSDS. Please could you help me how to do the evaluation?, I critically found that this is very crucial part, because its related with the comparison with another method.
Thank you

@fyangneil
Copy link
Owner

@ftlong6666 my code does not use 'edgesEvalImg.m', it uses edgesEvalDir_crack.m, the line 70 is 'im1=fullfile(resDir,[idRes '.mat']); gt1=fullfile(gtDir,[id '.png']);' . So my code use mat file for prediction results and png file for ground truth.
'

@ftlong6666
Copy link
Author

@fyangneil Yes I also use it but the edgesEvalDir_crack.m also point to edgesEvalImg as stated in line 72 in edgesEvalImg(jobs{i}{:}); (edgesEvalDir_Crack.m line 72). should I change it to another function?

for i=1:n, id=ids{i}(1:end-4);idRes=idsRes{i}(1:end-4);
  res{i}=fullfile(evalDir,[id '_ev1.txt']); do(i)=~exist(res{i},'file');
  im1=fullfile(resDir,[idRes '.mat']); gt1=fullfile(gtDir,[id '.png']);
  jobs{i}={im1,gt1,'out',res{i},'thrs',p.thrs,'maxDist',p.maxDist,...
    'thin',p.thin}; if(0), edgesEvalImg(jobs{i}{:}); end
end

@fyangneil
Copy link
Owner

@ftlong6666 in my code, the line 56 in edgesEvalImg is load(E).

@ftlong6666
Copy link
Author

ftlong6666 commented Mar 8, 2020 •
edited
Loading

@fyangneil Hi finally I resolved all the issues. Actually I run your code but because I have tested the edge detection before and my matlab point to my previous experiment, so it point to the original edge detection metric code because the function name is simillar. After I used the tescrop data I got this result.

Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 8).
ODS=0.604 OIS=0.635 AP=0.363 R50=0.677 - FPHB
AIU 0.4299
IdleTimeout has been reached.
Parallel pool using the 'local' profile is shutting down.

From here I got the same OIS and ODS in the paper. However in here my AIU is 0.4299 which is lower than the paper which is 0.489. I got your pre-computed result that you shared in Gdrive. BTW is there any mistake that I have done in here?, Also how to the plot of the AIU, Precission, and Recall?
Thank you so much for your help this 2 days, it help a lot :) .

@fyangneil
Copy link
Owner

@ftlong6666 To compute AIU, you do not need nms, just use the provided prediction results. To display the plot of curve, you can use edgesEvalPlot.

@ftlong6666
Copy link
Author

@fyangneil Hi it works, thank you so much for your help.

@ftlong6666
Copy link
Author

@fyangneil Sorry last question. Do you have others method results that you presented in your paper? such as HED,RCF,FCN, and CrackForest? I only found your method results in your drive. Thanks a lot for your help.

@fyangneil
Copy link
Owner

@ftlong6666 regarding the results of other method, i just upload the CrackForest results on google drive. For the other method's results, you may need to run their codes. Since it is a long time ago that i conducted the experiments, the results cannot be found anymore.

@ftlong6666
Copy link
Author

Hello @fyangneil, thank you for your support, now I have 3 questions for the evaluation.

  1. What is the input image size for FPHB network?, since its fully convolutional it should be not fixed right. However if we use original size it will be time consuming. In your code at test_fphb_crack.prototxt I found this line.
name: "fphb"

input: "data"
input_dim: 1
input_dim: 3
input_dim: 500
input_dim: 500

did you resize the image to 500x500 and do inference later up-sampled again to original size?

  1. I always see N/A for AIU measurement in crackforest dataset, what is it means?

  2. You also measure the time of FPHB. Related with question 1, if your input is 500x500 then why different dataset has different running time?

@fyangneil
Copy link
Owner

fyangneil commented Apr 24, 2020 via email

@ftlong6666
Copy link
Author

Hello @fyangneil thank you for your reply. As you mentioned before the result of Crackforest is binary so the AIU is N/A then how can we evaluate the ODS and OIS? since there is no threshold is used. You also produce same OIS and ODS but different for each dataset. How to get this number?

@fyangneil
Copy link
Owner

fyangneil commented Dec 14, 2020 via email

@lian666-ch
Copy link

lian666-ch commented Jan 9, 2022 •
edited
Loading

我直接给matlab添加地址后直接就能用了
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@fyangneil @ftlong6666 @lian666-ch