Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

big errors #2

Open
fatemehkalantari1993 opened this issue Jul 2, 2023 · 6 comments
Open

big errors #2

fatemehkalantari1993 opened this issue Jul 2, 2023 · 6 comments
Assignees
@micchu

Comments

@fatemehkalantari1993
Copy link

I had a problem with Caffe installation on ubuntu20.4 and I sent an email to you. You suggested me a code in the Torch framework (https://github.com/KamitaniLab/brain-decoding-cookbook-public/tree/main/reconstruction).
I did reconstruction with torch and Keras frameworks but I had bigger a error than Caffe (https://github.com/KamitaniLab/DeepImageReconstruction). when I ran this code, I got a big error too. I ran this code and compare its results with the results of Caffe (https://github.com/KamitaniLab/DeepImageReconstruction).

I said my problem to you and you said that this problem may due to my Cuda and Torch versions. I sent my results with your data, my Python and Torch versions, and python code for you. I have big errors. why????

Torch version: 1.12.1+cu116
python version: 3.10.6
result in iter200: error= 2594284032.0 for n01443537_22563.jpeg, sub-01
`

import argparse
import glob
from itertools import product
import os
import pickle

from bdpy.recon.torch.icnn import reconstruct
from bdpy.recon.utils import normalize_image, clip_extreme
from bdpy.dl.torch.models import VGG19, AlexNetGenerator, layer_map
from bdpy.dataform import Features, DecodedFeatures
from bdpy.feature import normalize_feature
from bdpy.util import dump_info
import numpy as np
import PIL.Image
import scipy.io as sio
import torch
print(torch.version)
import torch.optim as optim
import yaml

Functions

def image_preprocess(img, image_mean=np.float32([104, 117, 123])):
'''convert to Caffe's input image layout'''
return np.float32(np.transpose(img, (2, 0, 1))[::-1]) - np.reshape(image_mean, (3, 1, 1))

def image_deprocess(img, image_mean=np.float32([104, 117, 123])):
'''convert from Caffe's input image layout'''
return np.dstack((img + np.reshape(image_mean, (3, 1, 1)))[::-1])

Network settings -------------------------------------------------------

features_dir = '/home/mvl/kalantari/data/decoded_features/ImageNetTest/deeprecon_originals/VGG19'
output_dir = '/home/mvl/kalantari/results/'
subject = 'sub-01'
roi = 'VC'
#device = 'cuda:0'

encoder_param_file = '/home/mvl/kalantari/data/net/VGG_ILSVRC_19_layers/VGG_ILSVRC_19_layers.pt'

layers = [
'conv1_1', 'conv1_2', 'conv2_1', 'conv2_2',
'conv3_1', 'conv3_2', 'conv3_3', 'conv3_4',
'conv4_1', 'conv4_2', 'conv4_3', 'conv4_4',
'conv5_1', 'conv5_2', 'conv5_3', 'conv5_4',
]

layer_mapping = layer_map('vgg19')
encoder_input_shape = (224, 224, 3)

generator_param_file = '/home/mvl/kalantari/data/net/bvlc_reference_caffenet_generator_ILSVRC2012_Training/generator_relu7.pt'

image_mean_file = '/home/mvl/kalantari/data/net/VGG_ILSVRC_19_layers/ilsvrc_2012_mean.npy'
image_mean = np.load(image_mean_file)
image_mean = np.float32([image_mean[0].mean(), image_mean[1].mean(), image_mean[2].mean()])

feature_std_file = '/home/mvl/kalantari/data/net/VGG_ILSVRC_19_layers/estimated_cnn_feat_std_VGG_ILSVRC_19_layers_ImgSize_224x224_chwise_dof1.mat'
feature_range_file = '/home/mvl/kalantari/data/net/bvlc_reference_caffenet_generator_ILSVRC2012_Training/act_range/3x/relu7.txt'

std_ddof = 1
channel_axis = 0

n_iter = 200

# Reconstruction options -------------------------------------------------

opts = {
'loss_func': torch.nn.MSELoss(reduction='sum'),
'n_iter': n_iter,
'lr': (2., 1e-10),
'momentum': (0.9, 0.9),
'decay': (0.01, 0.01),
'blurring': False,
'channels': None,
'masks': None,
'disp_interval': 1,}

Initial image for the optimization (here we use the mean of ilsvrc_2012_mean.npy as RGB values)

initial_image = np.zeros((224, 224, 3), dtype='float32')
initial_image[:, :, 0] = image_mean[2].copy()
initial_image[:, :, 1] = image_mean[1].copy()
initial_image[:, :, 2] = image_mean[0].copy()

Feature SD estimated from true DNN features of 10000 images

feat_std0 = sio.loadmat(feature_std_file)

Feature upper/lower bounds

cols = 4096
up_size = (4096,)
upper_bound = np.loadtxt(feature_range_file,
delimiter=' ',
usecols=np.arange(0, cols),
unpack=True)
upper_bound = upper_bound.reshape(up_size)

Initial features -------------------------------------------------------

initial_gen_feat = np.random.normal(0, 1, (4096,))

Setup results directory ------------------------------------------------

if not os.path.exists(output_dir):
os.makedirs(output_dir)

Set reconstruction options ---------------------------------------------

opts.update({
# The initial image for the optimization (setting to None will use random noise as initial image)
'initial_feature': initial_gen_feat,
'feature_upper_bound': upper_bound,
'feature_lower_bound': 0.,
})

decoded = subject is not None and roi is not None
print('----------------------------------------')
if decoded:
print('Subject: ' + subject)
print('ROI: ' + roi)
print('')

if decoded:
save_dir = os.path.join(output_dir, subject, roi)
else:
save_dir = os.path.join(output_dir)

if not os.path.exists(save_dir):
os.makedirs(save_dir)

Get images if images is None

if decoded:
matfiles = glob.glob(os.path.join(features_dir, layers[0], subject, roi, '.mat'))
else:
matfiles = glob.glob(os.path.join(features_dir, layers[0], '.mat'))

images = [os.path.splitext(os.path.basename(fl))[0] for fl in matfiles]

Load DNN features

if decoded:
features = DecodedFeatures(os.path.join(features_dir), squeeze=False)
else:
features = Features(features_dir)

Images loop

for image_label in images[:1]:
print('Image: ' + image_label)

# Encoder model
encoder = VGG19()
encoder.load_state_dict(torch.load(encoder_param_file))
encoder.eval()

# Generator model
generator = AlexNetGenerator()
generator.load_state_dict(torch.load(generator_param_file))
generator.eval()

# Districuted computation control
snapshots_dir = os.path.join(save_dir, 'snapshots', 'image-%s' % image_label)
if os.path.exists(snapshots_dir):
   print('Already done or running. Skipped.')
   continue

# Load DNN features
if decoded:
   feat = {
       layer: features.get(layer=layer, subject=subject, roi=roi, image=image_label)
       for layer in layers
   }
else:
    labels = features.labels
    feat = {
        layer: features.get_features(layer)[np.array(labels) == image_label]
        for layer in layers
    }

for layer, ft in feat.items():

    ft0 = normalize_feature(
          ft[0],
          channel_wise_mean=False, channel_wise_std=False,
          channel_axis=channel_axis,
          shift='self', scale=np.mean(feat_std0[layer]),
          std_ddof=std_ddof
    )
    ft = ft0[np.newaxis]
    feat.update({layer: ft})

                # Norm of the DNN features for each layer
feat_norm = np.array([np.linalg.norm(feat[layer])
                      for layer in layers],
                     dtype='float32')
weights = 1. / (feat_norm ** 2)

        # Normalise the weights such that the sum of the weights = 1
weights = weights / weights.sum()
layer_weights = dict(zip(layers, weights))

opts.update({'layer_weights': layer_weights})

# Reconstruction
snapshots_dir = os.path.join(save_dir, 'snapshots', 'image-%s' % image_label)
recon_image, loss_list = reconstruct(feat,
                                     encoder,
                                     generator=generator,
                                     layer_mapping=layer_mapping,
                                     optimizer=optim.SGD,
                                     image_size=encoder_input_shape,
                                     crop_generator_output=True,
                                     preproc=image_preprocess,
                                     postproc=image_deprocess,
                                     output_dir=save_dir,
                                     save_snapshot=True,
                                     snapshot_dir=snapshots_dir,
                                     snapshot_ext='tiff',
                                     snapshot_postprocess=normalize_image,
                                     return_loss=True,
                                     **opts)

# Save the raw reconstructed image
recon_image_mat_file = os.path.join(save_dir, 'recon_image' + '-' + image_label + '.mat')
sio.savemat(recon_image_mat_file, {'recon_image': recon_image})

recon_image_normalized_file = os.path.join(save_dir, 'recon_image_normalized' + '-' + image_label + '.tiff')
PIL.Image.fromarray(normalize_image(clip_extreme(recon_image, pct=4))).save(recon_image_normalized_file)

print('All done')`
@ShuntaroAoki
Copy link
Member

Thank you for the detailed explanation. We have noticed the issue and are currently working on replicating and resolving it. However, due to limited resources, we have not yet been able to resolve the issue. I apologize for the inconvenience. I will try to provide you with a response within a week.

@fatemehkalantari1993
Copy link
Author

fatemehkalantari1993 commented Jul 3, 2023 via email

@fatemehkalantari1993
Copy link
Author

fatemehkalantari1993 commented Jul 4, 2023 via email

Open
@micchu
Copy link
Member

micchu commented Jul 20, 2023

Hello. Thank you for your question.
Unfortunately, we were unable to reproduce the issue.
Below is the environment during testing. We can successfully reconstruct without any problems using the script provided in this repository (recon_icnn_image_vgg19_dgn_relu7gen_gd.py).

  • OS: Ubuntu20.04
  • GPU driver version: 510.60.02
  • Python: 3.10.6
  • Torch+CUDA: 1.12.1+cu116
  • bdpy: 0.19

We know that there are cases where reconstruction fails with specific versions of Torch, such as Pytorch 1.9.1.
However, this was not the case this time.

Furthermore, the caffe output of the conv1_1 layer you provided appears to be correct.
For example, the activation of conv1_1 when the input is "n01443537_22563" is as follows, and the scale of values is close:

array([[[[ 2.51477509e+02,  1.94084152e+02,  1.90671219e+02, ...,
           1.94927628e+02,  1.95883209e+02, -2.81779814e+00],
         [ 1.57871948e+02, -1.71377697e+01, -1.65778790e+01, ...,
           3.12216401e+00,  1.02690327e+00, -2.09711792e+02],
         [ 1.51870148e+02, -1.34124165e+01, -6.63964939e+00, ...,
           3.86457968e+00,  3.44568682e+00, -2.05663879e+02],
         ...,
         [-1.31289581e+02, -1.28243113e+01, -3.33981857e+01, ...,
           4.42263079e+00,  5.39223528e+00, -2.66478455e+02],
         [-7.83458023e+01,  4.40543594e+01,  3.59504509e+01, ...,
           4.12912798e+00,  1.47437406e+00, -2.67864655e+02],
         [ 3.04511051e+01,  1.47859955e+02,  1.61898727e+02, ...,
          -2.52271164e+02, -2.55292587e+02, -3.83337128e+02]],

        [[-1.58220177e+01, -3.06688843e+01, -3.00492077e+01, ...,
          -3.19152451e+01, -3.08211956e+01, -4.19827347e+01],
         [-1.85093632e+01, -3.61226463e+01, -3.44978371e+01, ...,
          -3.92450104e+01, -3.79475136e+01, -5.40956421e+01],
         [-1.91209679e+01, -3.49747925e+01, -3.33080101e+01, ...,
          -3.95044479e+01, -3.87038498e+01, -5.48260994e+01],
          ...

On the other hand, your Keras output of conv1_1 seems to have all vectors taking the same value, which is clearly unusual.
It is likely that your problem is dependent on the implementation in Keras.
Could you please check if there are any errors in the preprocessing of the input image?

@fatemehkalantari1993
Copy link
Author

I have this problem with Torch and Keras. Errors in your code are too bigger than Caffe. why???
I don't have any error in the preprocessing of the input image because I checked the output of every line in Torch and Keras with the output of every line in the Caffe code. In fact, the problem was found from this line onwards.

@fatemehkalantari1993
Copy link
Author

fatemehkalantari1993 commented Jul 25, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@micchu micchu

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ShuntaroAoki @micchu @fatemehkalantari1993