Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ResNet-50 visualization #36

Open
t-softic opened this issue Nov 14, 2020 · 3 comments
Open

ResNet-50 visualization #36

t-softic opened this issue Nov 14, 2020 · 3 comments

Comments

@t-softic
Copy link

Hi,

I used fine-tuned resnet-50 and added a new classifier on top. When I try to visualize with CAM methods, I get meaningless heatmaps. On the other hand, when visualizing with gradient, there is an attention in regions that should be highlighted. I know ResNet is hard to visualize due to its skip connections, but do you have any tips on what I could do to improve CAM visualizations, or if there is any good library that could help me (I tried keras-vis same outcome).

Thanks,
Tarik

@keisen
Copy link
Owner

keisen commented Nov 14, 2020

Hi, @t-softic .

I believe that GradCAM method can work well with resnet-50 as follow:

https://github.com/raghakot/keras-vis/blob/master/examples/resnet/attention.ipynb

Could you please submit the heatmaps and a code snippet to reproduce and examine the problem?

Thanks!

@anotinelg
Copy link

Tensorflow 2.3.0

Hi, i have a "similar" problem.. it happens when i construct a new model from an already created.

Construction of model:

from tensorflow.keras.preprocessing.image import load_img
from tensorflow.keras.applications.resnet_v2 import ResNet50V2 as Model
from tensorflow.keras.applications.resnet_v2 import preprocess_input


# Load model
base_model = Model(weights='imagenet', include_top=False)

inputs = tf.keras.Input(shape=(224, 224, 3))
x = preprocess_input(inputs)
outputs = base_model(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

result of the plot:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_9 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
tf_op_layer_RealDiv_3 (Tenso [(None, 224, 224, 3)]     0         
_________________________________________________________________
tf_op_layer_Sub_3 (TensorFlo [(None, 224, 224, 3)]     0         
_________________________________________________________________
resnet50v2 (Functional)      (None, None, None, 2048)  23564800  
=================================================================
Total params: 23,564,800
Trainable params: 23,519,360
Non-trainable params: 45,440
_________________________________________________________________
None

then i basically code the same code which is in https://github.com/keisen/tf-keras-vis/blob/master/examples/attentions.ipynb

The error is:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-6193c76b1152> in <module>
     67 cam = gradcam(loss,
     68               X,
---> 69               penultimate_layer=-1, # model.layers number
     70              )
     71 cam = normalize(cam)

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tf_keras_vis/gradcam.py in __call__(self, loss, seed_input, penultimate_layer, seek_penultimate_conv_layer, activation_modifier, normalize_gradient, expand_cam, training)
     55         # Processing gradcam
     56         model = tf.keras.Model(inputs=self.model.inputs,
---> 57                                outputs=self.model.outputs + [penultimate_output_tensor])
     58         with tf.GradientTape(watch_accessed_variables=False) as tape:
     59             tape.watch(seed_inputs)

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py in __new__(cls, *args, **kwargs)
    240       # Functional model
    241       from tensorflow.python.keras.engine import functional  # pylint: disable=g-import-not-at-top
--> 242       return functional.Functional(*args, **kwargs)
    243     else:
    244       return super(Model, cls).__new__(cls, *args, **kwargs)

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
    455     self._self_setattr_tracking = False  # pylint: disable=protected-access
    456     try:
--> 457       result = method(self, *args, **kwargs)
    458     finally:
    459       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py in __init__(self, inputs, outputs, name, trainable)
    113     #     'arguments during initialization. Got an unexpected argument:')
    114     super(Functional, self).__init__(name=name, trainable=trainable)
--> 115     self._init_graph_network(inputs, outputs)
    116 
    117   @trackable.no_automatic_dependency_tracking

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
    455     self._self_setattr_tracking = False  # pylint: disable=protected-access
    456     try:
--> 457       result = method(self, *args, **kwargs)
    458     finally:
    459       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py in _init_graph_network(self, inputs, outputs)
    189     # Keep track of the network's nodes and layers.
    190     nodes, nodes_by_depth, layers, _ = _map_graph_network(
--> 191         self.inputs, self.outputs)
    192     self._network_nodes = nodes
    193     self._nodes_by_depth = nodes_by_depth

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py in _map_graph_network(inputs, outputs)
    929                              'The following previous layers '
    930                              'were accessed without issue: ' +
--> 931                              str(layers_with_complete_input))
    932         for x in nest.flatten(node.outputs):
    933           computable_tensors.add(id(x))

ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_8_1:0", shape=(None, None, None, 3), dtype=float32) at layer "conv1_pad". The following previous layers were accessed without issue: []

@anotinelg
Copy link

anotinelg commented Nov 17, 2020

hi @t-softic, have you tried to use GradCam on a Custom model that includes a resnet and then some top classifiers layers that you add?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants