How can I train the adalora about nn.Conv1d?

[girlsending0]

Hi,
I face the problem when I finetune the Whisper model.
Whisper model is consisted of nn.Conv1d layer. so, I want finetune only encoder and try below:)

'''Model load'''
    pretrained = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base.en")
    pretrain_conv1 = nn.Conv1d(roi, 512, kernel_size = 3, stride=1, padding=1, bias=True)
    pretrained.model.encoder.conv1 = pretrain_conv1

    # decoder parameter freeze # 
    for name, param in pretrained.model.named_parameters():
        if param.requires_grad and 'decoder' in name:
            param.requires_grad = False

    model = Whisper_brain(pretrained_layers = pretrained)
    model = model.to(device)
    model = torch.nn.DataParallel(model, device_ids=device_ids)
    
    ''' AdaLora setting '''
    config = AdaLoraConfig(
                beta1=0.85,
                beta2=0.85,
                target_modules = ['encoder.conv2', 'encoder.self_attn.k_proj', 'encoder.self_attn.v_proj', 'encoder.self_attn.q_proj', 'encoder.self_attn.out_proj', 'encoder.fc1', 'encoder.fc2']# ['Conv1D', 'k_proj', 'v_proj', 'q_proj', 'out_proj', 'fc1', 'fc2']
            )
    
    model = get_peft_model(model, config, "default")

but I got error,

I think this error is because the whisper model consists of nn.Conv1d layer. However, Adalora only accepts Conv1D of the hugging face as target_modules.
Does anyone know about this error?

[BenjaminBossan]

Indeed, as you observed, the torch.nn.Conv1d layer (note: small d at the end) is not supported, only transformers.pytorch_utils.Conv1D (capital D). As such, you cannot fine-tune this type of layers until someone adds an implementation for it to PEFT.

The error message could certainly be improved, as the small vs capital d difference is very easy to miss.

[girlsending0]

@BenjaminBossan
Thank you so much for the new information!
I'll find another solution.