Recovering the Learnt Prompt in text

[pharringtonp19]

I added the following function to the notebook in an attempt to recover the learnt prompt:

I get "['Åį', 'ĠIPO', 'वà¥Īधानिà¤ķ', 'Ġನಿಯ', 'Ġlistener', 'çĤºãĢĮ', 'ÙĦا', 'Ġcomplaint']".

Are there better strategies?

# Normalize the embeddings
def normalize_embeddings(embeddings):
    norms = torch.norm(embeddings, p=2, dim=1, keepdim=True)
    return embeddings / norms


def print_learning(mymodel):
    prompt_embeddings = mymodel.prompt_encoder['default'].embedding.weight
    prompt_embeddings = normalize_embeddings(prompt_embeddings)
    word_embeddings = normalize_embeddings(model.word_embeddings.weight)
    # Cosine similarity
    cosine_sim = torch.mm(prompt_embeddings, word_embeddings.t())
    # Get the top k nearest neighbors for each virtual token
    k = 1  # for example, top 5 nearest neighbors
    nearest_neighbors = torch.topk(cosine_sim, k, largest=True)
    nearest_neighbors.values.shape
    vocab_size = len(tokenizer)  # as per your model's embedding layer size
    vocab = {i: tokenizer.convert_ids_to_tokens(i) for i in range(vocab_size)}
    print([vocab[nearest_neighbors.indices[i].item()] for i  in range(peft_config.num_virtual_tokens)])

[BenjaminBossan]

From my understanding, the prompt does not necessarily have to make any sense, it is just the learned prompt that best steers the model in the right direction.

[pharringtonp19]

@BenjaminBossan That's my understanding as well. However, I am working on the following problem:

I have a sequence of documents. I would like to learn what changes in the documents along the sequence. One approach that I've considered is to learn this pattern via prompt tuning where I pass as input to the model [prompt] + [document_n] and ask it to predict [document_n+1]. By learning the prompt in this context I should be able to learn what's changing along the sequence of documents.

[BenjaminBossan]

I'm not sure I completely understand your use case, but regardless of that, I wouldn't expect the result to make any human-interpretable sense.

As to the function you proposed, it looks correct to me. As a sanity check, you could try running it on an embedding of known tokens (+ some noise) and see if it recovers the initial tokens.

[pharringtonp19]

@BenjaminBossan Will do! Thanks for the help.