Only failed or skipped attack whith textfooler on custom bert model

[alixmouchet]

Hi all,

I have a pretrained Bert model for binary calssification that I wish to attack but all attack only fail or are skipped.

The output :

textattack: Unknown if model of class <class 'transformers.models.bert.modeling_bert.BertForSequenceClassification'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.

The Attack Recipe is: textfoolerjin2019
Attack(
(search_method): GreedyWordSwapWIR(
(wir_method): delete
)
(goal_function): UntargetedClassification
(transformation): WordSwapEmbedding(
(max_candidates): 50
(embedding): WordEmbedding
)
(constraints):
(0): WordEmbeddingDistance(
(embedding): WordEmbedding
(min_cos_sim): 0.5
(cased): False
(include_unknown_words): True
(compare_against_original): True
)
(1): PartOfSpeech(
(tagger_type): nltk
(tagset): universal
(allow_verb_noun_swap): True
(compare_against_original): True
)
(2): UniversalSentenceEncoder(
(metric): angular
(threshold): 0.840845057
(window_size): 15
(skip_text_shorter_than_window): True
(compare_against_original): False
)
(3): RepeatModification
(4): StopwordModification
(5): InputColumnModification(
(matching_column_labels): ['premise', 'hypothesis']
(columns_to_ignore): {'premise'}
)
(is_black_box): True
)

[Succeeded / Failed / Skipped / Total] 0 / 1 / 0 / 1: 7%| | 1/15 [00:13<03:15,

--------------------------------------------- Result 1 ---------------------------------------------
[[0 (67%)]] --> [[[FAILED]]]

-lrb- i -rrb- contains viruses , corrupted data or other harmful , disruptive or destructive file viruses , cancel bots , trojan horses , harmful code , flood pings , denial-of-service attacks , packet or ip spoofing , forged routing or electronic mail address information or similar methods or technology ;

[Succeeded / Failed / Skipped / Total] 0 / 2 / 0 / 2: 13%|▏| 2/15 [00:19<02:04,

--------------------------------------------- Result 2 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

if you have purchased a paid subscription using a code , your paid subscription will automatically terminate at the end of the pre-paid period , or when there is an insufficient pre-paid balance to pay for the service .

[Succeeded / Failed / Skipped / Total] 0 / 3 / 0 / 3: 20%|▏| 3/15 [00:20<01:21,

--------------------------------------------- Result 3 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

the services and the content are provided to you strictly on an `` as is '' basis .

[Succeeded / Failed / Skipped / Total] 0 / 4 / 1 / 5: 27%|▎| 4/15 [00:21<00:58,

--------------------------------------------- Result 4 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

you may opt out of this part of the terms .

--------------------------------------------- Result 5 ---------------------------------------------
[[0 (67%)]] --> [[[SKIPPED]]]

if however any consumer laws -lrb- e.g. , in australia , the competition and consumer act 2010 -lrb- cth -rrb- -rrb- do apply and can not otherwise be lawfully excluded , nothing in these user terms will restrict , exclude or modify any statutory warranties , guarantees , rights or remedies you have , and our liability is limited -lrb- at our option -rrb- to the replacement , repair or resupply of the services or the pro-rata refund to customer of pre-paid fees for your subscription covering the remainder of the term .

[Succeeded / Failed / Skipped / Total] 0 / 5 / 1 / 6: 33%|▎| 5/15 [00:23<00:47,

--------------------------------------------- Result 6 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

you can delete the link between your account and your facebook account at any time through the `` verifications '' section of your profile .

[Succeeded / Failed / Skipped / Total] 0 / 6 / 1 / 7: 40%|▍| 6/15 [00:26<00:39,

--------------------------------------------- Result 7 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

you hereby represent and warrant that you have the full power and authority to enter into and perform under these terms .

[Succeeded / Failed / Skipped / Total] 0 / 7 / 1 / 8: 47%|▍| 7/15 [00:28<00:32,

--------------------------------------------- Result 8 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

if you are a customer -lrb- defined below -rrb- , these customer terms govern your access and use of our services .

[Succeeded / Failed / Skipped / Total] 0 / 8 / 1 / 9: 53%|▌| 8/15 [00:29<00:25,

--------------------------------------------- Result 9 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

please read through these terms , our cookies policy and our privacy policy -lrb- together the `` data

[Succeeded / Failed / Skipped / Total] 0 / 9 / 1 / 10: 60%|▌| 9/15 [00:34<00:22

--------------------------------------------- Result 10 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

• is illegal , or intended to promote or commit an illegal act of any kind , including but not limited to violations of intellectual property rights , privacy rights , or proprietary rights of spotify or a third party ;

[Succeeded / Failed / Skipped / Total] 0 / 10 / 1 / 11: 67%|▋| 10/15 [00:38<00:

--------------------------------------------- Result 11 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

if you are joining a workspace created by your friend using her personal email address to work on her new startup idea , she is our customer and she is authorizing you to join her workspace .

[Succeeded / Failed / Skipped / Total] 0 / 11 / 1 / 12: 73%|▋| 11/15 [00:41<00:

--------------------------------------------- Result 12 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

we warrant to you that any item purchased from us through this website is of satisfactory quality and reasonably fit for all of the purposes for which items of the kind are commonly supplied .

[Succeeded / Failed / Skipped / Total] 0 / 12 / 2 / 14: 80%|▊| 12/15 [00:44<00:

--------------------------------------------- Result 13 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

seat access : if you are a pro or business subscriber , you may grant access to your account to other vimeo members subject to a seat limit .

--------------------------------------------- Result 14 ---------------------------------------------
[[0 (71%)]] --> [[[SKIPPED]]]

in addition to arbitration , you or the company may file a claim in small claims court or bring a claim to the attention of the federal , state or local government .

[Succeeded / Failed / Skipped / Total] 0 / 13 / 2 / 15: 87%|▊| 13/15 [00:46<00:

--------------------------------------------- Result 15 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

couchsurfing does not represent or warrant that our tools are sufficient to determine whether it is appropriate for you to interact with another member .

[Succeeded / Failed / Skipped / Total] 0 / 14 / 2 / 16: 93%|▉| 14/15 [00:50<00:

--------------------------------------------- Result 16 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

17.1 to contact skype in relation to the skype software or `` skype '' branded products , please submit a support request to support.skype.com .

[Succeeded / Failed / Skipped / Total] 0 / 15 / 2 / 17: 100%|█| 15/15 [00:54<00:

--------------------------------------------- Result 17 ---------------------------------------------
[[0 (71%)]] --> [[[FAILED]]]

we endeavour to provide the best service we can , but you understand and agree that the spotify service is provided as is '' and as available '' , without express or implied warranty or condition of any kind .

+-------------------------------+--------+
| Attack Results | |
+-------------------------------+--------+
| Number of successful attacks: | 0 |
| Number of failed attacks: | 15 |
| Number of skipped attacks: | 2 |
| Original accuracy: | 88.24% |
| Accuracy under attack: | 88.24% |
| Attack success rate: | 0.0% |
| Average perturbed word %: | nan% |
| Average num. words per input: | 30.29 |
| Avg num queries: | 324.87 |
+-------------------------------+--------+
CPU times: user 13.8 s, sys: 5.12 s, total: 18.9 s
Wall time: 54.6 s

I use the following wrapper:

class CustomModelWrapper(ModelWrapper):
def init(self, model):
self.model = model

def __call__(self, text_input_list):
    device = self.model.device
    encoded_input = tokenizer(
        text_input_list,
        truncation=True,
        padding="max_length",
        max_length=128,
        return_tensors="pt",
    ).to(device)
    #         print(encoded_input.device)
    with torch.no_grad():
        output = self.model(**encoded_input)
    logits = output.logits
    preds = torch.sigmoid(logits)
    return preds

with the model :
config = BertConfig.from_pretrained("bert_config.json")

bert_model = BertForSequenceClassification.from_pretrained("pytorch_model.bin", config=config)

bert_model = bert_model.to(device)

And then to attack :

def generate_attacks(
recipe, model_wrapper, dataset_to_attack, num_examples=-1, parallel=False
):
print(f"The Attack Recipe is: {recipe}")
if recipe == "textfoolerjin2019":
attack = TextFoolerJin2019.build(model_wrapper)
elif recipe == "TextBuggerLi2018":
attack = TextBuggerLi2018.build(model_wrapper)

attack_args = textattack.AttackArgs(
    num_examples=num_examples, shuffle=True, attack_n=10,#, num_workers_per_device=5
)  ## num_examples = -1 means the entire dataset
attacker = Attacker(attack, dataset_to_attack, attack_args)
attack_results = attacker.attack_dataset()
return attack_results

recipe = 'textfoolerjin2019'
test_attack_results = generate_attacks(recipe, model_wrapper, test_dataset_to_attack, num_examples=15)

if you have any idea of why it is the case it would be really helpful

[colinpannikkat]

I am also running into the same issue on a different dataset. Unsure how to proceed.