[Bug Report] Sometimes attention score on real text is < - 1e5 , set attn.IGNORE to -infinity

[neelnanda-io]

Minimal example:

model = HookedTransformer.from_pretrained("pythia-70m")
t2 = torch.tensor([   0, 1814, 1149, 1538, 1063, 1753, 1870, 1873, 1058, 1686, 1249, 1435,
        1134, 1557, 1230, 1845, 1459, 1188, 1135, 1219, 1545, 1284, 1278, 1810,
        1858, 1486, 1801, 1939, 1697, 1612, 1014, 1987, 1892, 1872, 1541, 1137,
        1386, 1703, 1355, 1197, 1228, 1331, 1394, 1383, 1261, 1629, 1559, 1309,
        1342, 1615, 1425, 1562, 1158, 1624, 1823, 1459, 1389, 1734, 1252, 1212,
        1422, 1666, 1792, 1940, 1599, 1685, 1516, 1116, 1946, 1929, 1347, 1373,
        1442, 1982, 1416, 1248, 1820, 1439, 1300, 1452, 1114, 1971, 1140, 1384,
        1593, 1922, 1382, 1206, 1290, 1333, 1298, 1733, 1897, 1880, 1585, 1234,
        1868, 1921, 1133, 1670, 1847, 1002, 1861, 1661, 1690, 1346, 1025, 1652,
        1245, 1256, 1221, 1388, 1455, 1785, 1297, 1299, 1166, 1583, 1027, 1108,
        1791, 1083, 1959, 1250, 1519, 1270, 1063, 1999, 1059, 1814, 1149, 1538,
        1063, 1753, 1870, 1873, 1058, 1686, 1249, 1435, 1134, 1557, 1230, 1845,
        1459, 1188, 1135, 1219, 1545, 1284, 1278, 1810, 1858, 1486, 1801, 1939,
        1697, 1612, 1014, 1987, 1892, 1872, 1541, 1137, 1386, 1703, 1355, 1197,
        1228, 1331, 1394, 1383, 1261, 1629, 1559, 1309, 1342, 1615, 1425, 1562,
        1158, 1624, 1823, 1459, 1389, 1734, 1252, 1212, 1422, 1666, 1792, 1940,
        1599, 1685, 1516, 1116, 1946, 1929, 1347, 1373, 1442, 1982, 1416, 1248,
        1820, 1439, 1300, 1452, 1114, 1971, 1140, 1384, 1593, 1922, 1382, 1206,
        1290, 1333, 1298, 1733, 1897, 1880, 1585, 1234, 1868, 1921, 1133, 1670,
        1847, 1002, 1861, 1661, 1690, 1346, 1025, 1652, 1245, 1256, 1221, 1388,
        1455, 1785, 1297, 1299, 1166, 1583, 1027, 1108, 1791, 1083, 1959, 1250,
        1519, 1270, 1063, 1999, 1059])
_, c2 = model.run_with_cache(t2)
line(c2["attn_scores", 4][0, 3, 214])

This gives attention scores that are -116,000 on the correct segment, such that the pattern is uniform across things outside the causal mask. I have no idea why this happens, and it seems ridiculous, but we should set attn.IGNORE to -torch.inf probably