https://arxiv.org/abs/2012.00857
StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling (Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, Aaron Courville)
constituency와 dependency 구조를 모두 추출하는 unsupervised parsing. 파서 모듈로 dependency 구조를 추출한 다음 self attention에 주입하고 mlm으로 학습.
#parse #attention #pretraining #mlm