New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

단어 전처리 bpe 진행 #21

Open

rainmaker712 opened this issue Mar 10, 2018 · 0 comments

Contributor

rainmaker712 commented Mar 10, 2018

단어를 전처리할때 bpe(byte pair encoding)으로 rare word를 sub word로 분해해서 사전의 크기를 조절하는기법이 있던데 이방법을 써보는건 어떨까요?

-> fasttext랑 비슷한 식으로 처리 할 수 있긴 합니다.

우선은 기본 세팅 해보고 향후 적용해보죠 의견 내주셔서 감사해요

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment