NLP

Natural Language Processing techniques give machines ability to ready, understand, and derive meaning from human languages.

Applications

Machine Translation, Speech Recognition, Sentiment Analysis, Question Answering, Automatic Summarization, Chatbots, Text Classification, Character Recognition, Spell Checking

Topic Modeling

• Useful Model

  • Latent Dirichlet Allocation (LDA)
    • Latent Dirichlet Allocation with Non-negative Matrix Factorization (LDA&NMF)
    • Hierarchical Latent Dirichlet Allocation (hLDA)
  • Latent Semantic Analysis (LSA)
  • Probabilistic Latent Semantic Analysis (PLSA)
  • Correlated Topic Model (CTM)
  • Author Topic Model
  • BERTopic

• Tools

  • Visualization:
    • pyLDAvis
    • BertViz
    • exBert
  • Quick LDA Topic Modeling:https://lettier.com/projects/lda-topic-modeling/

LLM Base-Model

• LLaMA
• GLM
• BLOOM
• GPT
• PaLM

Fine-Tuning Technique

• LoRA
• Prefix Tuning
• P-Tuning
• AdaLoRA
• QLoRA
• Adapter
• MaM
• RLHF

Paper

• Latent Dirichlet Allocation
• Attention is All You Need (Transformer)
• Algorithms for Non-negative Matrix Factorization
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• BERTopic: Neural topic modeling with a class-based TF-IDF procedure
• T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• GPT-2: Language Models are Unsupervised Multitask Learners
• GPT-3: Language Models are Few-Shot Learners
• Prompt Learning: Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
• UniLM: Unified Language Model Pre-training for Natural Language Understanding and Generation
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• PaLM: Scaling Language Modeling with Pathways
• Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
• LoRA: Low-Rank Adaptation of Large Language Models
• RLHF: Deep Reinforcement Learning from Human Preferences
• P-Tuning V1: GPT Understands, Too
• P-Tuning V2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• Prefix: Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Adapter: Parameter-Efficient Transfer Learning for NLP
• ROUGE: A Package for Automatic Evaluation of Summaries
• Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark
• Constitutional AI: Harmlessness from AI Feedback
• Proximal Policy Optimization Algorithms(PPO)
• Knowledge Distillation: A Survey
• A Survey on Model Compression for Large Language Models
• DistillBert
• TinyBert: Distilling BERT for Natural Language Understanding
• RAG: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• ReAct
• CoT: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• A Survey of Large Language Models
• Secrets of RLHF in Large Language Models Part I: PPO
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Baichuan 2: Open Large-scale Language Models
• GLM-130B:An Open Bilingual Pre-trained Model