Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix and rework GPT-TF.js #807

Open
wants to merge 18 commits into
base: develop
Choose a base branch
from
Open

Fix and rework GPT-TF.js #807

wants to merge 18 commits into from

Commits on Nov 14, 2024

  1. docs/CONTRIBUTING: add documentation on debug statements and cypress

    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    686e2a0 View commit details
    Browse the repository at this point in the history

  2. discojs/src/default_tasks: change default batch size and block size

    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    1be33d6 View commit details
    Browse the repository at this point in the history

  3. cli: add train_gpt script

    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    0491ee6 View commit details
    Browse the repository at this point in the history

  4. discojs/src/models/gpt/models: fix loss averaging across iterations

    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    7f7b778 View commit details
    Browse the repository at this point in the history

  5. discojs & cli: change gpt2 vocab size to 50257 instead of 50258

    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    7a14023 View commit details
    Browse the repository at this point in the history

  6. discojs/src/models/gpt: allow model init config to be partial

    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    d87f51a View commit details
    Browse the repository at this point in the history

  7. discojs/src/models/gpt/index: link source repo

    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    e947919 View commit details
    Browse the repository at this point in the history

  8. discojs/src/models/gpt: always use the token embeddings, a language m… 

    …odeling head and attention bias
    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    929deb8 View commit details
    Browse the repository at this point in the history

  9. discojs/src/models/gpt: generation code: clean, document and improve.… 

    … implement topk sampling
    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    89b3cac View commit details
    Browse the repository at this point in the history

  10. discojs/src/models/gpt/layers: document tensor operations, rename lay… 

    …ers following GPT2 convention, use LogLayer
    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    b8945aa View commit details
    Browse the repository at this point in the history

  11. discojs/src/models/gpt/layers: share weights between token embeddings… 

    … and language modeling head
    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    38e5882 View commit details
    Browse the repository at this point in the history

  12. discojs/src/models/gpt/layers: fix weight initializations

    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    2632737 View commit details
    Browse the repository at this point in the history

  13. discojs/src/models/gpt: add seed, loss is now identical between runs

    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    a6a517b View commit details
    Browse the repository at this point in the history

  14. discojs/src/task/training_information: make maxSequenceLength a requi… 

    …red task parameter for text tasks
    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    a596ead View commit details
    Browse the repository at this point in the history

  15. discojs/src/default_tasks/wikitext: use task's maxSequenceLength in t… 

    …he model init config
    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    c6beac9 View commit details
    Browse the repository at this point in the history

  16. *: replace line by line text loaders by chunk by chunk text loaders 

    discojs/src/dataset: implement and test repeat and batchWithOverlap
    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    c477bb3 View commit details
    Browse the repository at this point in the history

  17. discojs*: rename .unbatch() to .flat()

    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    cb806c0 View commit details
    Browse the repository at this point in the history

  18. discojs*,cli*: rename blockSize and maxSequenceLength to contextLength

    @JulienVig
    JulienVig committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    8cbc96e View commit details
    Browse the repository at this point in the history