-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integrate X-LoRA #1491
Merged
Merged
Integrate X-LoRA #1491
Changes from 182 commits
Commits
Show all changes
190 commits
Select commit
Hold shift + click to select a range
9aeb69c
Initial commit of integration
EricLBuehler b9d3878
Pass a reference back to the peftmodel when creating
EricLBuehler bbd3ad4
Check base model in from_pretrained
EricLBuehler ea53917
Fix assert, attr
EricLBuehler 79c4df9
Fix inheritance from peft model
EricLBuehler 2ee88af
Update inheritance again
EricLBuehler 19063dc
Update inheritance again and properly instantiate
EricLBuehler d364e5b
Update comment
EricLBuehler c9ab310
Export config, model
EricLBuehler 9ed665e
Remove use of default attr
EricLBuehler ae32006
Remove use of default attr
EricLBuehler 9bfe8a6
Update imports
EricLBuehler 048958f
Update imports
EricLBuehler 024ce15
Update imports
EricLBuehler 3d08fc9
Work on circular import
EricLBuehler 7e5332b
Work on circular import again
EricLBuehler d5193f4
Remove another circular import
EricLBuehler 5d7ff64
Remove another circular import
EricLBuehler b41089d
Slightly refactor
EricLBuehler bfd794f
Update with EricLBuehler/xlora#20
EricLBuehler 726f8a8
Make signature an exact copy
EricLBuehler 8a1c68c
Refactor util fns
EricLBuehler 1a34615
Update the typing structure
EricLBuehler 1f0e03a
Pass super() the dict
EricLBuehler c326560
Add capability to disable default injection
EricLBuehler c9dab76
Set base model for loading adapter
EricLBuehler 980b2c8
Set peft type
EricLBuehler c3f4ed9
Default adapter name
EricLBuehler effaca0
Ensure all nonzero length
EricLBuehler 031fabb
Fix get nb trainable params
EricLBuehler 9a7943e
Add some docs to tuner class
EricLBuehler deeb169
Update post init of xlora config
EricLBuehler 1e73f3d
Mention method swapping
EricLBuehler 73901e8
Remove incorrect example
EricLBuehler 279036d
Update to use default args for compat
EricLBuehler 644a562
Move API for nice visibility
EricLBuehler 6e3dc3d
Remove use of __slots__
EricLBuehler 012ed6f
Remove passing of classifier
EricLBuehler 510247d
Remove passing of classifier
EricLBuehler f3e3d1f
Remove many asserts, converting to exceptions
EricLBuehler a24f51b
Update naming convention
EricLBuehler 358ea19
Refactor Lora layers toreduce code repetion
EricLBuehler e6126fa
Remove _disable_inject hack
EricLBuehler de76531
Remove passing peftmodel to super
EricLBuehler 1dace56
Use post init fn to improve separation of concerns
EricLBuehler e11e433
Update naming and fix call
EricLBuehler 63debdb
Remove 'n_predictions_lifetime'
EricLBuehler 226ab92
Move generate to XLoraModel, simplifying PeftModelWrapper
EricLBuehler feba4f6
Simplify save pretrained and from pretrained
EricLBuehler fd0ab4b
Do not save to nested dir
EricLBuehler c44cd5a
Use new _save_pretrained_hook to reduce code duplication
EricLBuehler 1f36837
Add explaining comment
EricLBuehler d2d3b6a
Call .clear on log scalings
EricLBuehler 9a7032b
Seperate method use cases
EricLBuehler f4a5314
Prefix with _
EricLBuehler cf4fa9d
Remove get, print trainable params as they are redundant
EricLBuehler 862b880
Fix inclusion of old kwarg
EricLBuehler 9c5e3be
Remove circular imports
EricLBuehler 34694d5
Override method
EricLBuehler 18bd1b8
Set target modules to None
EricLBuehler 306bb34
Do not set target modules to None
EricLBuehler b8121ac
Try to avoid checking target_modules
EricLBuehler 44cdc07
Remove circular import
EricLBuehler ba905a6
Override another method
EricLBuehler d5446f6
Override _check_target_module_exists
EricLBuehler 2e7f51a
Make method instance
EricLBuehler 7c354d9
Make a nicer check for having target_modules
EricLBuehler 76b26b5
Avoid unnecessary injection
EricLBuehler 5828ab4
Remove xlora conf from mark only adapters as trainable
EricLBuehler 9b56c2a
Account for property
EricLBuehler 0ea1fa2
Call correct method
EricLBuehler 8f67a6b
Fix for scoping
EricLBuehler a4823fe
Set active adapter to not 'default'
EricLBuehler 1bafcd8
Fix recursion err
EricLBuehler b5f5563
Index into config to check
EricLBuehler e6b45c3
Get and pass scalings
EricLBuehler b8ba669
Remove deprecated attr
EricLBuehler 31b121c
Remove the xloralayer for a hook, refactoring changes
EricLBuehler df6867d
Add copywright notice
EricLBuehler 9025310
Remove some typing things
EricLBuehler 506e96e
Remove some typing things
EricLBuehler 07ff71f
Add note to docstring
EricLBuehler a111787
Add to config
EricLBuehler 15b7bec
Update for new saving
EricLBuehler 0ef9181
Fix topk impl
EricLBuehler 3804d6a
Update based on comments
EricLBuehler ea3ea8f
Add the xlora layer structure back in
EricLBuehler 1ce8391
Add the xlora layer structure back in
EricLBuehler b48d4af
Make some style changes
EricLBuehler 4d48c59
Fix some bugs
EricLBuehler 93069bb
Remove base model id as unnecessary
EricLBuehler 3ee9350
Format with ruff
EricLBuehler 7c1bcfa
Add xlora test
EricLBuehler e627525
Handle lack of target modules in lora method
EricLBuehler aa189c6
Handle lack of layer_replication in lora method
EricLBuehler 2df1d5f
Handle lack of target_modules in check target modules exists
EricLBuehler b14c335
Handle lack of target_modules in inject_adapter
EricLBuehler 8bec326
Fix the recursion err
EricLBuehler c79b37c
Formatting
EricLBuehler e453747
Do not include device in config
EricLBuehler 10161b3
Cleaner method to eliminate adapters
EricLBuehler f223168
Cleaner method to eliminate adapters
EricLBuehler 31078a6
Remove unnecessary methods
EricLBuehler d52f6ce
Error on multiple xlora adapters
EricLBuehler 7e3c0d0
Error on use of dora
EricLBuehler 1edeb96
Raise errors and other misc fixes
EricLBuehler bb5e546
Depend on use trainable adapters
EricLBuehler a4479b6
Implement another test and restructure the model
EricLBuehler 81b069b
Inherit
EricLBuehler 33c2623
Impl some abstract methods and remove the special cases
EricLBuehler 63f3418
Fix return
EricLBuehler 31863fa
Make some progress
EricLBuehler 34f7594
Make some progress
EricLBuehler a234676
Merge branch 'main' into add_xlora
EricLBuehler 7468114
Working version and add some tests
EricLBuehler e4053bc
Explain need for refreezing
EricLBuehler cc0cfee
Fix some ruff lints
EricLBuehler 9debbe3
Get adapter names and handle gracefully with subfolders
EricLBuehler a283ea8
Fix unused
EricLBuehler aca6044
Rewrite the flush_log_scalings function with new functionality
EricLBuehler d10f4ee
Use some fixtures
EricLBuehler 677c64e
Fix passing arg
EricLBuehler 93db8bc
Remove unnecessary checks
EricLBuehler 99fff5c
Fix test
EricLBuehler 42db87c
Fix a bug
EricLBuehler 6787b9f
More sensible defaults
EricLBuehler 6fe86d1
Cast
EricLBuehler f948883
Refactor forward pass
EricLBuehler 83ecd37
Finish refactoring of model forward
EricLBuehler ba4093a
Remove some dead code
EricLBuehler e19cbc6
Run formatting
EricLBuehler cf54d6a
Add test for saving and loading
EricLBuehler 750e7ac
Fix determining adapter device for embedding
EricLBuehler a7ac01c
Fix the save and load
EricLBuehler 7fd76ca
Fix the save and load
EricLBuehler dc8bd3b
Comment
EricLBuehler b888a69
Merge remote-tracking branch 'upstream/main' into add_xlora
EricLBuehler c379f70
Update the docstrings
EricLBuehler eaf5052
Remove the hacking
EricLBuehler 76070d6
Merge
EricLBuehler a510560
Remove the post_init_lora
EricLBuehler c9339a6
Fix
EricLBuehler a6d92b0
More tests
EricLBuehler 261b336
Remove custom load and save code
EricLBuehler 763bf7f
Merge remote-tracking branch 'upstream/main' into add_xlora
EricLBuehler 3c07d18
Improve tests
EricLBuehler 44a3bb9
Remove InhibitorFlagPayload
EricLBuehler b9c80ef
Improve the docstrings
EricLBuehler 72f9610
Fix the docstrings and clean up a bit
EricLBuehler f4b2df9
Remove some redundant config options
EricLBuehler a8fbf85
Remove redundant case
EricLBuehler 3fb0cd0
Better docstrings
EricLBuehler 5e62d66
Remove *et_use_trainable_adapters
EricLBuehler 8faaa0f
Remove method and clean up checks
EricLBuehler 6b54244
Minor fixes
EricLBuehler 0d7f9d3
Remove custom loading and saving code
EricLBuehler 8c58e1c
Update tests
EricLBuehler 430f9e4
Remove monkey patching for scalings passing
EricLBuehler 53f0342
Add tests for disabling adapters
EricLBuehler ee19910
Add test for embedding model
EricLBuehler 1735deb
Use peft forward hook
EricLBuehler d661266
Fix disable adapter test
EricLBuehler 8601832
Remove unnecessary var
EricLBuehler a3b83c1
Add futures annotations
EricLBuehler e3db8df
Merge remote-tracking branch 'upstream/main' into add_xlora
EricLBuehler b0f3062
Fix changes to lora prepare adapter config
EricLBuehler 81e337d
Fix tests
EricLBuehler 7ddd14b
Fix hacks
EricLBuehler 7ea1a12
Use infer_device
EricLBuehler 7058c15
Fix the tests and provide fake default values
EricLBuehler 587c78a
Fix a mistake
EricLBuehler 2b1193a
Merge remote-tracking branch 'upstream/main' into add_xlora
EricLBuehler 6a809cb
Fix .weight
EricLBuehler 164af91
Format
EricLBuehler 096927a
Merge remote-tracking branch 'upstream/main' into add_xlora
EricLBuehler fadc5d9
Embedding does not support DoRA
EricLBuehler 013a084
Remove xlora for embedding layers
EricLBuehler 14efafa
Change order of scalings application
EricLBuehler 624d316
Add tests
EricLBuehler 1505f31
Handle case when inserting
EricLBuehler 0b2dcd0
Check exact type
EricLBuehler abdcb50
Fix target modules
EricLBuehler aecd492
Somehow it didn't get formatted
EricLBuehler 68e5f2f
Make tmp dir and tokenizer function scoped
EricLBuehler 1628227
Scope the lora adapters too
EricLBuehler 5a889be
Use unique temp dirs for lora adapters, all tests
EricLBuehler 072987e
Seperation of concerns for xlora and lora
EricLBuehler 13edc61
Merge remote-tracking branch 'upstream/main' into add_xlora
EricLBuehler 77fb6b0
Prevent inf recursion as per 1892
EricLBuehler 2f85f91
Prevent inf recursion on lora_model
EricLBuehler File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# Copyright 2023-present the HuggingFace Inc. team. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
from .config import XLoraConfig | ||
from .model import XLoraModel | ||
|
||
|
||
__all__ = ["XLoraConfig", "XLoraModel"] |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the recent refactor of
XLoraModel
, can all the changes toLoraModel
be reverted?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, those cannot because during init those functions are called.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I found a way to allow us to remove all these changes to
lora/model.py
. The main issue is the following: DuringXLoraModel.__init__
, we want to create theLoraModel
instance based on theXLoraConfig
. ThisLoraModel
is not supposed to contain any actual LoRA adapter, as those will come later through theconfig.adapters
. So what we need is to implement a way to create this "empty"LoraModel
without needing to changeLoraModel
itself. Here is my idea:First, let's make a change to
XLoraModel.__init__
by creating a copy of theXLoraConfig
that "imitates" a normalLoraConfig
:So we set the required attributes so that we no longer need the extra checks in
LoraModel
that this PR adds. Also, we have addedDUMMY_TARGET_MODULES
. What is this? It's a constant defined inconstants.py
asDUMMY_TARGET_MODULES = "dummy-target-modules"
. This is a special value that allows us to create an "empty"LoraModel
. For this to work, we also need a small change inBaseTuner
:I tested this locally and the tests pass after reverting all changes to
lora/model.py
.I think this is the more elegant solution, because it should not be necessary for
LoraModel
to know about XLoRA.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That sounds good, I'll add those 2 changes - this means I can revert all changes to lora/model.py to
main
, right?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exactly, that's the idea. You'd also have to add
DUMMY_TARGET_MODULES
toutils/constants.py
and add a few imports, the rest should work.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great! I pushed those changes now, and all the tests pass locally.