Add initial support for Late Chunking #97
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…hods - Updated class docstring for clarity. - Added new parameters: `min_characters_per_sentence` and `chunk_size` to the constructor. - Implemented `_create_token_chunks`, `_token_chunk`, and `_split_sentences` methods for better chunking functionality. - Refactored chunking logic to support both token and sentence modes. - Introduced embedding handling for chunk embeddings. These changes improve the flexibility and performance of the LateChunker for various text processing tasks.
- Added `embed_as_tokens` method to obtain token embeddings for individual texts, accommodating longer texts than the maximum sequence length. - Introduced `embed_as_tokens_batch` method for batch processing of token embeddings. - Updated the class to import numpy and added a property for `max_seq_length` to improve usability and flexibility in embedding operations. These changes enhance the functionality of the SentenceTransformerEmbeddings class for token-level embedding tasks.
- Introduced an `approximate` parameter to control token count estimation. - Added `_prepare_sentences` method for improved sentence handling and token counting. - Refactored `_sentence_chunk` method to utilize new sentence preparation logic. - Implemented `_create_sentence_chunk` for better chunk creation from sentences. - Updated error handling for embedding model compatibility. These changes improve the accuracy and efficiency of the LateChunker for processing text into meaningful chunks.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces several enhancements and new features to the Chonkie library, particularly focusing on the addition of the experimental
LateChunkerand related functionalities. The most important changes include updates to theREADME.md, thesentence_transformer.pyandtypes.pyfiles, and the addition of new test cases for theLateChunker.Enhancements and New Features:
Documentation Update:
README.md: Added a description for the newLateChunker (experimental)which embeds text and splits it to improve chunk embeddings.Embedding Enhancements:
src/chonkie/embeddings/sentence_transformer.py: Addedembed_as_tokensandembed_as_tokens_batchmethods to embed text as tokens, and introduced amax_seq_lengthproperty. Also, importednumpyasnpto handle embeddings. [1] [2] [3]New Data Classes:
src/chonkie/types.py: AddedLateSentenceandLateChunkdataclasses to represent sentences and chunks with embeddings.Testing:
tests/chunker/test_late_chunker.py: Added comprehensive test cases for theLateChunker, including initialization, mode validation, chunking functionality, handling of empty text, single sentence text, sentence boundaries, and embedding dimensions.