Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ORC-1614: Use directOut.put(out) instead of directOut.put(out.array()) in TestBrotli test #1790

Closed
wants to merge 2 commits into from

Conversation

cxzl25
Copy link
Contributor

@cxzl25 cxzl25 commented Feb 7, 2024

What changes were proposed in this pull request?

Set ByteBuffer limit in TestBrotli test

Why are the changes needed?

TestBrotli#testDirectDecompress attempts to put the compressed result in direct ByteBuffer.
When calling decompress, we will find that the input buffer length is still 10000 not 217 since we put the array without limit on the length.

      directOut.put(out.array());
      directOut.flip();

This PR is aimed to use directOut.put(out) instead of directOut.put(out.array())

How was this patch tested?

GA

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions bot added the JAVA label Feb 7, 2024
out.flip();
// copy heap buffer to direct buffer.
directOut.put(out.array());
directOut.flip();
directOut.limit(position);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be I didn't understand the issue correctly, but from documentation of flip(), limit is set by it.

"Flips this buffer. The limit is set to the current position and then the position is set to zero"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

out is compressed by brotliCodec, and the length after compression is 217, but the out.array() used here is still the 10000 length of out, so the length that can be read after directOut.flip() is 10000, not 217

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Thanks !

@deshanxiao
Copy link
Contributor

Good catch, Thanks!

In fact, I made a wrong decision to directly put the array into directOut since I misjudged the length of the array.

I prefer to change directOut.put(out.array()); to directOut.put(out);

What do you think? @cxzl25

@dongjoon-hyun
Copy link
Member

  • Could you update the PR title and description accordingly, @cxzl25 ?
  • Could you review this once more, @deshanxiao ?

@deshanxiao deshanxiao changed the title ORC-1614: Set ByteBuffer limit in TestBrotli test ORC-1614: Use directOut.put(out) instead of directOut.put(out.array()) in TestBrotli test Feb 10, 2024
@deshanxiao
Copy link
Contributor

Thanks @dongjoon-hyun , I have updated the PR title and description.

@deshanxiao
Copy link
Contributor

deshanxiao commented Feb 10, 2024

Merged to main/2.1. Thank you, @cxzl25 @paliwalashish @dongjoon-hyun !

@dongjoon-hyun
Copy link
Member

Great! Thank you all!

@dongjoon-hyun dongjoon-hyun added this to the 2.0.0 milestone Feb 10, 2024
@dongjoon-hyun
Copy link
Member

Hi, @deshanxiao . branch-2.0 didn't have this patch. Did you use this merge script?

Screenshot 2024-02-09 at 19 29 40

@dongjoon-hyun
Copy link
Member

Please note that main is Apache ORC 2.1.0.

@deshanxiao
Copy link
Contributor

Thanks @dongjoon-hyun , I have changed the comment and jira.

@dongjoon-hyun
Copy link
Member

Oh, we had better backport this to branch-2.0, didn't we?

@deshanxiao deshanxiao reopened this Feb 10, 2024
deshanxiao pushed a commit that referenced this pull request Feb 10, 2024
…y())` in `TestBrotli` test

### What changes were proposed in this pull request?
Set ByteBuffer limit in `TestBrotli` test

### Why are the changes needed?
`TestBrotli#testDirectDecompress` attempts to put the compressed result in direct ByteBuffer.
When calling `decompress`, we will find that the input buffer length is still `10000` not `217` since we put the array without `limit` on the length.

```java
      directOut.put(out.array());
      directOut.flip();
```

This PR is aimed to use `directOut.put(out)` instead of `directOut.put(out.array())`

### How was this patch tested?
GA

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #1790 from cxzl25/ORC-1614.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: deshanxiao <deshanxiao@microsoft.com>
@deshanxiao
Copy link
Contributor

Thanks @dongjoon-hyun Cherry-pick the fix to branch-2.0

@deshanxiao deshanxiao closed this Feb 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants