refcounted_he_(new|fetch)_pvn: Don't roll-own code #22638

Outdent and reflow some comments and code in preparation for them to be moved out of the loop

This is for clarity. All this very-unlikely-to-be-used code was in the middle of what is really going on, creating a distraction.

The previous version did not make sure that it wasn't reading beyond the end of the buffer in all cases, and the first pass through the input string already ruled out it having most problems. Thus we don't need the full generality here of the macro UTF8_IS_DOWNGRADEABLE_START; and this simplifies things

These were misleading. On ASCII platforms, many calls to this function won't use the per-word algorithm. That's only done for long-enough strings.

The new name, s0, is used in more other places for this meaning, and is more descriptive.

This is an internal function, designed to be an extension of utf8_to_bytes(), with a slightly different API. This commit just adds it and calls it from just utf8_to_bytes. Future commits will extend this API.

This variable should not be being changed by the function

The argument is currently unused. The macro is a public facing API that calls this function with the correct argument

This makes the next commit smaller

This causes this function to be able to both overwrite the input, and to instead create new memory. It changes bytes_from_utf8() to use this new capability instead of being a near duplication of the core code of this function. Prior to this commit, bytes_from_utf8() just allocated memory the size of the original string, and started copying into it. When it came to a sequence that wasn't convertible, it stopped, and freed up the copy. The new behavior has it checking first before the malloc that the string is convertible. That has the advantage that there is no malloc without being sure it will be useful; but the disadvantage that there is an extra pass through the input string, but that pass is per-word. The next commit will introduce another advantage. Thanks to Tony Cook for the 'free_me' idea

Prior to this commit, the size malloced was just the same as the length of the input string, which is a worst case scenario. This commit changes so the new pass through the input (introduced in the previous commit) also calculates the needed length. The additional cost of doing this is minimal. It has advantages on a very long string with lots of sequences that are convertible.

This is a non-destructive conversion of the input into native bytes, and with any new memory required set for destruction via SAVEFREEPV. This allows the caller to not have to be concerned at all if memory was created or not. A new macro is created that calls this internal function with the correct parameter to force this behavior.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refcounted_he_(new|fetch)_pvn: Don't roll-own code #22638

refcounted_he_(new|fetch)_pvn: Don't roll-own code #22638

Commits on Nov 28, 2024