Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrating fast_float with Valkey to replace strtod invocation #1260

Open
wants to merge 56 commits into
base: unstable
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
42c1d0c
feat (deps): adds fast_float library and a c wrapper for from_chars f…
swaingotnochill Oct 15, 2024
d25e29b
Changes to complete the integration of fast_float into Redis codebase.
parthpatel Nov 5, 2024
939a766
Fixing code-style issue due to empty space
parthpatel Nov 5, 2024
c2d6ff6
Adding 32-bit c++ libraries to library path to fix 32-bit compilation…
parthpatel Nov 5, 2024
597af83
Trying another approach to fix the CI workflows by installing require…
parthpatel Nov 5, 2024
8d07687
Adding -v to linker for more output to debug issues.
parthpatel Nov 5, 2024
4f03150
32-bit stdc++ libraries are in a different folder, so needed to speci…
parthpatel Nov 5, 2024
e9e5209
Also fixing the library to be non-cross version of libstdc++
parthpatel Nov 5, 2024
d7c72cf
This attempt uses the cross-compile libraries provided by ubuntu for …
parthpatel Nov 5, 2024
d69ec21
Forwarding CFLAGS and LDFLAGS to fast_float Makefile to honor 32 bit …
parthpatel Nov 5, 2024
50abf50
Putting LDFLAGS in quotes to escape spaces and /
parthpatel Nov 5, 2024
c0ec94f
More fixes
parthpatel Nov 6, 2024
6900219
Removing unnecessary library path declaration
parthpatel Nov 6, 2024
63a2f42
Including all the headers from fast_float project under a directory f…
parthpatel Nov 6, 2024
ba9c99a
Adding back the code to set errno when conversion fails but reducing …
parthpatel Nov 6, 2024
1471233
Update deps/fast_float/Makefile
parthpatel Nov 7, 2024
87654b0
Update src/Makefile
parthpatel Nov 7, 2024
9f1b352
Replacing strtod in util.c file as well.
parthpatel Nov 7, 2024
3048954
Made fast_float an optional library for build. Turning it off by defa…
parthpatel Nov 11, 2024
0941aec
Cleaning up README and adding license information
parthpatel Nov 11, 2024
a92d636
Fixing formatting issues as per Clang-format output
parthpatel Nov 11, 2024
adb46cd
Fixing more formatting issues
parthpatel Nov 11, 2024
ce38546
Move CXX to c++ instead of g++
parthpatel Nov 11, 2024
ac3af73
Refactoring to simplify things
parthpatel Nov 11, 2024
ad0f785
Moving the fast_float integration code into src/fast_float
parthpatel Nov 12, 2024
ad93430
Consolidating all ifdefs into fast_float integration header by using …
parthpatel Nov 12, 2024
8357b03
Forgot to remove ifdefs around header files. This commit fixes it.
parthpatel Nov 12, 2024
f7efd98
Taking care of warnings
parthpatel Nov 12, 2024
87f979f
Fixing more CI related issues. Making build work with both fast_float…
parthpatel Nov 12, 2024
5d3268f
Fixed formatting issues from clang-format
parthpatel Nov 12, 2024
5bb9606
Merge branch 'valkey-io:unstable' into unstable
parthpatel Nov 12, 2024
05e12e9
Update src/Makefile
parthpatel Nov 12, 2024
d66bd37
Update src/fast_float/fast_float_strtod.h
parthpatel Nov 12, 2024
7280cb3
Consolidating all fast_float code into a single if block inside Makefile
parthpatel Nov 13, 2024
c99a80c
Combining Makefile and moving unit test to src/unit directory.
parthpatel Nov 13, 2024
ee1d369
Fixed unit tests and build
parthpatel Nov 13, 2024
f42e9c6
Add QUIET_CXX definition in Makefile.
parthpatel Nov 13, 2024
5be910d
Moving the valkey_strtod.* files back to deps folder.
parthpatel Nov 13, 2024
acb10e6
Merge branch 'valkey-io:unstable' into unstable
parthpatel Nov 14, 2024
8c1ab28
Update deps/Makefile
madolson Nov 14, 2024
2050389
Update deps/README.md
parthpatel Nov 14, 2024
836bfdb
Fixing Makefiles and CI workflows
parthpatel Nov 14, 2024
9450ebf
Fixing a few naming and compilation issues.
parthpatel Nov 15, 2024
9d681ac
Merge branch 'valkey-io:unstable' into unstable
parthpatel Nov 16, 2024
a69568f
Applying yamlfmt suggestions
parthpatel Nov 16, 2024
2cb45b5
Fixing 32-bit build by setting appropriate variables in fast_float_c_…
parthpatel Nov 16, 2024
129fdbf
Fixing Makefile related issues failing CI
parthpatel Nov 16, 2024
43696b5
Remove some unnecessary imports and use SPDX for license
madolson Nov 18, 2024
c8c3571
Revising as per feedback
parthpatel Nov 19, 2024
97ef78c
Fixing clang-format errors
parthpatel Nov 19, 2024
91edab2
Madelyn's opinion about workflows
madolson Nov 22, 2024
dab5913
Fixed whitespace
madolson Nov 22, 2024
41250e9
Remove exceptions to allow building with out c++ stdlib
madolson Nov 22, 2024
bc235ce
Remove extra build step
madolson Nov 22, 2024
0cfcc7f
Remove fast float from daily, seems to need more work to build
madolson Nov 22, 2024
3f7cee8
Don't set errno = 0 in fast_float_strtod
zuiderkwast Nov 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,24 @@ jobs:
run: |
./src/valkey-unit-tests

test-ubuntu-latest-fast-float:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: make
# Fail build if there are warnings
# build with TLS and FAST_FLOAT just for compilation coverage
run: make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_FAST_FLOAT=yes
- name: test
run: |
sudo apt-get install tcl8.6 tclx
./runtest --verbose --tags -slow --dump-logs
- name: module api test
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs
- name: unit tests
run: |
./src/valkey-unit-tests

test-ubuntu-latest-cmake:
runs-on: ubuntu-latest
steps:
Expand Down Expand Up @@ -110,6 +128,13 @@ jobs:
- name: make
run: make -j3 all-with-unit-tests SERVER_CFLAGS='-Werror'

build-macos-latest-fast-float:
madolson marked this conversation as resolved.
Show resolved Hide resolved
runs-on: macos-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: make
run: make -j3 all-with-unit-tests SERVER_CFLAGS='-Werror' USE_FAST_FLOAT=yes

build-32bit:
runs-on: ubuntu-latest
steps:
Expand All @@ -119,6 +144,13 @@ jobs:
sudo apt-get update && sudo apt-get install libc6-dev-i386
make -j4 SERVER_CFLAGS='-Werror' 32bit

build-32bit-fast-float:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: make
run: "# Fast float requires C++ 32-bit libraries to compile on 64-bit ubuntu machine i.e. \"-cross\" suffixed version. \n# Cross-compiling c++ to 32-bit also requires multilib support for g++ compiler i.e. \"-multilib\" suffixed version of g++.\n# g++-multilib should generally include libstdc++.*cross version as well, but also added explicitly just in case. \nsudo apt-get update && sudo apt-get install libc6-dev-i386 libstdc++-11-dev-i386-cross gcc-multilib g++-multilib\nmake -j4 SERVER_CFLAGS='-Werror' 32bit USE_FAST_FLOAT=yes\n#magic___^_^___line\n"
zuiderkwast marked this conversation as resolved.
Show resolved Hide resolved

build-libc-malloc:
runs-on: ubuntu-latest
steps:
Expand All @@ -137,6 +169,17 @@ jobs:
dnf -y install epel-release gcc make procps-ng which
make -j4 SERVER_CFLAGS='-Werror'

build-almalinux8-jemalloc-fast-float:
runs-on: ubuntu-latest
container: almalinux:8
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

- name: make
run: |
dnf -y install epel-release gcc gcc-c++ make procps-ng which
make -j4 SERVER_CFLAGS='-Werror' USE_FAST_FLOAT=yes

format-yaml:
runs-on: ubuntu-latest
steps:
Expand Down
7 changes: 7 additions & 0 deletions deps/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ distclean:
-(cd jemalloc && [ -f Makefile ] && $(MAKE) distclean) > /dev/null || true
-(cd hdr_histogram && $(MAKE) clean) > /dev/null || true
-(cd fpconv && $(MAKE) clean) > /dev/null || true
-(cd fast_float_c_interface && $(MAKE) clean) > /dev/null || true
-(rm -f .make-*)

.PHONY: distclean
Expand Down Expand Up @@ -116,3 +117,9 @@ jemalloc: .make-prerequisites
cd jemalloc && $(MAKE) lib/libjemalloc.a

.PHONY: jemalloc

fast_float_c_interface: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
cd fast_float_c_interface && $(MAKE)

.PHONY: fast_float_c_interface
15 changes: 15 additions & 0 deletions deps/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ should be provided by the operating system.
* **linenoise** is a readline replacement. It is developed by the same authors of Valkey but is managed as a separated project and updated as needed.
* **lua** is Lua 5.1 with minor changes for security and additional libraries.
* **hdr_histogram** Used for per-command latency tracking histograms.
* **fast_float** is a replacement for strtod to convert strings to floats efficiently.

How to upgrade the above dependencies
===
Expand Down Expand Up @@ -105,3 +106,17 @@ We use a customized version based on master branch commit e4448cf6d1cd08fff51981
2. Copy updated files from newer version onto files in /hdr_histogram.
3. Apply the changes from 1 above to the updated files.

fast_float
---
The fast_float library provides fast header-only implementations for the C++ from_chars functions for `float` and `double` types as well as integer types. These functions convert ASCII strings representing decimal values (e.g., `1.3e10`) into binary types. The functions are much faster than comparable number-parsing functions from existing C++ standard libraries.

Specifically, `fast_float` provides the following function to parse floating-point numbers with a C++17-like syntax (the library itself only requires C++11):

template <typename T, typename UC = char, typename = FASTFLOAT_ENABLE_IF(is_supported_float_type<T>())>
from_chars_result_t<UC> from_chars(UC const *first, UC const *last, T &value, chars_format fmt = chars_format::general);

To upgrade the library,
1. Check out https://github.com/fastfloat/fast_float/tree/main
2. cd fast_float
3. Invoke "python3 ./script/amalgamate.py --output fast_float.h"
4. Copy fast_float.h file to "deps/fast_float/".
Loading
Loading