Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] F2 KV store #922

Open
wants to merge 179 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
179 commits
Select commit Hold shift + click to select a range
9db72bd
[C++] [WIP] Hot/cold design (#508)
kkanellis Aug 6, 2021
c647508
Rename testcases
kkanellis Oct 14, 2021
a842dd9
Initial implementation of conditional insert
kkanellis Oct 14, 2021
5a30fb8
Cleanup
kkanellis Oct 16, 2021
19ad421
Fix hot-cold Read-Modify-Write operation
kkanellis Oct 17, 2021
ddd4a6d
Replace hot-cold compaction login with Upserts
kkanellis Oct 18, 2021
d0a605d
Minor fix
kkanellis Oct 20, 2021
3669a14
Add support for variable-length keys in hot-cold
kkanellis Oct 27, 2021
7ee21e3
Minor changes
kkanellis Oct 28, 2021
76e9552
Add RMW ops in variable length key tests
kkanellis Oct 28, 2021
68f7bfc
Bugfix on hot-cold read operation
kkanellis Jan 9, 2022
fb23158
Add test that exposes Reads-Compaction race condition
kkanellis Jan 11, 2022
504d332
Fix for Read-Compaction race condition
kkanellis Jan 11, 2022
b07bf92
hot-cold: Add test for variable-length value records
kkanellis Jan 11, 2022
bc0dbcb
Bugfix in CleanHashTableBuckets
kkanellis Jan 11, 2022
52d8420
Fix wrong assertion, for when RMW entry is deleted
kkanellis Mar 13, 2022
9d90648
Working prototype -- suboptimal HC context management
kkanellis Mar 14, 2022
d80d7cf
Better context management memory-wise
kkanellis Mar 15, 2022
416d67e
Bugfixes caused by ops inside assertions
kkanellis Mar 17, 2022
2e69abe
HC RMW operation: proper memory free-ing for contexts
kkanellis Mar 17, 2022
6d059ee
Minor fix in hot-cold test
kkanellis Mar 18, 2022
83e019c
Modify compaction test to use variable number of threads
kkanellis Mar 18, 2022
1a7cf1d
Bugfix
kkanellis Mar 30, 2022
1ad3bc9
Bugfix on log scan
kkanellis Mar 30, 2022
be1f82b
Support for hybrid log checkpointing after compaction
kkanellis Mar 30, 2022
6209ae3
Add tests for checkpointing after compaction
kkanellis Mar 30, 2022
36b9693
[WIP] Decoupling index from log
kkanellis Apr 2, 2022
d05b84b
Move index checkpointing to hash index class
kkanellis Apr 7, 2022
d8f5079
Fix error from gtest
kkanellis Apr 9, 2022
2fc04bc
Minor changes
kkanellis Apr 9, 2022
7aacb5d
[WIP] Support for index ops that go pending
kkanellis Apr 9, 2022
9914e46
[WIP] More work towards decoupling
kkanellis Apr 12, 2022
5adb91e
Add HashIndex class template to FasterKv
kkanellis Apr 13, 2022
72f6575
Minor cleanup
kkanellis Apr 13, 2022
8deed96
[WIP] Common interface for index classes
kkanellis Apr 13, 2022
3afd50d
Update FasterKvHC to accomodate index changes
kkanellis Apr 13, 2022
1ff138a
[WIP] More work to decouple index-related structures
kkanellis Apr 18, 2022
86eb58f
Minor fixes in tests
kkanellis Apr 18, 2022
6cc1cd7
Use overflow bucket entries only in hot index
kkanellis Apr 19, 2022
68e1f9a
Make root_path accessible from outside of class
kkanellis Jun 21, 2022
779c643
Expand hash index interface with FASTER helper methods
kkanellis Jun 21, 2022
596e096
Parameterize Hot/Cold hash index bucket bit structure
kkanellis Jun 21, 2022
09a3a21
Auxiliary methods in key hash for cold-index ops
kkanellis Jun 21, 2022
58d4ee0
Contexts for index-related I/O operations
kkanellis Jun 21, 2022
77e9307
Store address of newly-created record in async index context
kkanellis Jun 22, 2022
2b0945b
Initial implementation of cold index
kkanellis Jun 22, 2022
4eaffd8
Initial integration of cold-index to FASTER
kkanellis Jun 22, 2022
305c23a
Cold index improvements
kkanellis Jun 27, 2022
5887d25
Add cold index test
kkanellis Jun 27, 2022
c90f30b
Simplify some FASTER contexts
kkanellis Jun 28, 2022
e4c0d37
Integrate cold index to new FASTER design
kkanellis Sep 24, 2022
f188f91
Add/fix minor checks
kkanellis Sep 24, 2022
fda1562
Modify bitfields to accomodate read-cache integration
kkanellis Sep 24, 2022
0ec1fee
Initial read cache implementation
kkanellis Sep 24, 2022
a935c18
Integrate read cache with FASTER
kkanellis Sep 24, 2022
3c57a95
Several read-cache bugfixes
kkanellis Sep 25, 2022
c490c90
In-mem & paging test now check multiple FASTER configs
kkanellis Sep 25, 2022
82df01b
Support compaction with Read Cache
kkanellis Sep 26, 2022
2254c23
Minor bugfixes
kkanellis Sep 26, 2022
88431e6
Improve debug logging
kkanellis Oct 2, 2022
2e3bbad
Bugfix on in-mem hash index
kkanellis Oct 2, 2022
b3e9521
Multiple fixes in read-cache implementation
kkanellis Oct 2, 2022
facd7bb
Read cache bugfix when concurrent compaction & read
kkanellis Oct 2, 2022
e7fd748
Minor bugfixes
kkanellis Oct 4, 2022
00d5ec8
Include thread ID to log messages
kkanellis Oct 4, 2022
65a8e23
Bugfixes in Read Cache eviction
kkanellis Oct 4, 2022
c50904a
Better support for epoch framework on two logs
kkanellis Oct 4, 2022
e65b101
Support concurrent hash index GC with read-cache GC
kkanellis Oct 7, 2022
962579e
Rework ConditionalInsert op for hot-cold design
kkanellis Oct 7, 2022
8cc8877
Fix one memory leak issue in file I/O buffer pool
kkanellis Oct 8, 2022
2ef02e6
Minor fixes
kkanellis Oct 9, 2022
588b835
Move read-cache block allocate to FasterKv
kkanellis Oct 10, 2022
0e6547a
Support for inserting cold log records to read cache
kkanellis Oct 10, 2022
53c3c12
Add WARN level in log message module
kkanellis Oct 10, 2022
87fac73
Working epoch framework sync on hot-cold design
kkanellis Oct 10, 2022
2e748be
Fix bug in FasterIndex when chunk has been updated
kkanellis Oct 16, 2022
4a56fd1
Fix bug where epoch framework made no progress
kkanellis Oct 18, 2022
c268af7
Add config structs
kkanellis Oct 18, 2022
ac8368c
Implement single-log auto-compaction
kkanellis Oct 18, 2022
5fe8249
Fix bug in cold index
kkanellis Oct 21, 2022
89786a9
Fix in cold-index garbage collect algorithm
kkanellis Oct 24, 2022
2884fde
Minor cold-index fixes
kkanellis Oct 24, 2022
bc17715
Minor read cache fix
kkanellis Oct 24, 2022
07a220d
Minor changes & cleanup old code
kkanellis Nov 1, 2022
b3e98e0
Make key hash function a class
kkanellis Nov 1, 2022
ceab7c9
Update & modernized tests
kkanellis Nov 1, 2022
0241ced
Update hot-cold compaction inner-workings
kkanellis Nov 1, 2022
f94776c
Implement unsafe buffer resize method
kkanellis Nov 8, 2022
cbd19b0
Minor bugfix in Read Cache eviction
kkanellis Nov 9, 2022
f20ce1a
Exception is now thrown if savedir not present
kkanellis Nov 9, 2022
648b2e5
Fix hot-cold compaction memory leaks
kkanellis Nov 14, 2022
8d5adcf
Minor bugfix in cold-index
kkanellis Nov 14, 2022
8565b28
Add write counts when measuring I/O statistics
kkanellis Nov 14, 2022
57d54d1
Call user-provided callback if I/O request failed
kkanellis Nov 18, 2022
a554545
Minor var name changes
kkanellis Nov 20, 2022
0e73433
Multi-thread eviction in ReadCache
kkanellis Dec 22, 2022
ca26bf1
Implement FASTER-level actions statistics capturing
kkanellis Dec 22, 2022
b29d495
Avoid unecessary IOps when min address is specified
kkanellis Dec 22, 2022
2963e19
Bugfix in cold-index key hashing
kkanellis Jan 11, 2023
558fbe8
Add method to check when compaction is/will be active
kkanellis Jan 11, 2023
6f5b165
UnsafeBufferResize bugfixes
kkanellis Jan 18, 2023
713ad19
Implement concurrent hybrid log scan iterator
kkanellis Feb 1, 2023
48e8f22
Minor cleanup
kkanellis Mar 11, 2023
fd6cd59
Better logging in cold-index
kkanellis Mar 16, 2023
61327ac
Copy read-hot records to read-cache tail
kkanellis Mar 16, 2023
e0d137c
Cold-index: changed from 4kB chunks to 512B ones
kkanellis Mar 16, 2023
42831ff
Better hash index statistics
kkanellis Mar 25, 2023
91dea53
Minor organizational changes in hash index
kkanellis Mar 25, 2023
3c26603
Minor changes in config
kkanellis May 1, 2023
8bd8476
Minor bugfix in read-cache
kkanellis May 1, 2023
1c74011
Cold index hash chunk to 256B. More options in comments.
kkanellis May 1, 2023
2a0698f
Enable user threads to participate to compaction
kkanellis May 1, 2023
edfca4e
Update hot cold test
kkanellis May 1, 2023
3ab87ef
Add benchmark code for YCSB
kkanellis May 1, 2023
149cc1e
Add README.md
kkanellis May 1, 2023
eacf866
Bump up standard to C++17
kkanellis Jul 4, 2024
7144093
Update CMake gtest installation
kkanellis Jul 4, 2024
65e97c0
TOML support for FasterKv/FasterKvHC configuration
kkanellis Jul 4, 2024
54ac379
Configurable cold-index w/ number of index entries
kkanellis Jul 4, 2024
1624d93
Informative messages when initializing hlog
kkanellis Jul 4, 2024
8d77fd2
Helpers to view checkpointing/recovery metadata
kkanellis Jul 10, 2024
f0cec5c
Implement cold-index checkpoint & recovery
kkanellis Jul 10, 2024
b147f41
Better handling of file paths
kkanellis Jul 10, 2024
f2082ac
Add cold-index checkpointing/recovery tests
kkanellis Jul 10, 2024
00a7def
Bugfix when creating/normalizing dirs
kkanellis Jul 16, 2024
034604a
Support FasterKv checkpoint with read-cache
kkanellis Jul 16, 2024
2aa99a7
CompactWithLookup: replace static contexts with proper ones
kkanellis Jul 16, 2024
ed7581a
Checkpoint/Recovery for FasterKvHC (HotCold store)
kkanellis Jul 16, 2024
545f2bc
Checkponting/Recovery tests for hot-cold store
kkanellis Jul 16, 2024
1dfe37b
Minor fixes in tests
kkanellis Jul 16, 2024
e5ec73c
ShiftBeginAddress: add support for context in callbacks
kkanellis Jul 17, 2024
314916f
Checkpoint: minor fix on callback context behavior
kkanellis Jul 17, 2024
b71207e
Minor fixes
kkanellis Jul 17, 2024
29d6095
Minor fix
kkanellis Jul 17, 2024
e5eed6c
Minor changes in CMake files
kkanellis Jul 17, 2024
49c39cf
Minor cleanup
kkanellis Sep 4, 2024
d6aff4a
Add missing copyright header
kkanellis Sep 4, 2024
314f031
Fix F2 performance regression on in-memory tests
kkanellis Sep 15, 2024
31582c9
Fit pending context struct in a single cache-line
kkanellis Sep 15, 2024
6fc9ae9
Minor changes
kkanellis Sep 15, 2024
1c33250
Simplify hash function usage
kkanellis Sep 16, 2024
bac8551
Support for ops-based benchmarking
kkanellis Sep 16, 2024
2bb75bc
Support more benchmarking workloads
kkanellis Sep 16, 2024
fbc5873
F2: Add hash index class as template arguments
kkanellis Sep 16, 2024
4a58734
Add blocking method that waits for log compaction completion.
kkanellis Sep 16, 2024
2ba9337
F2: Add benchmarking code
kkanellis Sep 16, 2024
ab7eaee
Minor cleanup
kkanellis Sep 19, 2024
253b089
Add missing pragma for header file
kkanellis Sep 19, 2024
8dcaca4
Minor fix in log messages
kkanellis Sep 19, 2024
cda0b40
Minor index class reordering
kkanellis Sep 19, 2024
e2cbe3d
F2: Changes in classes naming
kkanellis Sep 19, 2024
5090cc8
Naming improvements for index classes
kkanellis Sep 19, 2024
ea51a9c
Remove deprecated benchmark files
kkanellis Sep 20, 2024
16a959f
Provide config defaults for FASTER hlog compaction
kkanellis Sep 26, 2024
43dc93b
Throttle threads when max hlog budget has been reached
kkanellis Sep 26, 2024
771534b
Minor change
kkanellis Oct 25, 2024
a62b930
Bugfix
kkanellis Oct 25, 2024
94cbf90
Set I/O queue size in a single place
kkanellis Oct 26, 2024
95beaf1
Add report (i.e., always print) logging level
kkanellis Nov 22, 2024
73b8e86
Bugfix on thread throttling during compaction
kkanellis Nov 22, 2024
d596937
Add some warnings for return values
kkanellis Nov 24, 2024
a135487
Fix class member reordering warnings
kkanellis Nov 24, 2024
bd08e85
Fix warnings wrt pointer casting
kkanellis Nov 24, 2024
ad1a1f1
Remove unused variables
kkanellis Nov 25, 2024
d043d87
String repr for Status enum
kkanellis Nov 25, 2024
95916f9
Fix int types due to warnings
kkanellis Nov 25, 2024
c183e6f
Suppress warnings on unused typedefs.
kkanellis Nov 25, 2024
21c3eb7
Minor changes
kkanellis Nov 25, 2024
16d8bcc
Minor structural change
kkanellis Nov 25, 2024
227d592
Bugfixes
kkanellis Nov 25, 2024
788c7de
Fix warnings on tests
kkanellis Nov 25, 2024
e791571
Minor naming changes
kkanellis Dec 14, 2024
4891630
Enhance C++ FasterKv, F2Kv documentation
kkanellis Dec 14, 2024
d9fcb56
Fix minor compilation
kkanellis Dec 14, 2024
8e6b3d2
Bump toml11 version to 4.2.0
kkanellis Dec 14, 2024
109f076
Remove unecessary debug messages
kkanellis Dec 14, 2024
b30c945
Merge branch 'main' into cc-lmhc-v2
kkanellis Dec 15, 2024
33263c7
Fix compilation error
kkanellis Dec 15, 2024
125e8b3
Add (unsafe) index compaction method
kkanellis Dec 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 29 additions & 32 deletions cc/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
cmake_minimum_required (VERSION 3.2.2)
cmake_minimum_required (VERSION 3.14)

set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

enable_testing()

include(ExternalProject)
include(FetchContent)
project(FASTER)

# The set of additional options that we can pass into cmake. We currently support
Expand All @@ -23,10 +25,10 @@ if (MSVC)
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} /DEBUG /OPT:NOICF")
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} /DEBUG /OPT:NOICF")
else()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++17")

set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -O0 -g -D_DEBUG")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -O3 -g")
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -O0 -g -D_DEBUG -DTOML_CONFIG")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -O3 -g -DTOML_CONFIG")

if (USE_URING)
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -DFASTER_URING")
Expand All @@ -38,34 +40,27 @@ endif()
set_directory_properties(PROPERTIES COMPILE_DEFINITIONS_DEBUG "_DEBUG")

##### BEGIN GOOGLE TEST INSTALLATION #####
# Copied from https://github.com/google/googletest/tree/master/googletest#incorporating-into-an-existing-cmake-project
# Download and unpack googletest at configure time
configure_file(CMakeLists.txt.in googletest-download/CMakeLists.txt)
execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}" .
RESULT_VARIABLE result
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/googletest-download)
if(result)
message(FATAL_ERROR "CMake step for googletest failed: ${result}")
endif()
execute_process(COMMAND ${CMAKE_COMMAND} --build .
RESULT_VARIABLE result
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/googletest-download )
if(result)
message(FATAL_ERROR "Build step for googletest failed: ${result}")
endif()

# Prevent overriding the parent project's compiler/linker
# settings on Windows
FetchContent_Declare(
googletest
GIT_REPOSITORY https://github.com/google/googletest.git
GIT_TAG v1.14.0
)
set(BUILD_GMOCK OFF)
# For Windows: Prevent overriding the parent project's compiler/linker settings
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)

# Add googletest directly to our build. This defines
# the gtest and gtest_main targets.
add_subdirectory(${CMAKE_BINARY_DIR}/googletest-src
${CMAKE_BINARY_DIR}/googletest-build
EXCLUDE_FROM_ALL)

FetchContent_MakeAvailable(googletest)
##### END GOOGLE TEST INSTALLATION #####

#### BEGIN TOML11 INSTALLATION #####
FetchContent_Declare(
toml11
GIT_REPOSITORY https://github.com/ToruNiina/toml11
GIT_TAG v4.2.0
)
FetchContent_MakeAvailable(toml11)
include_directories(${toml11_SOURCE_DIR}/include)
#### END TOML11 INSTALLATION #####

include_directories(${CMAKE_SOURCE_DIR}/src)

# Set the directory targets when build in libs and binaries
Expand All @@ -78,7 +73,7 @@ set (FASTER_LINK_LIBS
)

# Set the link libraries to for test compilation
set (FASTER_TEST_LINK_LIBS ${FASTER_LINK_LIBS} gtest)
set (FASTER_TEST_LINK_LIBS ${FASTER_LINK_LIBS} gtest_main)
if(WIN32)
set(FASTER_TEST_LINK_LIBS ${FASTER_TEST_LINK_LIBS} rpcrt4)
else()
Expand Down Expand Up @@ -108,6 +103,7 @@ FUNCTION(ADD_FASTER_TEST TEST_NAME HEADERS)
add_executable(${TEST_NAME} ${HEADERS} ${TEST_NAME}.cc)

target_link_libraries(${TEST_NAME} ${FASTER_TEST_LINK_LIBS})
target_include_directories(${TEST_NAME} PRIVATE ${toml11_SOURCE_DIR})
add_test(${TEST_NAME} ${CMAKE_BINARY_DIR}/${TEST_NAME})
ENDFUNCTION()

Expand All @@ -116,6 +112,7 @@ FUNCTION(ADD_FASTER_BENCHMARK BENCHMARK_NAME)
add_executable(${BENCHMARK_NAME} ${BENCHMARK_HEADERS} ${BENCHMARK_NAME}.cc)

target_link_libraries(${BENCHMARK_NAME} ${FASTER_BENCHMARK_LINK_LIBS})
target_include_directories(${BENCHMARK_NAME} PRIVATE ${toml11_SOURCE_DIR})
ENDFUNCTION()

# Build each subdirectory
Expand Down
3 changes: 2 additions & 1 deletion cc/benchmark-dir/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
set(BENCHMARK_HEADERS
benchmark.h
file.h
)

set(BENCHMARK_SOURCES
)

ADD_FASTER_BENCHMARK(benchmark)

ADD_FASTER_BENCHMARK(benchmark_f2)
add_executable(process_ycsb process_ycsb.cc)
Loading