Skip to content

Latest commit

 

History

History
328 lines (267 loc) · 16.5 KB

README.md

File metadata and controls

328 lines (267 loc) · 16.5 KB
Raw

alt text alt text alt text alt text

vanilla-rtb

Real Time Bidding (RTB) - Demand Side Platform framework

open-source library utilizing modern C++11/14/17 features and latest Boost.

What makes us different from other open-source RTB projects on GitHub :

  • Our stack is fairly small and easy to integrate with any cmake project.
  • Partitioned targeting data
  • Monolithic bidder
  • Code generation for targeting matchers and bidder executable
  • Minimum dependency on outside vendors.
  • Decoupled by C++ templates
  • Very high throughput up to 105K QPS on 16 cores Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
  • GUI for editing campaign budget

As a model cmake project please visit https://github.com/vanilla-rtb/rapid-bidder

We provide VanillaRTB extensions including bindings to popular languages NodeJS/Go/Java/PHP/Python
and custom targetings and bidder executable generators https://github.com/vanilla-rtb/extensions

vanilla-rtb ecosystem

Multi-bidder-model-with-communicator-for-Win-notifications

best performance compared to other stacks - 105K QPS

runs on cloud with docker - see instructions

Join the chat at https://gitter.im/vanilla-rtb/Lobby build GPLv3 license Installing Dependencies contributions welcome Beerpay Backers on Open Collective Sponsors on Open Collective

Recommended build environment: Linux or macOS, CMake >= 3.9.2, GCC >= 7.0, Clang >= 5.0 , Boost >= 1.67

Structure :

  • / -- the root directory

    • benchmarks/ -- optionaly built benchmarks for IPC caches, json parsers and low overhead IPC audit logger
    • CRUD/ -- C++11 high performance HTTP-restful handlers based on boost.ASIO and CRUD API
    • docker/ -- vanilla docker files and instructions on how to build and run
    • jsonv/ -- DSL mapper of json encoded objects to C++ structures
    • parsers/ -- fast zero copy, zero memory allocation parsers
    • rapidjson/ -- very fast json tokenizer including SAX / DOM API but requires more memory
    • rtb/ -- RTB framework
      • core/ -- generic structures shared in the project ( RTB specific )
      • common/ -- generic RTB agnostic structures
      • datacache/ -- IPC cache generic classes for fast targeting and other lookups
      • exchange/ -- exchange handlers implementation
      • DSL/ -- DSL formats for jsonv , boost::any and rapidjson
    • examples/ -- root to our sandbox with examples
      • bidder/ -- collection of application specific classes to support targeting
      • bidder_experimental/ -- bidder based on chained matchers model
      • loader/ -- collection of application specific classes to support campaign loading
      • campaign/ -- add/modify/delete campaign API + UI ( work in progress )
      • datacache/ -- IPC cache implementation based on rtb/datacache model
      • UI/ -- HTML and javascript for campaign budget management

  • [CMakeLists.txt] - cmake file

The stack of vanilla-rtb includes other C++11/14 projects and is referencing them via gh-subree. To update to the latest version of json-voorhees or CRUD we use the following commands :

  • git subtree pull --prefix jsonv git@github.com:tgockel/json-voorhees.git master --squash
  • git subtree pull --prefix CRUD git@github.com:venediktov/CRUD.git master --squash

Creating bidder in 49 lines of code

using Selector = vanilla::ad_selector<vanilla::BudgetManager, Ad>;
using DSLT = DSL::GenericDSL<std::string, DSL::rapid_mapper> ;

//Return from each lambda becomes input for next lambda in the tuple of functions
auto retrieve_domain_f = [&cacheLoader](const std::string& dom, auto&& ...) {
    Domain domain;
    if(!cacheLoader.retrieve(domain,dom)) {
        return boost::optional<uint32_t>();
    }
    return boost::optional<uint32_t>(domain.dom_id);
};

auto retrieve_ico_campaign_f = [&cacheLoader](boost::optional<uint32_t> dom_id, auto&& ...)  {
    std::vector<ICOCampaign> ico_campains;
    if (!cacheLoader.retrieve(ico_campains,*dom_id)) {
        return boost::optional<decltype(ico_campains)>();
    }
    return boost::optional<decltype(ico_campains)>(ico_campains);
};

vanilla::core::Banker<BudgetManager> banker;
auto retrieve_campaign_ads_f = [&](boost::optional<std::vector<ICOCampaign>> campaigns, auto && req, auto && imp)  {
    std::vector<Ad> retrieved_cached_ads;
    for (auto &campaign : *campaigns) {
        if (!cacheLoader.retrieve(retrieved_cached_ads, campaign.campaign_id, imp.banner.get().w, imp.banner.get().h)) {
            continue;
        }
        auto budget_bid = banker.authorize(cacheLoader.get<CampaignCache<BidderConfig>>(), campaign.campaign_id);
        std::transform(std::begin(retrieved_cached_ads),
                       std::end(retrieved_cached_ads),
                       std::begin(retrieved_cached_ads), [budget_bid](Ad & ad){
                          ad.auth_bid_micros = std::min(budget_bid, ad.max_bid_micros);
                          return ad;
                       });
    }
    return retrieved_cached_ads;
};

//creating bidder endpoint utilizing self-referencing pattern
exchange_handler<DSLT> bid_handler(std::chrono::milliseconds(10));
bid_handler
.logger([](const std::string &data) {
    LOG(debug) << "bid request=" << data ;
})
.error_logger([](const std::string &data) {
    LOG(debug) << "bid request error " << data ;
})
.auction_async([&](const BidRequest &request) {
    thread_local vanilla::Bidder<DSLT, Selector> bidder(std::move(Selector()));
    return bidder.bid(request,
                      request.site.get().ref,
                      //chained matchers lambdas defined above
                      retrieve_domain_f,
                      retrieve_ico_campaign_f,
                      retrieve_campaign_ads_f
   );
});

(📗) To build vanilla-rtb use following commands in the root of vanilla-rtb

(installing dependencies before building vanilla stack)

Linux :

$ mkdir Release
$ cd Release
$ cmake -DCMAKE_BUILD_TYPE=Release .. -G "Unix Makefiles"
$ make -j4 install
####### Creating  Debug build #######
$ cd ..
$ mkdir Debug
$ cd Debug
$ cmake -DCMAKE_BUILD_TYPE=Debug .. -G "Unix Makefiles"
$ make -j4 install

Windows :

same steps as above for linux , only difference is depending on your environment either Visual Studio or NMake project can be used

######### for NMake ####################
cd Release
cmake -DCMAKE_BUILD_TYPE=Release .. -G "NMake Makefiles"
cd ../Debug
cmake -DCMAKE_BUILD_TYPE=Debug   .. -G "NMake Makefiles"
######### for Visual Studio ############
cd Release
cmake -DCMAKE_BUILD_TYPE=Release .. -G "Visual Studio 14 2015"
cd ../Debug
cmake -DCMAKE_BUILD_TYPE=Debug   .. -G "Visual Studio 14 2015"

Mac OS X (Xcode) :

For the reliable results it is suggested to have the build directory out of source tree. The process involves creating a build directory, generating an Xcode project in that directory with CMake, opening the project file generated in the build directory with Xcode, and lastly, adjusting project settings as requried and kicking off the build.

To generate an Xcode project invoke cmake from an empty build directory with command line similar to cmake -G Xcode -DCMAKE_BUILD_TYPE=Release.

Mac OS X (command line tools) :

$ xcode-select --install
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew doctor
$ brew install cmake
$ brew install boost
$ mkdir Release
$ cd Release
$ cmake -DCMAKE_BUILD_TYPE=Release .. -G "Unix Makefiles"
$ make -j4 install

Mac OS X ( with llvm ) :

$ brew doctor
$ brew install cmake
$ brew install boost
$ brew install --with-clang llvm
$ mkdir Release
$ cd Release
$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/usr/local/opt/llvm/bin/clang -DCMAKE_CXX_COMPILER=/usr/local/opt/llvm/bin/clang++ -DCMAKE_RANLIB=/usr/local/opt/llvm/bin/llvm-ranlib -DCMAKE_AR=/usr/local/opt/llvm/bin/llvm-ar .. -G "Unix Makefiles"

Parallel Builds

When building on Linux and Mac OS X with Make it's possible to automatically adjust the concurreny of the build using nproc for Linux and sysctl -n hw.physicalcpu for Mac OS X command line tools that returns number of CPUs available to the Make execution context:

Linux :

$ make -j $(nproc) ...

Mac OS X :

make -j $(sysctl -n hw.physicalcpu) ...

It's also possible to specifying the target Load Average with -l flag to prevent machine overloading:

$ make -l $(nproc) ...

And lastly, on Linux, it's possible to run the build with the BATCH scheduling mode (throughput oriented) as:

$ chrt --batch 0 make ...

All above considered the ultimate Make invocation combo on Linux would be something like:

$ chrt --batch 0 make -j$(nproc) -l$(nproc)

Running examples:

Testing simple case - single bidder bound to ip/port

  • HTTP-Bidder
    • vanilla-rtb/Release/examples/bin$ ./http_bidder_test --config etc/config.cfg
  • test with curl and apache benchmark
    • vanilla-rtb/Release/examples/bin$ ./curl.sh --bidder
    • vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --bidder

Testing different json mappers to openrtb::BidRequest available in vanilla-rtb stack

  • Start Exchange Handler with HTTP handler
    • vanilla-rtb/Release/examples/bin$ ./exchange_handler_test --config etc/config.cfg
    • vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --auction
    • vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --auction-any
    • vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --auction-rapid

Testing mock-bidders e.g. no bid multi bidders via communicator pattern ( work in progress )

  • Start Exchange Handler distributing to multi-bidders via communicator
    • vanilla-rtb/Release/examples/bin$ ./exchange_handler_test --config etc/config.cfg
  • Mock-bidders starting multiple in one swoop, currently configured as 5 bidders in config
    • vanilla-rtb/Release/examples/bin$ ./mock_bidder_test --config etc/config.cfg
    • vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --mock-bidders

Testing multi bidders via communicator pattern ( work in progress )

  • multi-bidders starting multiple in one swoop (currently configured as 3 bidders in config ) and starting exchange handler
    • vanilla-rtb/Release/examples/bin$ ./multi_bidder --config etc/config.cfg
    • vanilla-rtb/Release/examples/bin$ ./multi_exchange_handler --config etc/config.cfg

Testing cache loader of ad campaigns and ad budgets ( needs to be extended to get cache update events from outside )

  • Cache loader
    • vanilla-rtb/Release/examples/bin$ ./cache_loader_test --config etc/config.cfg

Testing Mock exchange writen in python

  • Mock exchange - emulating bid requests
    • To run mock exchange you need any python, and python "requests" library installed.
    • for simple exchange please run
    • vanilla-rtb/Release/examples/bin/mock_exchange$ python mock-x.py
    • for various geo and size rotation please run
    • vanilla-rtb/Release/examples/bin/mock_exchange$ python mock-x.py --geo "Russia:Moscow USA:NY USA:Washington USA:Chicago" --size '100:300 240:400 420:280'
    • for more info please run
    • vanilla-rtb/Release/examples/bin/mock_exchange$ python mock-x.py --help

Testing campaign manager paired with Win-notification service

  • Campaign manager - Budget

    • vanilla-rtb/Release/examples/bin$ ./campaign_manager_test --config etc/config.cfg

  • Notification service and Slave-Banker

    • vanilla-rtb/Release/examples/bin$ ./notification_service_test --config etc/config.cfg
    • vanilla-rtb/Release/examples/bin$ ./slavebanker_service_test --config etc/config.cfg

  • To add/delete/modify campign budgets fire up UI by connecting to manager via browser http:://localhost:11081/campaign/index.html

    campaign

Support on Beerpay

We hope you like our project , fork our repo or chip in for couple of 🍻!

Beerpay Beerpay

Contributors

This project exists thanks to all the people who contribute. [Contribute].

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]