Skip to content

Commit

Permalink
Make curl log level configurable by exchange
Browse files Browse the repository at this point in the history
  • Loading branch information
sjanel committed Oct 24, 2023
1 parent 906e66a commit f380a28
Show file tree
Hide file tree
Showing 31 changed files with 251 additions and 125 deletions.
2 changes: 2 additions & 0 deletions CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,8 @@ Refer to the hardcoded default json example as a model in case of doubt.
| *asset* | **preferredPaymentCurrencies** | Ordered array of coin acronyms (ex: `["USDT", "BTC"]`)) | Coins that can be used for smart buy and sell as base payment currency. They should be ordered by decreasing priority. |
| *query* | **acceptEncoding** | Comma separated list of accepted encodings (ex: `"br, gzip, deflate"`), or empty | Sets list of accepted encodings that will be passed to `curl` requests as `Accept-Encoding` header. More information [here](https://curl.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html) |
| *query* | **dustAmountsThreshold** | Unordered array of monetary amounts (ex: `["1 USDT", "0.000001 BTC"]`) | For dust sweeper option, if total balance of a currency is convertible to any of these monetary amounts and if their amount is under the threshold, it will be eligible for the selling |
| *query* | **logLevels.requestsCall** | String log level for requests call ("off", "critical", "warning", "info", etc) | Specifies the log level for this exchange requests call. It prints the full public URL and the HTTP request type (GET, POST, etc) |
| *query* | **logLevels.requestsAnswer** | String log level for requests call ("off", "critical", "warning", "info", etc) | Specifies the log level for this exchange requests replies. It prints the full answer if it is in *json* type, otherwise it will be truncated to a maximum of around 10 Ki to avoid logging too much data. |
| *query* | **dustSweeperMaxNbTrades** | Positive integer | Maximum number of trades performed by the automatic dust sweeper process. A high value may have a higher chance of successfully sell to 0 the wanted currency, at the cost of more trades (and fees) paid to the exchange. |
| *query* | **privateAPIRate** | Duration string (ex: `500ms`) | Minimum time between two consecutive requests of private account |
| *query* | **publicAPIRate** | Duration string (ex: `250ms`) | Minimum time between two consecutive requests of public account |
Expand Down
2 changes: 1 addition & 1 deletion src/api/common/src/exchangeprivateapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ void ExchangePrivate::addBalance(BalancePortfolio &balancePortfolio, MonetaryAmo
if (optConvertedAmountEquiCurrency) {
equivalentInMainCurrency = *optConvertedAmountEquiCurrency;
} else {
log::warn("Cannot convert {} into {} on {}", amount.currencyStr(), equiCurrency, exchangeName);
log::warn("Cannot convert {} -> {} on {}", amount.currencyStr(), equiCurrency, exchangeName);
equivalentInMainCurrency = MonetaryAmount(0, equiCurrency);
}
log::debug("{} Balance {} (eq. {})", exchangeName, amount, equivalentInMainCurrency);
Expand Down
2 changes: 2 additions & 0 deletions src/api/exchanges/src/binanceprivateapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,8 @@ BinancePrivate::BinancePrivate(const CoincenterInfo& coincenterInfo, BinancePubl
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(exchangeInfo().privateAPIRate())
.setAcceptedEncoding(exchangeInfo().acceptEncoding())
.setRequestCallLogLevel(exchangeInfo().requestsCallLogLevel())
.setRequestAnswerLogLevel(exchangeInfo().requestsAnswerLogLevel())
.build(),
coincenterInfo.getRunMode()),
_tradableCurrenciesCache(
Expand Down
2 changes: 2 additions & 0 deletions src/api/exchanges/src/binancepublicapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,8 @@ BinancePublic::CommonInfo::CommonInfo(const CoincenterInfo& coincenterInfo, cons
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(_exchangeInfo.publicAPIRate())
.setAcceptedEncoding(_exchangeInfo.acceptEncoding())
.setRequestCallLogLevel(_exchangeInfo.requestsCallLogLevel())
.setRequestAnswerLogLevel(_exchangeInfo.requestsAnswerLogLevel())
.build(),
runMode) {}

Expand Down
2 changes: 2 additions & 0 deletions src/api/exchanges/src/bithumbprivateapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,8 @@ BithumbPrivate::BithumbPrivate(const CoincenterInfo& config, BithumbPublic& bith
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(exchangeInfo().privateAPIRate())
.setAcceptedEncoding(exchangeInfo().acceptEncoding())
.setRequestCallLogLevel(exchangeInfo().requestsCallLogLevel())
.setRequestAnswerLogLevel(exchangeInfo().requestsAnswerLogLevel())
.build(),
config.getRunMode()),
_currencyOrderInfoRefreshTime(exchangeInfo().getAPICallUpdateFrequency(kCurrencyInfoBithumb)),
Expand Down
2 changes: 2 additions & 0 deletions src/api/exchanges/src/bithumbpublicapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,8 @@ BithumbPublic::BithumbPublic(const CoincenterInfo& config, FiatConverter& fiatCo
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(exchangeInfo().publicAPIRate())
.setAcceptedEncoding(exchangeInfo().acceptEncoding())
.setRequestCallLogLevel(exchangeInfo().requestsCallLogLevel())
.setRequestAnswerLogLevel(exchangeInfo().requestsAnswerLogLevel())
.build(),
config.getRunMode()),
_tradableCurrenciesCache(
Expand Down
2 changes: 2 additions & 0 deletions src/api/exchanges/src/huobiprivateapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,8 @@ HuobiPrivate::HuobiPrivate(const CoincenterInfo& coincenterInfo, HuobiPublic& hu
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(exchangeInfo().privateAPIRate())
.setAcceptedEncoding(exchangeInfo().acceptEncoding())
.setRequestCallLogLevel(exchangeInfo().requestsCallLogLevel())
.setRequestAnswerLogLevel(exchangeInfo().requestsAnswerLogLevel())
.build(),
coincenterInfo.getRunMode()),
_accountIdCache(CachedResultOptions(std::chrono::hours(96), _cachedResultVault), _curlHandle, apiKey),
Expand Down
2 changes: 2 additions & 0 deletions src/api/exchanges/src/huobipublicapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ HuobiPublic::HuobiPublic(const CoincenterInfo& config, FiatConverter& fiatConver
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(_exchangeInfo.publicAPIRate())
.setAcceptedEncoding(_exchangeInfo.acceptEncoding())
.setRequestCallLogLevel(_exchangeInfo.requestsCallLogLevel())
.setRequestAnswerLogLevel(_exchangeInfo.requestsAnswerLogLevel())
.build(),
config.getRunMode()),
_healthCheckCurlHandle(
Expand Down
2 changes: 2 additions & 0 deletions src/api/exchanges/src/krakenprivateapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,8 @@ KrakenPrivate::KrakenPrivate(const CoincenterInfo& config, KrakenPublic& krakenP
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(exchangeInfo().privateAPIRate())
.setAcceptedEncoding(exchangeInfo().acceptEncoding())
.setRequestCallLogLevel(exchangeInfo().requestsCallLogLevel())
.setRequestAnswerLogLevel(exchangeInfo().requestsAnswerLogLevel())
.build(),
config.getRunMode()),
_depositWalletsCache(
Expand Down
8 changes: 5 additions & 3 deletions src/api/exchanges/src/krakenpublicapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@ KrakenPublic::KrakenPublic(const CoincenterInfo& config, FiatConverter& fiatConv
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(exchangeInfo().publicAPIRate())
.setAcceptedEncoding(exchangeInfo().acceptEncoding())
.setRequestCallLogLevel(exchangeInfo().requestsCallLogLevel())
.setRequestAnswerLogLevel(exchangeInfo().requestsAnswerLogLevel())
.build(),
config.getRunMode()),
_tradableCurrenciesCache(
Expand Down Expand Up @@ -492,9 +494,9 @@ MarketOrderBook KrakenPublic::OrderBookFunc::operator()(Market mk, int count) {
}

KrakenPublic::TickerFunc::Last24hTradedVolumeAndLatestPricePair KrakenPublic::TickerFunc::operator()(Market mk) {
Market krakenMarket(_tradableCurrenciesCache.get().getOrThrow(mk.base()).altCode(),
_tradableCurrenciesCache.get().getOrThrow(mk.quote()).altCode());
json result = PublicQuery(_curlHandle, "/public/Ticker", {{"pair", krakenMarket.assetsPairStrUpper()}});
const Market krakenMarket(_tradableCurrenciesCache.get().getOrThrow(mk.base()).altCode(),
_tradableCurrenciesCache.get().getOrThrow(mk.quote()).altCode());
const json result = PublicQuery(_curlHandle, "/public/Ticker", {{"pair", krakenMarket.assetsPairStrUpper()}});
for (const auto& [krakenAssetPair, details] : result.items()) {
std::string_view last24hVol = details["v"][1].get<std::string_view>();
std::string_view lastTickerPrice = details["c"][0].get<std::string_view>();
Expand Down
2 changes: 2 additions & 0 deletions src/api/exchanges/src/kucoinprivateapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,8 @@ KucoinPrivate::KucoinPrivate(const CoincenterInfo& coincenterInfo, KucoinPublic&
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(exchangeInfo().privateAPIRate())
.setAcceptedEncoding(exchangeInfo().acceptEncoding())
.setRequestCallLogLevel(exchangeInfo().requestsCallLogLevel())
.setRequestAnswerLogLevel(exchangeInfo().requestsAnswerLogLevel())
.build(),
coincenterInfo.getRunMode()),
_depositWalletsCache(
Expand Down
2 changes: 2 additions & 0 deletions src/api/exchanges/src/kucoinpublicapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ KucoinPublic::KucoinPublic(const CoincenterInfo& config, FiatConverter& fiatConv
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(exchangeInfo().publicAPIRate())
.setAcceptedEncoding(exchangeInfo().acceptEncoding())
.setRequestCallLogLevel(exchangeInfo().requestsCallLogLevel())
.setRequestAnswerLogLevel(exchangeInfo().requestsAnswerLogLevel())
.build(),
config.getRunMode()),
_tradableCurrenciesCache(
Expand Down
2 changes: 2 additions & 0 deletions src/api/exchanges/src/upbitprivateapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,8 @@ UpbitPrivate::UpbitPrivate(const CoincenterInfo& config, UpbitPublic& upbitPubli
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(exchangeInfo().privateAPIRate())
.setAcceptedEncoding(exchangeInfo().acceptEncoding())
.setRequestCallLogLevel(exchangeInfo().requestsCallLogLevel())
.setRequestAnswerLogLevel(exchangeInfo().requestsAnswerLogLevel())
.build(),
config.getRunMode()),
_tradableCurrenciesCache(
Expand Down
2 changes: 2 additions & 0 deletions src/api/exchanges/src/upbitpublicapi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ UpbitPublic::UpbitPublic(const CoincenterInfo& config, FiatConverter& fiatConver
PermanentCurlOptions::Builder()
.setMinDurationBetweenQueries(exchangeInfo().publicAPIRate())
.setAcceptedEncoding(exchangeInfo().acceptEncoding())
.setRequestCallLogLevel(exchangeInfo().requestsCallLogLevel())
.setRequestAnswerLogLevel(exchangeInfo().requestsAnswerLogLevel())
.build(),
config.getRunMode()),
_marketsCache(CachedResultOptions(exchangeInfo().getAPICallUpdateFrequency(kMarkets), _cachedResultVault),
Expand Down
31 changes: 17 additions & 14 deletions src/engine/src/coincenterinfo_create.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -52,30 +52,33 @@ MonitoringInfo MonitoringInfo_Create(std::string_view programName, const Coincen

CoincenterInfo CoincenterInfo_Create(std::string_view programName, const CoincenterCmdLineOptions &cmdLineOptions,
settings::RunMode runMode) {
auto dataDir = cmdLineOptions.dataDir;
const auto dataDir = cmdLineOptions.dataDir;

json generalConfigData = LoadGeneralConfigAndOverrideOptionsFromCLI(cmdLineOptions);
const json generalConfigData = LoadGeneralConfigAndOverrideOptionsFromCLI(cmdLineOptions);

Duration fiatConversionQueryRate = ParseDuration(generalConfigData["fiatConversion"]["rate"].get<std::string_view>());
ApiOutputType apiOutputType = ApiOutputTypeFromString(generalConfigData["apiOutputType"].get<std::string_view>());
const Duration fiatConversionQueryRate =
ParseDuration(generalConfigData.at("fiatConversion").at("rate").get<std::string_view>());
const ApiOutputType apiOutputType =
ApiOutputTypeFromString(generalConfigData.at("apiOutputType").get<std::string_view>());

// Create LoggingInfo first as it is a RAII structure re-initializing spdlog loggers.
// It will be held by GeneralConfig and then itself by CoincenterInfo though.
LoggingInfo loggingInfo(LoggingInfo::WithLoggersCreation::kYes, dataDir,
static_cast<const json &>(generalConfigData["log"]));
const auto &logConfigJsonPart = static_cast<const json &>(generalConfigData.at("log"));
LoggingInfo loggingInfo(LoggingInfo::WithLoggersCreation::kYes, dataDir, logConfigJsonPart);

RequestsConfig requestsConfig(generalConfigData["requests"]["concurrency"]["nbMaxParallelRequests"].get<int>());
RequestsConfig requestsConfig(
generalConfigData.at("requests").at("concurrency").at("nbMaxParallelRequests").get<int>());

GeneralConfig generalConfig(std::move(loggingInfo), std::move(requestsConfig), fiatConversionQueryRate,
apiOutputType);

LoadConfiguration loadConfiguration(dataDir, LoadConfiguration::ExchangeConfigFileType::kProd);
const LoadConfiguration loadConfiguration(dataDir, LoadConfiguration::ExchangeConfigFileType::kProd);

File currencyAcronymsTranslatorFile(dataDir, File::Type::kStatic, "currencyacronymtranslator.json",
File::IfError::kThrow);
File stableCoinsFile(dataDir, File::Type::kStatic, "stablecoins.json", File::IfError::kThrow);
File currencyPrefixesTranslatorFile(dataDir, File::Type::kStatic, "currency_prefix_translator.json",
File::IfError::kThrow);
const File currencyAcronymsTranslatorFile(dataDir, File::Type::kStatic, "currencyacronymtranslator.json",
File::IfError::kThrow);
const File stableCoinsFile(dataDir, File::Type::kStatic, "stablecoins.json", File::IfError::kThrow);
const File currencyPrefixesTranslatorFile(dataDir, File::Type::kStatic, "currency_prefix_translator.json",
File::IfError::kThrow);

return CoincenterInfo(runMode, loadConfiguration, std::move(generalConfig),
MonitoringInfo_Create(programName, cmdLineOptions), currencyAcronymsTranslatorFile,
Expand All @@ -84,7 +87,7 @@ CoincenterInfo CoincenterInfo_Create(std::string_view programName, const Coincen

ExchangeSecretsInfo ExchangeSecretsInfo_Create(const CoincenterCmdLineOptions &cmdLineOptions) {
if (cmdLineOptions.noSecrets) {
StringOptionParser anyParser(*cmdLineOptions.noSecrets);
const StringOptionParser anyParser(*cmdLineOptions.noSecrets);

return ExchangeSecretsInfo(anyParser.getExchanges());
}
Expand Down
2 changes: 2 additions & 0 deletions src/http-request/include/curlhandle.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ class CurlHandle {
TimePoint _lastQueryTime{};
BestURLPicker _bestUrlPicker;
string _queryData;
log::level::level_enum _requestCallLogLevel;
log::level::level_enum _requestAnswerLogLevel;
};

// Simple RAII class managing global init and clean up of Curl library.
Expand Down
24 changes: 23 additions & 1 deletion src/http-request/include/permanentcurloptions.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

#include <string_view>

#include "cct_log.hpp"
#include "cct_string.hpp"
#include "timedef.hpp"

Expand All @@ -19,6 +20,9 @@ class PermanentCurlOptions {

bool followLocation() const { return _followLocation; }

log::level::level_enum requestCallLogLevel() const { return _requestCallLogLevel; }
log::level::level_enum requestAnswerLogLevel() const { return _requestAnswerLogLevel; }

class Builder {
public:
Builder() noexcept = default;
Expand Down Expand Up @@ -58,33 +62,51 @@ class PermanentCurlOptions {
return *this;
}

Builder &setRequestCallLogLevel(log::level::level_enum requestCallLogLevel) {
_requestCallLogLevel = requestCallLogLevel;
return *this;
}

Builder &setRequestAnswerLogLevel(log::level::level_enum requestAnswerLogLevel) {
_requestAnswerLogLevel = requestAnswerLogLevel;
return *this;
}

Builder &setFollowLocation() {
_followLocation = true;
return *this;
}

PermanentCurlOptions build() {
return {std::move(_userAgent), std::move(_acceptedEncoding), _minDurationBetweenQueries, _followLocation};
return {std::move(_userAgent), std::move(_acceptedEncoding), _minDurationBetweenQueries,
_requestCallLogLevel, _requestAnswerLogLevel, _followLocation};
}

private:
string _userAgent;
string _acceptedEncoding;
Duration _minDurationBetweenQueries{};
log::level::level_enum _requestCallLogLevel = log::level::level_enum::info;
log::level::level_enum _requestAnswerLogLevel = log::level::level_enum::trace;
bool _followLocation = false;
};

private:
PermanentCurlOptions(string userAgent, string acceptedEncoding, Duration minDurationBetweenQueries,
log::level::level_enum requestCallLogLevel, log::level::level_enum requestAnswerLogLevel,
bool followLocation)
: _userAgent(std::move(userAgent)),
_acceptedEncoding(std::move(acceptedEncoding)),
_minDurationBetweenQueries(minDurationBetweenQueries),
_requestCallLogLevel(requestCallLogLevel),
_requestAnswerLogLevel(requestAnswerLogLevel),
_followLocation(followLocation) {}

string _userAgent;
string _acceptedEncoding;
Duration _minDurationBetweenQueries;
log::level::level_enum _requestCallLogLevel;
log::level::level_enum _requestAnswerLogLevel;
bool _followLocation;
};

Expand Down
Loading

0 comments on commit f380a28

Please sign in to comment.