semi-endless loop around add_candle call #447
Comments
|
@movy please pull the new master version and try it out. can you also tell me. what tf your example was? the number of routes and the simulation start / end time? |
|
Thanks for prompt response, I import candles for ~6 months, then split them into 40 days periods and get candles for each period. Basic code: for index, date_range in enumerate(backtest_dates, start=0):
print(f"Importing candles for {date_range[0]} - {date_range[1]}...")
try:
candles_warmup, candles_period = research.get_candles(exchange, symbol, tf, int((datetime.strptime(date_range[0], '%Y-%m-%d')).timestamp() * 1000), int((datetime.strptime(date_range[1], '%Y-%m-%d')).timestamp() * 1000), warmup_candles_num=config["warm_up_candles"], caching=True, is_for_jesse=True)
period_candles = {
jh.key(exchange, symbol): {
'exchange': exchange,
'symbol': symbol,
# backtest candles tf is always 1m
'candles': candles_period
}
}
warmup_candles = {
jh.key(exchange, symbol): {
'exchange': exchange,
'symbol': symbol,
'candles': candles_warmup
}
}
candles_refs.append(ray.put(period_candles))
# print(candles_refs)
warmup_candles_refs.append(ray.put(warmup_candles))
[...]
for candles_ref in candles_refs:
result = backtest(config=ray.get(config_ref),
routes=ray.get(routes_refs[params["tf"]]),
extra_routes=ray.get(extra_routes_refs[anchor_timeframe(params["tf"])]),
candles=ray.get(candles_ref), # candles within backtest date range
warmup_candles=ray.get(warmup_candles_refs[candles_refs.index(candles_ref)]),
generate_csv=False,
generate_json=True,
hyperparameters=params,
fast_mode=False)Will try master branch now. |
|
the higher the timeframe you use the better theperformance of fast_mose. |
|
Hey Movy, I pushed the patch that Yakir is talking about. It is version 0.48.1. Please give it a try and see if it fixes your issue. |
|
So, good news and bad news: this regression was not caused by 0.48 (i.e. 0.48.1. did not fix it). The last version that does not hang is 0.45, so I'm sticking to it for now. I guess some change related to p.s. inconsistent params names 'warm_up_candles' vs 'warmup_candles' gave me some avoidable pain while I was refactoring |
|
@movy |
|
Yes, I cancel on every candle if order was not executed. This never was a problem prior to 0.46. |
|
Thank you, @yakir4123 , I tried your branch, unfort same result. I will try to create a small script that consistently reproduces problematic behaviour, will post it here later. |
|
@movy , Yeah im facing the same problemim trying to solve it. I know the root of the problem. I guess i need to go back to v0.46 to see what was there |
|
0.45 is the final version working w/o hanging, 0.46 introduced this bug |
|
@movy https://github.com/yakir4123/jesse/tree/yakir/feat/delete-orders |
|
@yakir4123 , still hangs unfort. |
|
@movy |
|
Last commit I tested was 5ed48de |
|
:/ I wonder what is it, please give me a simple strategy that repreduce that ill probably find the problem |
|
Here's one of Jesse's default strategies. I added trailing take-profit and offset-entries (i.e. entry not at current price, but at Tested on multiple coins and 4h candles for speed, same result. I run tests using Ray.tune (to enable more complex hyperparams perturbations and schedulers), but this particular code should run with jesse out of the box. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Hello, |
|
Ill profile your strategy this week, @movy if you can provide symbol, timeframe and timestapms that would help |
|
I believe any symbol / tf will cause looping, but you can try ALICE-USDT, HIGH-USDT, ASTR-USDT 4h at Binance Perp Futures |
|
@movy |
|
Okay, I think I found the culprit, it was a mistake in my code (thanks to Python's archaic scoping rules) when migrating to updated backtest function, but finding this mistake was so hard due to difference between Consider the following generic backtest: import importlib
from datetime import datetime
import jesse.helpers as jh
from jesse.research import backtest, import_candles, get_candles
from jesse.utils import anchor_timeframe
exchange = 'Binance Perpetual Futures'
symbol = 'ACH-USDT'
tf = '1h'
strategy_name = 'DUAL_THRUST'
strategy = getattr(importlib.import_module(f"strategies.{strategy_name}"), f"{strategy_name}")
timeframes = [tf]
config = {
'starting_balance': 1000,
'fee': 0.001,
'type': 'futures',
'futures_leverage': 1,
'futures_leverage_mode': 'cross',
'exchange': exchange,
'warm_up_candles': 255,
}
tests_start_date = '2024-01-01'
tests_end_date = '2024-08-30'
import_candles(exchange, symbol, tests_start_date, show_progressbar=False)
warmup_candles, candles = get_candles(exchange, symbol, tf, int((datetime.strptime(tests_start_date, '%Y-%m-%d')).timestamp() * 1000), int((datetime.strptime(tests_end_date, '%Y-%m-%d')).timestamp() * 1000), warmup_candles_num=config["warm_up_candles"], caching=True, is_for_jesse=True)
wrong_tests_start_date = '2024-05-01' # !!! IMPORTING WRONG CANDLES !!!
wrong_warmup_candles, wrong_candles = get_candles(exchange, symbol, tf, int((datetime.strptime(wrong_tests_start_date, '%Y-%m-%d')).timestamp() * 1000), int((datetime.strptime(tests_end_date, '%Y-%m-%d')).timestamp() * 1000), warmup_candles_num=config["warm_up_candles"], caching=True, is_for_jesse=True)
params={'stop_loss_atr_rate': 0.8, 'down_length': 30, 'up_length': 24, 'down_coeff': 1.8000000000000003, 'up_coeff': 0.5, 'tp_pc': 5.0, 'sl_pc': 0.5, 'trailing_profit': 5.0}
result = backtest(config,
[ {'exchange': exchange, 'strategy': strategy, 'symbol': symbol, 'timeframe': tf} ],
[ {'exchange': exchange, 'strategy': strategy, 'symbol': symbol, 'timeframe': anchor_timeframe(tf)} ],
candles = {
jh.key(exchange, symbol): {
'exchange': exchange,
'symbol': symbol,
'candles': candles,
}
},
warmup_candles = {
jh.key(exchange, symbol): {
'exchange': exchange,
'symbol': symbol,
'candles': wrong_warmup_candles, # !!! USING WRONG CANDLES !!!
}
},
generate_csv=False,
generate_json=True,
hyperparameters=params,
fast_mode=False
)
print(result)As you can see, here I deliberately offer With This error message is expected with I believe, this difference in handling of |
|
I have run a number of backtests using v0.46 and v1.1.1, and there're a slight, but important differences in tests outcome, even though resulting charts look similar to human eye (but not identical). In addition, I found that with v1.1.1 (or any version above v0.46 I assume) results get really weird when importing 1-minute candles to use with different timeframes, i.e. With v0.46 and below I always imported 1m candles for any timeframe and backtest results matched live trading quite well, that was why I liked Jesse so much -- good backtests gave me confidence to expect similar results in live trading. With new versions 1m candles produce absurd backtests (with fast and slow modes alike), and timeframe candles (i.e. 1h, 2h etc) produce slightly inaccurate results ( It might look insignificant for basic strategies, single symbol or short perioda, but with longer periods, portfolio of symbols and complex entry/exit rules such backtests lead to false conclusions. I am really at loss here, as I was very happy after finally figuring the problem with new backtests, but now I am quite frustrated with these regressions. Any particular change around v.47+ could have caused such behaviour? I think it's related to how Jesse handles Below are comparisons between Jesse versions, I used my private strategy here (not Donchian), but I'm sure any strategy with LIMIT entries or trailing stops will have similar issues. 1.1.1 slow mode: 
1.1.1 fast mode: 
0.46: 
|
|
@movy about the differences between the versions, i have found few very minor and rare bugs, so it might be this issue. (is your strategy has a lot of entries / exit orders concurrently?) |
|
To clarify: I use 1h routes, but in versions prior 0.47 we always had to request 1m candles via I think, fast_mode is very useful for many use-cases, but slow_mode should fall back to whatever logic used prior 0.47. In my strategies I use only one entry, but build a "ladder" of exits similar to [(qty / 10, entry_price * (1 + self.tp_pc / 100)), *[(qty / 10, exit_price * (1 + self.trailing_pc / 100 * i)) for i in range(1,10)]] |
|
im not sure i understand, because fast-mode introduced in 0.49, and in 0.48 there was an update on how to iterate on closed trades (both on fast and slow mode (because there wasn't fast- mode in that time)) lets talk on discord to find the root of this issue |
Describe the bug
Until this week I'd been running 0.44 w/o any problems, but 0.48 added some welcome changes, so I switched my backtesting to 0.48 and encountered a very long loop around https://github.com/jesse-ai/jesse/blob/master/jesse/store/state_candles.py#L102 call, which sometimes causes a backtest to hang for 10-15 minutes. This happens sporadically, maybe one out of 100-200 backtests run, but it effectively it blocks the whole pipeline until problematic test is finished with this loop.
This happens on all symbols and tfs.
When I say 'around' it means impossible to track down exact line that hangs, but what I could gather via py-spy is:
I've added some logging:
And from these
prints we can see this method is called millions of times by a single backtest process:Eventually number of calls falls:
Which makes me think there's loop inside loop somewhere.
I could not instantly pinpoint recent changes responsible for this. Will roll back to 0.44 for now and slowly work my way to 0.48, maybe I can find where regression first appeared.
Any help from those familiar with recent changes is appreciated @yakir4123
The text was updated successfully, but these errors were encountered: