diff --git a/docs/data.rst b/docs/data.rst index 57cc467..63f28de 100644 --- a/docs/data.rst +++ b/docs/data.rst @@ -1,7 +1,7 @@ Data ==== -Please see https://github.com/nkaz001/hftbacktest/tree/master/collector or +Please see `Data Collector `_ or :doc:`Data Preparation ` regarding collecting and converting the feed data. Format diff --git a/docs/debugging_backtesting_and_live_discrepancies.rst b/docs/debugging_backtesting_and_live_discrepancies.rst index 8af5b41..845915b 100644 --- a/docs/debugging_backtesting_and_live_discrepancies.rst +++ b/docs/debugging_backtesting_and_live_discrepancies.rst @@ -25,9 +25,9 @@ infrastructure provides a competitive advantage is beneficial. 2. Queue Model: Selecting an appropriate queue model that accurately reflects live trading results is essential. You can either develop -your own queue model or utilize existing ones. Hftbacktest offers three primary queue models such as PowerProbQueueModel -series, allowing for adjustments to align with your results. For further information, refer to -`ProbQueueModel `_. +your own queue model or utilize existing ones. Hftbacktest offers three primary queue models such as +``PowerProbQueueModel`` series, allowing for adjustments to align with your results. For further information, refer to +:ref:`ProbQueueModel `. One crucial point to bear in mind is the backtesting conducted under the assumption of no market impact. A market order, or a limit order that take liquidity, can introduce discrepancies, as it may cause market impact and consequently make diff --git a/docs/index.rst b/docs/index.rst index c0f732d..0039072 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -61,7 +61,7 @@ Or you can clone the latest development version from the Git repository with: Data Source & Format -------------------- -Please see `Data `_ or `Data Preparation `_. +Please see :doc:`Data `_ or :doc:`Data Preparation `. A Quick Example --------------- @@ -152,17 +152,17 @@ Get a glimpse of what backtesting with hftbacktest looks like with these code sn Tutorials ========= -* `Data Preparation `_ -* `Getting Started `_ -* `Working with Market Depth and Trades `_ -* `Integrating Custom Data `_ -* `Making Multiple Markets - Introduction `_ -* `High-Frequency Grid Trading `_ -* `Impact of Order Latency `_ -* `Order Latency Data `_ -* `Guéant–Lehalle–Fernandez-Tapia Market Making Model and Grid Trading `_ -* `Making Multiple Markets `_ -* `Risk Mitigation through Price Protection in Extreme Market Conditions `_ +* :doc:`Data Preparation ` +* :doc:`Getting Started ` +* :doc:`Working with Market Depth and Trades ` +* :doc:`Integrating Custom Data ` +* :doc:`Making Multiple Markets - Introduction ` +* :doc:`High-Frequency Grid Trading ` +* :doc:`Impact of Order Latency ` +* :doc:`Order Latency Data ` +* :doc:`Guéant–Lehalle–Fernandez-Tapia Market Making Model and Grid Trading ` +* :doc:`Making Multiple Markets ` +* :doc:`Risk Mitigation through Price Protection in Extreme Market Conditions ` Examples ======== @@ -171,7 +171,7 @@ You can find more examples in `examples `_. +Please see the :doc:`migration guide `. Roadmap ======= diff --git a/docs/jit_compilation_overhead.rst b/docs/jit_compilation_overhead.rst index df8f30f..0df8400 100644 --- a/docs/jit_compilation_overhead.rst +++ b/docs/jit_compilation_overhead.rst @@ -40,9 +40,7 @@ backtesting for multiple days, it can still be bothersome. To minimize this over .power_prob_queue_model3(3.0) .tick_size(0.01) .lot_size(0.001) - .maker_fee(-0.00005), - .taker_fee(0.0007), - .trade_len(0) + .trading_value_fee_model(0.0002, 0.0007) ) hbt = HashMapMarketDepthBacktest([asset]) diff --git a/docs/latency_models.rst b/docs/latency_models.rst index 1c5f0ba..b7852f7 100644 --- a/docs/latency_models.rst +++ b/docs/latency_models.rst @@ -13,16 +13,16 @@ HftBacktest has three types of latencies. This is the latency between the time the exchange sends the feed events such as order book change or trade and the time it is received by the local. -This latency is dealt with through two different timestamps: ``local_timestamp`` and ``exch_timestamp`` (exchange timestamp). +This latency is dealt with through two different timestamps: local timestamp and exchange timestamp. * Order entry latency -This is the latency between the time you send an order request and the time it is received by the exchange. +This is the latency between the time you send an order request and the time it is processed by the exchange's matching engine. * Order response latency -This is the latency between the time the exchange processes an order request and the time the order response is received by the local. -The response to your order fill is also affected by this type of latency. +This is the latency between the time the exchange's matching engine processes an order request and the time the order +response is received by the local. The response to your order fill is also affected by this type of latency. .. image:: images/latency-comparison.png @@ -35,98 +35,10 @@ ConstantLatency ~~~~~~~~~~~~~~~ It's the most basic model that uses constant latencies. You just set the latencies. -.. code-block:: python +You can find details below. - from hftbacktest import ConstantLatency - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=ConstantLatency(entry_latency=50, response_latency=50), - asset_type=Linear - ) - - -BackwardFeedLatency -~~~~~~~~~~~~~~~~~~~ -This model uses the latest feed latency as order latencies. -The latencies are calculated according to the given arguments as follows. - -.. code-block:: python - - feed_latency = local_timestamp - exch_timestamp - entry_latency = entry_latency_mul * feed_latency + entry_latency - resp_latency = resp_latency_mul * feed_latency + resp_latency - -.. code-block:: python - - from hftbacktest import BackwardFeedLatency - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=BackwardFeedLatency( - entry_latency_mul=1, - resp_latency_mul=1, - entry_latency=0, - response_latency=0 - ), - asset_type=Linear - ) - - -ForwardFeedLatency -~~~~~~~~~~~~~~~~~~ -This model uses the next feed latency as order latencies using forward-looking information. - -.. code-block:: python - - from hftbacktest import ForwardFeedLatency - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=ForwardFeedLatency( - entry_latency_mul=1, - resp_latency_mul=1, - entry_latency=0, - response_latency=0 - ), - asset_type=Linear - ) - - -FeedLatency -~~~~~~~~~~~ -This model uses the average of the latest and the next feed latency as order latencies. - -.. code-block:: python - - from hftbacktest import FeedLatency - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=FeedLatency( - entry_latency_mul=1, - resp_latency_mul=1, - entry_latency=0, - response_latency=0 - ), - asset_type=Linear - ) +* `ConstantLatency `_ + and :meth:`constant_latency ` IntpOrderLatency ~~~~~~~~~~~~~~~~ @@ -134,52 +46,28 @@ This model interpolates order latency based on the actual order latency data. This is the most accurate among the provided models if you have the data with a fine time interval. You can collect the latency data by submitting unexecutable orders regularly. -.. code-block:: python +You can find details below. - latency_data = np.load('order_latency') - - from hftbacktest import IntpOrderLatency - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=IntpOrderLatency(latency_data), - asset_type=Linear - ) +* `IntpOrderLatency `_ + and :meth:`intp_order_latency ` **Data example** .. code-block:: - request_timestamp_at_local, exch_timestamp, receive_response_timestamp_at_local - 1670026844751525, 1670026844759000, 1670026844762122 - 1670026845754020, 1670026845762000, 1670026845770003 + req_ts (request timestamp at local), exch_ts (exchange timestamp), resp_ts (receipt timestamp at local), _padding + 1670026844751525000, 1670026844759000000, 1670026844762122000, 0 + 1670026845754020000, 1670026845762000000, 1670026845770003000, 0 +FeedLatency +~~~~~~~~~~~ +If the live order latency data is unavailable, you can generate artificial order latency using feed latency. +Please refer to :doc:`this tutorial ` for guidance. Implement your own order latency model ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -You need to implement ``numba`` ``jitclass`` that has two methods: ``entry`` and ``response``. - -See `Latency model implementation `_ - -.. code-block:: python - - @jitclass - class CustomLatencyModel: - def __init__(self): - pass - - def entry(self, timestamp, order, proc): - # todo: return order entry latency. - return 0 - - def response(self, timestamp, order, proc): - # todo: return order response latency. - return 0 +You need to implement the following trait. - def reset(self): - pass +* `LatencyModel `_ +Please refer to `the latency model implementation `_. \ No newline at end of file diff --git a/docs/migration2.rst b/docs/migration2.rst index b597269..1b73ebc 100644 --- a/docs/migration2.rst +++ b/docs/migration2.rst @@ -9,20 +9,17 @@ The migration from version 1 to version 2 introduces several significant changes is used without modification. It is highly recommended to review the updated tutorials. This guide aims to help you avoid common pitfalls during the migration process. -Checking Success: Use if `elapse() == 0` Instead of if `elapse()` ------------------------------------------------------------------ -In version 1, `elapse` function returns `True` on success and `False` otherwise. Typically, the strategy loop checks -for successful elapsing using `while elapse(duration):`. However, in version 2, elapse returns a code instead of a -boolean, with `0` indicating success and any other value indicating an error. Consequently, the code should be updated -to check if the return value equals `0`. - -For instance: - -`while elapse(duration) == 0:` -If the code remains unchanged, it will fail because a return value of `0` (indicating success) will be treated as -`False`. Other methods that involve elapsing, such as `submit_buy_order` or `submit_sell_order`, also return a code -similar to `elapse` instead of a boolean. Ensure to check if their return values equal `0` to confirm success instead of -checking for `True`. +Checking Success: Use ``elapse() == 0`` +--------------------------------------- +In version 1, ``elapse`` function returns ``True`` on success and ``False`` otherwise. Typically, the strategy loop +checks for successful elapsing using ``while elapse(duration)``. However, in version 2, elapse returns a code instead +of a boolean, with ``0`` indicating success and any other value indicating an error. Consequently, the code should be +updated to check if the return value equals ``0``. + +For instance: ``while elapse(duration) == 0`` If the code remains unchanged, it will fail because a return value of +``0`` (indicating success) will be treated as ``False``. Other methods that involve elapsing, such as +``submit_buy_order`` or ``submit_sell_order``, also return a code similar to ``elapse`` instead of a boolean. Ensure to +check if their return values equal ``0`` to confirm success instead of checking for ``True``. Data Format Changes ------------------- @@ -33,12 +30,15 @@ is provided. The major changes are as follows: * SOA to AOS: The format has shifted from a columnar array (SOA) to a structured array (AOS). -* Side Column Removal: `side` column has been removed. In version 2, the side is indicated by the `ev` field flags, - `BUY_EVENT` and `SELL_EVENT`. + +* Side Column Removal: ``side`` column has been removed. In version 2, the side is indicated by the ``ev`` field flags, + :const:`BUY_EVENT ` and :const:`SELL_EVENT `. + * Timestamp Handling: In version 1, the data utility corrects the event order by replacing one of the timestamps with - `-1` to indicate an invalid event on either the exchange or the local side. - In version 2, the validity of events on the exchange or local side is determined by `ev` field's - `EXCH_EVENT` and `LOCAL_EVENT` flags. -Timestamp Unit: Although not strictly enforced, the timestamp unit has changed from microseconds to nanoseconds. + ``-1`` to indicate an invalid event on either the exchange or the local side. In version 2, the validity of events on + the exchange or local side is determined by `ev` field's :const:`EXCH_EVENT ` and + :const:`LOCAL_EVENT ` flags. + +* Timestamp Unit: Although not strictly enforced, the timestamp unit has changed from microseconds to nanoseconds. Additionally, the format for live order latency data has changed from SOA to AOS. \ No newline at end of file diff --git a/docs/order_fill.rst b/docs/order_fill.rst index 4377cb1..fbcb121 100644 --- a/docs/order_fill.rst +++ b/docs/order_fill.rst @@ -10,13 +10,15 @@ simulated market, no market impact is considered. Therefore, one of the most imp small enough not to make any market impact. In the end, you must test it in a live market with real market participants and adjust your backtesting based on the discrepancies between the backtesting results and the live outcomes. -Hftbacktest offers two types of exchange simulation. `NoPartialFillExchange`_ is the default exchange simulation where -no partial fills occur. `PartialFillExchange`_ is the extended exchange simulation that accounts for partial fills in +Hftbacktest offers two types of exchange simulation. :ref:`order_fill_no_partial_fill_exchange` is the default exchange simulation where +no partial fills occur. :ref:`order_fill_partial_fill_exchange` is the extended exchange simulation that accounts for partial fills in specific cases. Since the market-data replay-based backtesting cannot alter the market, some partial fill cases may still be unrealistic, such as taking market liquidity. This is because even if your order takes market liquidity, the replayed market data's market depth and trades cannot change. It is essential to understand the underlying assumptions in each backtesting simulation. +.. _order_fill_no_partial_fill_exchange: + NoPartialFillExchange --------------------- @@ -41,22 +43,12 @@ Liquidity-Taking Order Regardless of the quantity at the best, liquidity-taking orders will be fully executed at the best. Be aware that this may cause unrealistic fill simulations if you attempt to execute a large quantity. -Usage -~~~~~ - -.. code-block:: python +You can find details below. - from hftbacktest import NoPartialFillExchange +* `NoPartialFillExchange `_ + and :meth:`no_partial_fill_exchange ` - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - exchange_model=NoPartialFillExchange, # Default - asset_type=Linear - ) +.. _order_fill_partial_fill_exchange: PartialFillExchange ------------------- @@ -94,22 +86,10 @@ Liquidity-Taking Order quantity do not change due to your execution. Be aware that this may cause unrealistic fill simulations if you attempt to execute a large quantity. -Usage -~~~~~ - -.. code-block:: python - - from hftbacktest import PartialFillExchange +You can find details below. - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - exchange_model=PartialFillExchange, - asset_type=Linear - ) +* `PartialFillExchange `_ + and :meth:`partial_fill_exchange ` Queue Models ============ @@ -120,7 +100,7 @@ If an exchange doesn't provide Market-By-Order, you have to guess it by modeling HftBacktest currently only supports Market-By-Price that is most crypto exchanges provide and it provides the following queue position models for order fill simulation. -Please refer to the details at :doc:`Queue Models `. +Please refer to the details at `Models `. .. image:: images/liquidity-and-trade-activities.png @@ -132,22 +112,12 @@ The decrease in quantity by cancellation or modification in the order book happe order queue position doesn't change. The order queue position will be advanced only if a trade happens at the price. -.. code-block:: python - - from hftbacktest import RiskAverseQueueModel - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=IntpOrderLatency(latency_data), - queue_model=RiskAverseQueueModel() # Default - asset_type=Linear - ) +You can find details below. +* `RiskAdverseQueueModel `_ + and :meth:`risk_adverse_queue_model ` +.. _order_fill_prob_queue_model: ProbQueueModel -------------- @@ -159,6 +129,25 @@ This model is implemented as described in * https://quant.stackexchange.com/questions/3782/how-do-we-estimate-position-of-our-order-in-order-book * https://rigtorp.se/2013/06/08/estimating-order-queue-position.html +You can find details below. + +* `ProbQueueModel `_ + +* `PowerProbQueueFunc `_ + and :meth:`power_prob_queue_model ` + +* `PowerProbQueueFunc2 `_ + and :meth:`power_prob_queue_model2 ` + +* `PowerProbQueueFunc3 `_ + and :meth:`power_prob_queue_model3 ` + +* `LogProbQueueFunc `_ + and :meth:`log_prob_queue_model ` + +* `LogProbQueueFunc2 `_ + and :meth:`log_prob_queue_model2 ` + By default, three variations are provided. These three models have different probability profiles. .. image:: images/probqueuemodel.png @@ -174,192 +163,20 @@ unlike power functions. When you set the function f, it should be as follows. * The probability at 0 should be 0 because if the order is at the head of the queue, all decreases should happen after -the order. + the order. * The probability at 1 should be 1 because if the order is at the tail of the queue, all decreases should happen before -the order. + the order. You can see the comparison of the models :doc:`here `. -LogProbQueueModel -~~~~~~~~~~~~~~~~~ - -.. code-block:: python - - from hftbacktest import LogProbQueueModel - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=IntpOrderLatency(latency_data), - queue_model=LogProbQueueModel() - asset_type=Linear - ) - -IdentityProbQueueModel -~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: python - - from hftbacktest import IdentityProbQueueModel - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=IntpOrderLatency(latency_data), - queue_model=IdentityProbQueueModel() - asset_type=Linear - ) - -SquareProbQueueModel -~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: python - - from hftbacktest import SquareProbQueueModel - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=IntpOrderLatency(latency_data), - queue_model=SquareProbQueueModel() - asset_type=Linear - ) - -PowerProbQueueModel -~~~~~~~~~~~~~~~~~~~ - -.. code-block:: python - - from hftbacktest import PowerProbQueueModel - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=IntpOrderLatency(latency_data), - queue_model=PowerProbQueueModel(3) - asset_type=Linear - ) - -ProbQueueModel2 ---------------- -This model is a variation of the `ProbQueueModel`_ that changes the probability calculation to -f(back) / f(front + back) from f(back) / (f(front) + f(back)). - -LogProbQueueModel2 -~~~~~~~~~~~~~~~~~~ - -.. code-block:: python - - from hftbacktest import LogProbQueueModel2 - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=IntpOrderLatency(latency_data), - queue_model=LogProbQueueModel2() - asset_type=Linear - ) - -PowerProbQueueModel2 -~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: python - - from hftbacktest import PowerProbQueueModel2 - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=IntpOrderLatency(latency_data), - queue_model=PowerProbQueueModel2(3) - asset_type=Linear - ) - -ProbQueueModel3 ---------------- -This model is a variation of the `ProbQueueModel`_ that changes the probability calculation to -1 - f(front / (front + back)) from f(back) / (f(front) + f(back)). - -PowerProbQueueModel3 -~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: python - - from hftbacktest import PowerProbQueueModel3 - - hbt = HftBacktest( - data, - tick_size=0.01, - lot_size=0.001, - maker_fee=-0.00005, - taker_fee=0.0007, - order_latency=IntpOrderLatency(latency_data), - queue_model=PowerProbQueueModel3(3) - asset_type=Linear - ) - -Implement a custom probability queue position model ---------------------------------------------------- - -.. code-block:: python - - @jitclass - class CustomProbQueueModel(ProbQueueModel): - def f(self, x): - # todo: custom formula - return x ** 3 - - Implement a custom queue model ------------------------------ -You need to implement ``numba`` ``jitclass`` that has four methods: ``new``, ``trade``, ``depth``, ``is_filled`` - -See `Queue position model implementation -`_ in detail. - -.. code-block:: python - - @jitclass - class CustomQueuePositionModel: - def __init__(self): - pass - - def new(self, order, proc): - # todo: when a new order is submitted. - pass - - def trade(self, order, qty, proc): - # todo: when a trade happens. - pass - - def depth(self, order, prev_qty, new_qty, proc): - # todo: when the order book quantity at the price is changed. - pass +You need to implement the following traits in Rust based on your usage requirements. - def is_filled(self, order, proc): - # todo: check if a given order is filled. - return False +* `QueueModel `_ +* `L3QueueModel `_ - def reset(self): - pass +Please refer to `the queue model implementation `_. References ========== diff --git a/py-hftbacktest/src/lib.rs b/py-hftbacktest/src/lib.rs index 60f01fc..0cc4ab2 100644 --- a/py-hftbacktest/src/lib.rs +++ b/py-hftbacktest/src/lib.rs @@ -228,6 +228,8 @@ impl BacktestAsset { /// Uses the `RiskAdverseQueueModel `_ /// for the queue position model. + /// + /// * `Order Fill - RiskAdverseQueueModel `_ pub fn risk_adverse_queue_model(mut slf: PyRefMut) -> PyRefMut { slf.queue_model = QueueModel::RiskAdverseQueueModel {}; slf @@ -237,6 +239,7 @@ impl BacktestAsset { /// /// Please find the details below. /// + /// * `Order Fill - ProbQueueModel `_ /// * `ProbQueueModel `_ /// * `LogProbQueueFunc `_ pub fn log_prob_queue_model(mut slf: PyRefMut) -> PyRefMut { @@ -248,6 +251,7 @@ impl BacktestAsset { /// /// Please find the details below. /// + /// * `Order Fill - ProbQueueModel `_ /// * `ProbQueueModel `_ /// * `LogProbQueueFunc2 `_ pub fn log_prob_queue_model2(mut slf: PyRefMut) -> PyRefMut { @@ -259,6 +263,7 @@ impl BacktestAsset { /// /// Please find the details below. /// + /// * `Order Fill - ProbQueueModel `_ /// * `ProbQueueModel `_ /// * `PowerProbQueueFunc `_ pub fn power_prob_queue_model(mut slf: PyRefMut, n: f64) -> PyRefMut { @@ -270,6 +275,7 @@ impl BacktestAsset { /// /// Please find the details below. /// + /// * `Order Fill - ProbQueueModel `_ /// * `ProbQueueModel `_ /// * `PowerProbQueueFunc2 `_ pub fn power_prob_queue_model2(mut slf: PyRefMut, n: f64) -> PyRefMut { @@ -281,6 +287,7 @@ impl BacktestAsset { /// /// Please find the details below. /// + /// * `Order Fill - ProbQueueModel `_ /// * `ProbQueueModel `_ /// * `PowerProbQueueFunc3 `_ pub fn power_prob_queue_model3(mut slf: PyRefMut, n: f64) -> PyRefMut {