Differences between FT-Transformer implementations

[jpgard]

Thanks for the great repos and for making this code open-source!

I noticed the disclaimer in the README:

Warning: if you are a researcher (not a practitioner) and plan to use the FT-Transformer model as a baseline in your paper, please, use the implementation that was used in the original paper (not from the rtdl package): [ft_transformer.py](https://github.com/Yura52/tabular-dl-revisiting-models/blob/main/bin/ft_transformer.py).

It is clear that there are differences in implementation between the Transformer class in this repo and the FTTransformer class in rtdl. I am wondering: what is the practical consequence of these differences?

If we are not interested in reproducing the exact results of the NeurIPS 2021 paper (but instead using FT-Transformer as a baseline for other tasks, where its architecture and hyperparameters will be tuned anyway), is there any reason not to use the (more mature) implementation in rtdl?

[Yura52]

Glad that the code is useful!

Today, I would go for the implementation in rtdl for such use cases as you described (I think the original disclaimer was written when rtdl was less mature). There should be no practical differences between the two implementations in terms of task performance.

P.S. FYI, this NeurIPS 2022 paper may also be relevant.

[Yura52]

Also, I recommend pinning the version of rtdl in your requirements.txt, since API changes are possible in rtdl.

[jpgard]

Awesome! Thanks for the quick reply - and I have seen the paper, great work!