Skip to content
/ FrFNet Public

The repository of the Fractional Fourier Transform Meets Transformer Encoder paper in IEEE Signal Processing Letters

Notifications You must be signed in to change notification settings

koc-lab/FrFNet

Repository files navigation

Fractional Fourier Transform Meets Transformer Encoder

In this study, we introduce fractional Fourier transform version of the FNet model, which is called FrFNet. Instead of the attention layer in transformer encoders, we use fractional Fourier transform layer. We can obtain better GLUE benchmark scores by optimizing fraction values. Our paper is published in IEEE Signal Processing Letters. Early access version of the paper: https://ieeexplore.ieee.org/document/9931916.

Pre-training and Fine-tuning

This repository is based on FNet repository and contains necessary modifications for implementation and adaptation of fractional Fourier transform to transformer encoder structure. For installation of the required libraries please refer to original repository.

For both training scheme SentencePiece vocabulary model should be downloaded. Assuming the current directory points out this repository, you can start training with the following command line:

python main.py --workdir=$WORKING_DIRECTORY --vocab_filepath=$VOCAB_FILE_PATH --config=$CONFIGURATION_FILE_PATH

Here, $WORKING_DIRECTORY determines the directory where checkpoints and other necessary files will be saved. $VOCAB_FILE_PATH indicates the directory of the SentencePiece vocabulary model and $CONFIGURATION_FILE_PATH depends on training scheme and directs to either pretraining.py or classification.py in the configs folder.

For example (assuming the current directory is this repository),

python main.py --workdir=./pretrainings/frac_order_0.75/ --vocab_filepath=./vocab/c4_bpe_sentencepiece.model --config=./configs/pretrainig.py

Adjusting the fractional order and FrFNet model configuration

For training by FrFNet configuration, in configs/base.py, please adjust model architecture as config.model_arch: ModelArchitecture = ModelArchitecture.FRAC_NET (default configuration) and set config.frac_order as desired fraction value. Other model parameters can also be customized from the same file.

Bibtex

@article{sahinuc2022fractional,  
  author={\c{S}ahinu\c{c}, Furkan and Ko\c{c}, Aykut},  
  journal={IEEE Signal Processing Letters},   
  title={Fractional Fourier Transform Meets Transformer Encoder},   
  year={2022},
  doi={10.1109/LSP.2022.3217975}
}

About

The repository of the Fractional Fourier Transform Meets Transformer Encoder paper in IEEE Signal Processing Letters

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages