BERT for Harmonic Time Series Modeling: A Multi-Stage Fine-Tuning Approach

Abstract

This study demonstrates the potential of a BERT-based transformer model in harmonic signal modeling using synthetic sinusoidal data. The model was trained through a three-stage fine-tuning process (reconstruction, linear analysis, full tuning) with a masked language modeling approach. In the first stage, the model successfully filled in missing data and learned the basic features, while in subsequent stages, its ability to capture temporal dependencies and sequential patterns was enhanced. Additionally, patch, time, and station embedding strategies effectively represented the harmonic structure of the signal. The results indicate that pre-training with synthetic data can overcome the limited access to real-world data, allowing transformer models to be efficiently used in these types of problems.