We designed the window function of the optimal Gabor transform based on the time-frequency rotation property of the fractional Fourier transform. Thus, we obtained the adaptive optimal Gabor transform in the fractional domain and improved the time-frequency concentration of the Gabor transform. The algorithm first searches for the optimal rotation factor, then performs the p-th FrFT of the signal and, finally, performs time and frequency analysis of the FrFT result. Finally, the algorithm rotates the plane in the fractional domain back to the normal time-frequency plane. This promotes the application of FrFT in the field of high-resolution reservoir prediction. Additionally, we proposed an adaptive search method for the optimal rotation factor using the Parseval principle in the fractional domain, which simplifies the algorithm. We carried out spectrum decomposition of the seismic signal, which showed that the instantaneous frequency slices obtained by the proposed algorithm are superior to the ones obtained by the traditional Gabor transform. The adaptive time frequency analysis is of great significance to seismic signal processing.