The Causes of Leakage Deviations from sinc Function in DFT and Ways to Minimize Them

The article describes a new model for the solution of the instability problem of discrete Fourier transform (DFT) spectra leakage for the incoherent real sinusoidal signal for solving the most accurate estimation of its frequency. This paper shows this phenomenon by a new form of DFT spectrum expression. It consists mainly of the spectrum of the rectangular signal in the form of the sinc function, modulated to the frequency of the tested signal in baseband. To do this, analogous spectra from other neighboring bands formed by the sampling effect are added as aliasing. The analysis shows some ways to minimize or correct it. The paper shows a simpler and substantially lesser effect of aliasing in the DFT spectrum for a complex valued sinusoidal signal.


Introduction
DFT or its fast variant FFT is a widely used instrument for spectral analysis in many branches. In military applications, this instrument is used by radar technology, by communication technology, for evaluation of the Doppler frequency changes of GPS receivers or ultrasonic signals, by sonars, etc.
Its use is very accurate when the analyzed discrete signal is periodic and coherent with DFT. However, for a common incoherent signal this condition cannot usually be fulfilled. In this case, the result of DFT spectral analysis is not accurate. The main effect of this error is known as leakage. It is a widely studied and published problem, for example in [1][2][3][4]. The exact value of this error is usually not evaluated, the effect is only minimized using a selected window function. Yet there are tasks where we try to find the most accurate parameters of the original signal from the result of DFT, when

Model for DFT Error Analysis
To analyze these problems, we use the real valued sinusoidal signal s(t) as a typical signal to demonstrate and describe the leakage effect. Let us consider the unit amplitude A, the frequency F0 and the initial phase ϕ0. The signal frequency F0 corresponds to the period T0 = 1/F0. The sampled signal can be expressed by Eq. The aforementioned DFT spectrum error does not exist if the discrete signal periodicity condition is satisfied and if the condition T = kT0 or NTS = kT0 is met, where k is an integer which expresses the number of periods of the s(n) signal in the period T (δ = 0 in Fig. 1). For the frequency axis it holds kF = F0 or kFS = NF0. In this case, the input signal is coherent with DFT, and the resulting spectrum consists of only one spectral component with the frequency F0 or k0, see Fig. 2a. We obtain incoherent signal if the condition of coherence is not fulfilled. The true frequency of the F0 signal is located between the two discrete values of the DFT frequency axis. This corresponds to a real number of the relative value of the frequency k0 of the input signal. The relative frequency of the input signal can be expressed as k0 = k + δ, where the frequency displacement δ is in the interval −0.5 < δ < 0.5, see DFT spectral analysis brings the already mentioned leakage effect in this test of incoherent sinusoidal signal (k0 is real number, δ ≠ 0). It calculates the erroneous, infinitely wide spectrum of harmonic components symmetrically spaced around the frequency F0 or k0. Their magnitudes correspond to the envelope function sin(x)/x = sinc(x), as shown in Fig. 2b and published in many references. Note that the DFT values in this article are multiplied by 2/N so that the size of the spectrum components corresponds to the amplitudes of the time sinusoidal signals (as in the Fourier series), which is physically more appropriate and more often used in practice.

Deviations of DFT Spectrum from sinc Function
The computed magnitude of the DFT spectrum of the real incoherent sinusoidal signal diverges slightly from the ideal sinc function in various ways. In addition to the F0/FS ratio, these deviations are caused by the change in the phase shift ϕ0 of the tested signal DFT window origin [14]. Depending on the value of ϕ0, the magnitudes of the spectrum can be changed so that the value of some of the components can drop to zero (decrease by up to 100 dBV), see Fig. 3.
The shape of the envelope function also depends on the value δ. For example, if δ = 0.5, the two adjacent highest components should have the same amplitude and the amplitude spectrum should be exactly symmetric around k0. However, in many cases this is not true, see Figs 3a and 3b, where the phase changes (20° and 110°) have turned the asymmetry direction even for the both highest value components. On the other hand, for phase ϕ0 = 65° as the center between the two cases, the shape of the spectrum maxima is almost symmetrical and closest to the sinc function (Fig. 3c).
Therefore, the dependence of the deviation error δ on the phase clearly confirms these conclusions, see Fig. 3d. This example illustrates the effect of the phase of the tested signal on the modulus of DFT spectrum.  Aliasing is the basic cause of the deviation variability from the magnitude of the sinc function. In addition to the aliasing associated with the sampling frequency, there is yet another aliasing, analogous to the aliasing of binary data modulations. The problem is illustrated in Fig. 4. Here, the procedure for creating N samples for DFT is transposed for the sake of clarity by first modulating the rectangular signal of the DFT window to a carrier continuous real valued sinusoidal signal with frequency F0. It is obvious that the continuous and infinite complex valued spectrum function of the rectangular pulse (Fig. 4f) is added to the positive and negative frequency components (−F0, F0) in this modulation, which corresponds to the convolution of both input spectra. This effect is known as image or negative spectral components of real signal. For this reason, they are mutually aliased as shown in Fig. 4h).

Fig. 4 (a) -(e) the time courses of the formation of N signal samples for DFT, (f) -(i) the corresponding spectral functions, (j) the discrete spectrum envelope of the real valued sinusoidal signal, (k) the discrete spectrum envelope of the complex valued sinusoidal signal
The continuous signal is sampled by the frequency FS (Fig. 4d) in the last step (Fig. 4e). The resulting discrete time signal is then equivalent to that of N samples for the DFT spectrum obtained by the standard rectangular windowing. The accordant continuous spectral function thus goes through the convolution with the spectrum of the sampling signal ( Fig. 4i) in the discrete spectrum, which is repeated for each multiple of yFS, where y is integer. Since the spectral function ( Fig. 4h) exceeds the frequency FS/2, the sampling theorem is not fulfilled here and classical aliasing takes place, see Fig. 4j, where the envelopes of discrete spectra are drawn.
Thus it is obvious that DFT adds to the spectrum of a rectangular signal, modulated to the frequency F0 in the baseband 0 to FS/2, also aliasing from the bands for the multiples of the sampling frequency, both for the −yFS of the negative frequency halfaxis and also for the bands with positive yFS frequencies. However, it is apparent that the greatest effect of aliasing will come from the two adjacent bands.
Another notable feature is the aliasing dependence on the relative position of the partial spectra modulated to the frequencies +F0 and −F0. This aliasing dependence will therefore depend on the F0/FS value. Furthermore, it will be evident that both types of aliasing will be limited to zero for high values of N, because increasing N and the corresponding period T decreases the width of the frequency step m, thereby narrowing the sinc band. It is also advantageous to choose the frequency value F0 as close as possible to FS/4, where the effect of the sum of aliasing on both sides is minimal because it is in the middle position between them.
By contrast, the DFT spectrum of a complex valued sinusoidal signal (see Fig. 4k), which is obtained analogously, but without image sub-spectra for the −F0 frequencies, is evidently simpler. Aliasing is manifested here only from adjacent subbands for the frequencies +F0. Thus, this aliasing does not depend on the value of F0/FS and ϕ0. The shape of the sub-spectra will therefore only depend on the value of N.

Spectrum of the Rectangular Signal, Modulated at F0
A suitable approach to obtaining this spectrum is shown for example as Eq. (3-73) in [2], where samples of the real sinusoidal signal are substituted in the general relationship for DFT. The used editing led to two independent expressions for the "carrier" frequencies +F0 and −F0 or +k0 and -k0 (and for ϕ0 = 0) according to Eq. (2): It should be noted that this part was omitted in the third edition of [2]. The second part of DFT spectrum (the second row of Eq. (2)) corresponds with negative spectral components of real valued signal (e.g. [6-9, 16, 19]). Under the above condition N→∞, the spectra of the rectangular window in the individual sub-bands will not influence each other by aliasing. Eq. (2) is then simplified, and the fractions will change to the sinc function. Similarly, the expression for the phase in the exponent is reduced. The spectrum in the basic band for +F0 (e.g. in Fig. 4j), including the influence of the phase shift of the signal ϕ0, can be expressed as sinc function by Eq. (3): The modulus of Eq. (3) is a modulus of the sinc function. The alternating sign values of the sinc function phase are corrected by the expression in the exponent. The resulting phase dependence is then constant and only for frequencies higher than F0 resp. k0 it is shifted by 180°. For the value δ = 0.5, it is +90° and −90°, and the change in the value δ causes an adequate phase shift in addition to module asymmetry, as shown in Fig. 5.
The shape of the envelope function also depends on the value δ. For example, if δ = 0.5, the two adjacent highest components should have the same amplitude and the amplitude spectrum should be exactly symmetric around k0. However, in many cases this is not true, see Fig. 3, where the phase change has turned the asymmetry direction even for the highest value components.

Expression of DFT as a Sum of Individual sinc Partial Spectra
The general equation for the DFT spectrum of a real sinusoidal signal can be expressed for the finite value N as the sum of the base band pairs corresponding to the frequency k0 and −k0 with other spectra pairs for all multiples of the relative sampling frequency yN, where y are integers in the interval −∞ < y < ∞ by Eq (4): In this way, aliasing from all other bands is reflected in the base frequency band and we obtain the DFT spectrum in a different way than according to Eq. (2). Of course, this calculation is more complicated than the calculation by Eq. (2) or by DFT, although for a deviation from DFT of less than about 10 −3 (−60 dB) it is sufficient to solve Eq. (4) for y only in the interval −5 < y < 5.
An important conclusion is that this way of expressing DFT shows the abovementioned effect of aliasing and the influence of the phase ϕ0 of the signal, which is the principal reason why the undefined change of the signal phase causes seemingly random changes to the DFT magnitude leakage, see Fig. 3. However, many articles on frequency estimation consider this effect as a bias of method with the cause of estimation nonlinearity, e.g. [10][11][12].
On the other hand, in the case of a complex sine wave, Eqs (2) and (4) can be used to express the DFT with only the first line valid for both equations: This corresponds to Fig. 4k and causes only classical aliasing without the aliasing effect of the image components. Therefore, the effect of aliasing depends only on N and it does not depend on F0/FS and ϕ0. It follows that the relation for deviation δ can be expressed exactly from the three largest components of the spectrum Xk-1 to Xk+1 only as a function of N, with absolute accuracy. It was derived universally and with consideration of zero-padding in [15]. In the case without padding, the relation for δ can be expressed as where U is expressed from the three largest components of the spectra

Ways for Minimizing Leakage Deviations from sinc Function
The previous analysis for a real valued sinusoidal signal has shown that the parameters N, F0/FS and ϕ0 have a decisive influence on the DFT spectrum deviations from the sinc function and on their instability. The evaluation of these deviations is also limited by the signal to noise ratio (SNR). The analysis clarifies that increasing the value of N is the easiest way to reduce the discussed deviations. This is well known but it has practical limits. Another way results from the mentioned symmetrical dependence of deviations on the value of F0/FS in the interval 0 < F0/FS < 0.5, where the maximum deviation is at the edges of the interval and the minimum is for the value F0/FS = 0.25. Examples of these dependencies are given e.g. in Figs 6a and 6b. In practice, it may be difficult to set 0.25. Then we at least avoid the boundary values of this interval with worse results.
The use of window functions is an essential tool for minimizing the discussed deviations. It is discussed in many publications, e.g. [15,16,18,19,21], but the results of these works are not unambiguous and they are even contradictory in some cases. The conclusions of the presented analysis explain why it is and what the principles for optimal use of suitable window functions should be. Obviously, the effect of the window function used depends on several parameters. Above all, the influence of the F0/FS ratio is obvious. The value of relative distance of the maximum of the nearest neighboring sub-band is the main reason. For F0/FS < 0.25, it is the first negative sub-band. If this relative range is very small, aliasing from the nearest sub-band has a dominant influence on the deviations from the sinc function. Therefore, the efficiency of such a window, whose frequency dependence has a minimum transmission of aliasing components from this neighboring band, is increased just for the evaluated baseband components (e.g. Hann's window). In the case of larger distances (e.g. F0/FS > 0.1), the influence of distant sub-bands increases. In this case, it is desirable that the frequency dependence of attenuation of the used window be increasing (e.g. Hann's window). The effect of the near image maximum aliasing effect can be so pronounced that for small F0/FS values, the error values of deviation δ, e.g. a rectangular window, are lower compared to e.g. Hann's window as shown in Figs 6c and 6d.
Similar considerations apply when we observe the degree of the deviations effect for different SNR. Obviously, for a low SNR, the effect of more distant sub-bands is below the noise level. Therefore, window functions can be used whose attenuation is greatest for the nearest sub-bands with the dominant effect above the noise level [13]. With a high SNR value, the aliasing from far sub-bands must also be considered. The width of the first sidelobe also has a similar effect [16].
These aspects can combine with each other. This explains why it is difficult to find the optimal selection of the window function so as to attenuate the transmission of aliasing from neighboring sub-bands with the most significant influence. This fact also explains the differences in the results of various publications on the selection of appropriate window functions, where the authors do not consider all the aspects and parameters resulting from the presented analysis.
A special case is the procedure where the authors used the estimated δ, calculated for a rectangular window, for other types of windows. The difference from the rectangular window was compensated by a fixed correction value (e.g. [15,20]). These procedures cannot work universally because under different conditions (N, F0/FS, SNR) the effect of window functions is different, as is evident from the presented analysis. The authors derived the respective correction coefficients for a particular case; however, for other conditions, these procedures can produce worse results.
It is appropriate to compare these relations with the frequently used Jacobsen's relation for a rectangular window [11,20] where P = 1.36 is recommended for Hann's window. It turned out, however, that by simple correction of Eq. (10), when the signs of both smaller terms in the denominator change analogously to Eq. (9) and if we use a multiplication coefficient with the value exactly 2, we get the relation 1 1 This adjusted relation achieves significantly higher accuracy of estimation δ (see Fig. 6b than the most commonly used Jacobson's estimator for the rectangular window according to Eq. (9), see Fig. 6a.
Analogously to the correction of Eq. (10), the correction of Eq. (11) was also performed. Here the value of Xk was doubled and the exact coefficient 2 was again used for the whole value of δ. The resulting relationship is 1 1 Again, the resulting relationship gives a very low estimate error value, as shown in Fig. 6b. The results are practically the same as for Eq. (12) with the complex components. By comparing Eq. (13) with the available literature, it can be concluded that Agrež has worked towards the same relationship [19]. a) relation (9), b) Eqs (12) and (13), comparison of Eq. (9) and Eqs (12) and (13) for N = 64 c) for low value of F0/FS (< 0.05), d) for high value of F0/FS (> 0.44) Another possible way to reduce the instability of the deviations from the sinc function is to eliminate the difficult-to-solve effect of the undefined phase ϕ0 of the test signal against the DFT window. It appears almost as random noise even for the amplitudes of the evaluated components, as explained in the analysis. Therefore, the solution is to use a time synchronization of the DFT window with the tested signal so that the phase ϕ0 is defined and constant.

326
DOI 10.3849/aimt.01375 The presented analysis shows that for a complex valued sinusoidal signal most of the above problems disappear. Neither changing the F0/FS value nor ϕ0 value of the test signal is of any effect. The deviation of the DFT spectrum from the sinc function is only given by the value N. Therefore, it can be expressed explicitly and a zero estimate error can be obtained [15]. That is why it is preferable to use only a rectangular window.
The paper further shows that the solution of frequency estimation can be based on the estimation for the sinc spectrum itself without the effect of aliasing Eq. (9) according to Eq. (3).

Conclusions
The presented model of the DFT spectrum of the incoherent sinusoidal signal shows aliasing from all partial spectra from the −F0 and +F0 sub-bands as the cause of leakage deviations from sinc. In the case of a real valued sinusoidal signal, the essential influence of ratio F0/FS is explained. This aliasing and the phase shift of the signal ϕ0 together with the phase jump by π between the two parts of the basic spectrum (see Fig. 5) cause an unpleasant sensitivity of the spectrum module to the phase of the analyzed signal. This effect appears as a practical problem, for example when estimating the frequency of a dominant signal in the DFT analysis for many measurement tasks.
By contrast, in the case of a complex valued sinusoidal signal, it is only aliasing from the sub-bands for the frequencies +F0, so that the above problems do not occur. The effect of aliasing is constant and defined only by value of N. This confirms the possibility of expressing an exact relationship for deviation δ (7,8) [15].
The ways to suppress these effects for a real valued sinusoidal signal result from this analysis and consist in suppressing the effect of aliasing including the effect of phase uncertainty. The easiest and well known way is to increase the value of N, but this is limited in practice. If possible, the discussed effect can further be reduced by setting the F0/FS ratio close to 0.25 in order to minimize the sum of aliasing from both sides.
Yet another way is to use a window function that suppresses the effective leakage band and thus the aliasing effect even for low N values. This way was used in many different approaches with ambiguous or even contradictory results. The optimal choice of the window used depends on the F0/FS and SNR values, as can be deduced from the analysis. These parameters affect both the degree of the aliasing effect from individual sub-bands on deviations from the sinc function and the accuracy of their evaluation. Therefore, different estimation results for different values of these parameters could be achieved in the cited publications. On the other hand, some authors used unsuitable procedures where the estimated value δ, calculated for a rectangular window, was used for other window types only with a fixed correction. These procedures appear to be inappropriate for the above reasons. Moreover, these relationships were corrected for the Hann's window. The resulting estimates for δ (12), (13) give high estimation accuracy δ in accordance with the expected effect of a suitably selected window.
Another possible way to minimize the analyzed effects is to synchronize the DFT window with the defined phase ϕ0. In this way, the leakage deviations from the sinc function caused by a random change in the value of ϕ0 can be suppressed as illustrated in Fig. 3 for a suitably selected ϕ0 that results in a minimum estimate error value.
In the case of complex sinusoidal signal aliasing, the effect of aliasing can be defined explicitly and there is no need to use special procedures for its minimization, as is the case with a real sine wave signal.