Vision-based Dynamics Monitoring (VDM) for Diagnosing the Variations of Wind Turbine Tower Foundation Conditions

A slight uneven settlement of the foundation may cause the wind turbine to shake, tilt or even collapse, so it is increasingly necessary to realize remote condition monitoring of the foundations. At present, the wind turbine foundation monitoring system is incomplete. The current monitoring research of the tower foundation is mainly of contact measurements, using acceleration sensors and static level sensors for monitoring multiple reference points. Such monitoring methods will face some disadvantages, such as the complexity of monitoring deployment, the cost of manpower, and the load effect on the tower structure. To solve above issues, this paper aims to investigate wind turbine tower foundation variation dynamic monitoring based on machine vision. Machine vision monitoring is a kind of non-contact measurement, which help to realize comprehensive diagnosis of early foundation uneven settlement and loose faults. The FEA model is firstly investigated as the theoretical foundation to investigate the dynamics of the tower foundation. Secondly, the Gaussian-based vibration detection is adopted by tracking the tower edge points. Finally, a tower structure with distributed foundation support is tested. The modal parameters obtained from the visual measurement are compared with those from the accelerometer, proving the vision method can effectively monitor the issues with tower foundation changes.


Introduction
In recent years, wind energy has become an important source of renewable energy, and with the rapid increase in installed wind power capacity, the number of wind turbine accidents has also been increasing.Compared with local failures, such as electrical failures and blade failures, a wind turbine tower failure will lead to a tower collapse accident, which is the most serious, so it is particularly important to strengthen the structural health monitoring of wind turbine towers during operation [1].
The supporting foundation of wind turbines needs to withstand eccentric loads of various wind directions, it plays a critical role in ensuring the stability and operational safety [2,3].A slight uneven settlement of the foundation will make the wind turbine produce a large horizontal deviation, uneven foundation will cause the wind turbine tower to tilt or collapse, in order to prevent foundation loosening and uneven subsidence in early stage, the basic health monitoring technology of Wind Turbine (WT) foundation is the technical basis of preventive maintenance [4,5].The condition monitoring of the transmission parts of wind turbines is relatively complete, but the condition monitoring of the tower barrel and the foundation is still insufficient [6].
At present, the geodetic method is mainly used in monitoring the wind turbine foundation, that is, the geometric leveling is used to monitor the settlement and tilt of foundation, and the reference points are buried in the deformation area of the foundation and the observation points of uneven settlement are arranged [7].Stress and strain instruments are also used in special projects to monitor the stress of concrete structures and steel structures, and static level and inclinometer are used to monitor the tilt of structures, but these methods are limited to special projects and not universal.Natural frequency is a key parameter of wind turbine foundation [8,9].WT and its foundation are coupled interaction system, the frequencies of foundation can be derived indirectly by monitoring the tower barrel [10][11][12].The traditional modal parameter identification of the wind turbine tower is generally measured by accelerometer and that heavily rely on knowledge of the excitation forces and precise sensor placement [13], this kind of methods have large workload, which will increase the cost of installation, operation and maintenance.In addition, these sensors cannot be placed in locations with strong stress or strain gradients [14].Ochieng [15] studied the use and concept of radar as a non-contact sensor for wind turbine blade monitoring.Light detection and Ranging (LIDAR) have been proposed as a contactless technology, but it requires a huge number of sensors to implement, which is expensive and not easy to maintain.Meanwhile, achieving high-resolution spatial measurements for large-scale structures is challenging [16].
Vision-based machine and structural monitoring and evaluation has become popular over the past few decades, the major advantage is the contactless nature property without affecting the vibration structure characteristics, it has the property of fullfield measurement [17,18].Benefiting from the rapid development of computer technology, vision-based measurement methods have the advantages of noncontact, low cost and high accuracy and it can realize multi-point measurement [19].Y. Bian, et al [20] proposes a method to monitor the health of wind turbine towers based on stereo vision.
H. Cai [21] proposes a pose measurement approach based on two stage binocular vision for docking large components.Due to the large displacement of the top of the wind turbine tower, the measurement positions are mainly sampled from the top of the tower.The top of the tower cylinder may be affected by other structures, such as bending and other highfrequency effects, however, because the natural frequency of wind turbine tower is low, the high-frequency effects can be ignored.
The wind turbine moves under the wind excitation, but the wind is random and complex, and the excitation of each point on the wind turbine tower is different, so the modal parameters estimation of operational structures is often a challenging task.Stochastic Subspace Identification (SSI) based method is usually chosen as a novel tool to do the modal parameter identification and uncertainty quantification of high-rise buildings [25,26], especially for the timevarying structural systems considering uncertainty quantification [27][28][29][30], it was developed for modal parameter identification of a linear system with outputonly data [31].This method can estimate a linear time-invariant state-space model from correlated sequences of observed data.SSI and its algorithmic variants continue to be employed as reliable means of operational modal analysis (OMA) [32,33].
In order to address the above issues of traditional measurement methods, such as complicated deployment and maintenance, large manpower consumption and load effect on the structure, and to achieve accurate system modal parameters to realize the diagnosis of basic faults, a vision-based dynamics monitoring (VDM) method for tower structure foundation conditions is proposed in this paper, which investigates the tower vibration patterns with stiffness variations of the WT tower foundation conditions.Through visual detection of tower frequencies, the conditions of the tower foundation can be detected.

Methodologies
In order to obtain the multimode parameters of the tower foundation structure, two main steps are employed, which are 1) Finite element analysis simulation assists in the vibration mode analysis of the stiffness change of tower structure foundation using COMSOL; 2) Gaussian Fitting based visual monitoring method is developed for diagnosing the variations of WT tower foundation conditions.Through the visual monitoring of the tower, the condition of the foundation can be detected.

Dynamics of A Tower-Foundation Structure under Variable Conditions
FEA method is adopted to setup the model of tower structure foundation with variable stiffness via COMSOL.In the practical scenarios, the ground is not rigid, it is deformable and uneven.Therefore, the flexibility of the foundation is usually considered [34].The schematic diagram of simplified WT with distributed foundation stiffness is shown in Fig. 1(a).� 1 , � 2 , � 3 , � 4 define the stiffnesses of the four corners of the foundation.� 1 , � 3 describes the stiffness of the two corners in accordance with the wind direction(fore-aft). � 2 , � 4 describes the stiffness of the two base feet perpendicular to the wind direction (Side-side).
(a) (b) ), the first two frequencies � 01 , � 02 are close to each other as one pair of homologous frequencies, and when the global stiffness decreases, the natural frequencies of the system decrease.When the foundation is axisymmetric as Case 6, the difference between � 01 and � 02 is the largest.With the non-uniformity diagonal variation of the foundation, the system will appear two similar vibration patterns.The non-uniformity of the four stiffness distributions result in a difference between the same root frequencies.In summary, this method can initially identify variations of WT tower foundation conditions.One is diagonal non-uniform distribution fault, showing that the difference between the first two orders of characteristic frequencies increases as Case 6, the other is the early variation characteristics of uneven subsidence, showing a first-order frequency decline, such as Case 2, Case 3, Case 4, Case5, �_Case 1 > �_Case 3 > �_퐶푎 4 > �_퐶푎 5 > �_퐶푎 2 , with the increase of the subsidence defect degree, the natural frequency of the system decreases gradually.

Vibration Extraction from Vision Data
A digital image consists of two-dimensional elements, each pixel contains a specific position �, � and pixel value � �, � .
Multiple pixel points at successive time instants can be extracted in the space using the edge extraction technology based on the multiple images of high-speed videos.The principle of image edge extraction is to regard the gray level of the image as a twodimensional surface, and the edge is the prominent part of the surface.To obtain the gradient of an image, it is necessary to calculate the position of each pixel in the image to find the point of change.
In order to obtain high-precision positioning of edge points from images, sub-pixel positioning technology can significantly improve the accuracy and efficiency of image processing, which makes the image processing results more in line with the actual requirements.For sub-pixel edge position, Gaussian fitting is often used as a filter to improve the accuracy of edge position acquisition [35].Gaussian function has good fitting property and can accurately describe the change of edge position, providing a reliable basis for subsequent data processing and analysis.Through the edge curve fitting of the pixels near the edge of the sub-pixel, some columns of points are usually selected near the edge, the gray value of these points is obtained.The Gaussian fitting curve for edge acquisition is defined as Eq.(1).
where A is the amplitude constant, (� 0 , � 0 ) defines the coordinates of the center pixel, � � , � � is the variance.Two-dimensional convolution can be used to find image edges, mainly by using a specific convolution kernel.These convolution nuclei detect edges in the image, where the pixels position change, the convolution in two dimensions is defined as Eq. (2).
where f represents the original image, K represents the convolution kernel, �, � is the pixel coordinate, �, � is the convolution kernel element coordinates.� ' �, � is the output result after convolution.At the edge position, the gradient has a peak value, and a 2-dimension convolution kernel � �, � = −4, − 3, − 2, − 1, 0, 1, 2, 3, 4 is constructed based on required resolution to realize vertical edge extraction.When it is cross correlated with the input image pixel matrix, if the horizontal adjacent elements are the same, the output is 0, otherwise the output is non-0; The position of edge points is obtained from the distribution of 0 and non-0 in the output matrix.
Finally, the symmetry axis position of the Gaussian curve is obtained through the fitting curve, that is, the position of the target pixel as shown as Eq.( 3).
where �, � is the position of the pixel point, � �,� is the edge points set.� � defines the position of left edge points set of the tower, and � � defines the position of the right edge points set of the tower, as shown as Eq.( 4).
The vibration process is then obtained by tracking the edge pixel point' spatial displacement in the videos by reading the continuous pixel position from the image sequence.The WT tower edge acquisition process is summarized as shown in Fig. 2. Fig. 2. Vibration extraction approach.

Experimental Setup
The purpose of the experiments is to investigate the vibration patterns of the wind turbine under various elastic support to simulate the base loose, collecting data through current collection, acceleration sensors, and vision systems, in order to demonstrate the Vision-based dynamics monitoring (VDM) method is effective for wind turbine tower foundation conditions monitoring.

Experimental Platform
The experimental facilities are composed of wind tunnel, wind turbine tower with distributed foundation, a high-speed camera with remote Bluetooth controller, one x-y-z accelerometer, one ORS, one set of external sliding rheostat, one laptop, one data acquisition system and materials different elastic materials as shown in Fig. 3(a).The experiment is setup in the wind tunnel lab according to schematic diagram as shown in Fig. 3

(b).
A wireless 3-axis acceleration sensor (ORS) is amounted onto the main rotor end, it is used to monitoring the rotation speed of the wind turbine, its sampling frequency is 1000Hz and the range is 2g.The camera is an iphone8plus camera with resolution 2448×2048 pixels, it is set at a distance 82.5cm away from the WT tower, collecting the vibration of the tower along the wind, the sampling rate is 240fps.
The wind tunnel is used in this experiment, along with addition axial wind blower.The wind speed can be acquired by Eq. ( 5).
where V is the wind speed generated by the wind tunnel, P is the display value of the barometer.
The rotating speed of the wind turbine can be adjusted by different load resistance values, in this experiment, the external resistance is set to the value of 155.1Ω.

Tests of the WT tower foundation early faults
The early manifestations of the WT tower's non-uniform settlement, tilt and other faults are the non-uniform variations of WT foundation conditions, that is, the stiffness changes in different directions of the WT tower foundation.In this experiment, 8 cases are set up to simulate the different variations of WT tower foundation conditions by placing four kinds of different elastic material supports under the base corners of WT tower foundation as shown in Fig. 4. The ground is often uneven, when the WT is placed on the ground, its foundation may not be able to reach a balanced condition; In addition, it is sometimes difficult for the naked eye to find the uneven variations of the foundations, so it is necessary to use visual and acceleration measurement to monitor the variations of the WT foundation conditions.
Corresponding to section 2.1, two types of faults are mainly tested, one is the uneven subsidence, which has a diagonal nonuniform stiffness variation, and the other is the foundation loosening, which has the overall stiffness decline variation.

Results and Discussion
This section mainly discusses the monitoring results of the vision method.The variations of WT foundation conditions monitoring are realized via visual measurement.The effectiveness of VDM method is discussed by comparing with the measurement results of acceleration sensor.Finally, SSI algorithm is associated with vision method to further consider the real scenarios.

Vision-based vibration process extraction
Firstly, the acquired image sequence is read, and then the edge point set is extracted according to the algorithm in Section 2.2.A total of 697 edge pixels are extracted.The direction of the visual shooting is parallel to the wind direction, that is consistent with the direction of the fore aft.The vibration process is obtained from a sequence of images in the sampling videos as shown in Fig. 5.The measurement positions are mainly sampled from the top of the tower, which has a good signal to noise ratio.Even the top of the tower cylinder may be affected by other structures, such as bending and other high-frequency effects, however, because the natural frequency of wind turbine tower is low, the high-frequency effects can be ignored.697 pixels edge points are obtained， the spectrum waterfall of the single pixel point is obtained under 8 cases of conditions by adding Hanning window to process the vibration signals as shown in Fig. 9.As the external wind speed excitation increases, the rotation speed of the WT also increases, the first peak is around 2.6Hz, that is rotation speed � � .VDM can sense the change of blade rotation and reflect the blade coupling vibration information in the spectrum diagram.As the resistance of the external load increases, the rotation speed will also increase.The rotation frequency can be reflected in the acceleration spectrum, the rotation frequencies change with the WT rotation speed.As can be seen from the spectrum from the images, when the foundation is not diagonal uniformly distributed as Case 6 in section 2.1, the frequency separation of the harmonic roots in band of 4.8Hz~6.3Hzwill occur.Two similar wave peaks will appear in cases of Group 1, and their spacing will change with the change of distribution, the instability of the structure will stimulate the low-frequency component of the vibration signal along the direction of the tower, resulting in the increase of the harmonic root modes' spacing of the system, which is reflected in the spectrum diagram as two relatively close wave peaks.Usually, the ground is uneven and there is an imbalance in the foundation stiffness on the ground, the use of large mat pad can make the stiffness of the foundation more evenly distributed.When two harmonic peaks ap-pear, the WT foundation have a diagonal asymmetric variation that will cause the uneven settlement fault.

Comparison of the VDM and conventional vibrations
In order to verify the validity of the VDM method in the experimental data, the results are compared with the acceleration measurement results.The spectrums of the vibration process data obtained from the accelerometer are shown in Fig. 10.Fig. 10.Spectrums of vibration process of 8 cases from the accelerometer.
It is consistent with the visual measurement results, when the foundation has a condition of diagonal uniformly variations, the first pair of characteristic frequencies presents two peaks.The comparison between vision and the accelerometer vibration shows that the deviation between these two methods is small, which can prove the validity of the visual method in measuring the WT tower structural vibration as shown in Fig. 11(a).
Meanwhile, as the distribution gradually becomes symmetrical equilibrium, the difference between the harmonic peak and peak values gradually decreases, as shown in Fig. 11(b).It shows that cases of Group 1 have diagonal inhomogeneous settlement fault.The result of VDM is consistent with the conclusion of the acceleration signal through the comparation.Cases with frequencies lower than Case Mat belong to the type B of fault, that is the early variation characteristics of uneven subsidence, showing the natural frequency decline.�_푀푎 > �_퐶푢 3푀푎 1 > �_퐶푢 2푀푎 2 > �_퐶푢 1푀푎 3 , with the decrease of the foundation stiffness, the natural frequency of the system decreases gradually.

Modal Properties-results from SSI
The core of condition monitoring of largescale WT tower is to perform structural modal analysis quickly and accurately, so as to determine modal parameters under unknown excitation conditions.In practice, the wind speed and direction are very complex, which will excite multi-order frequencies and multi-modes, in order to ensure the accuracy of the proposed VDM method, SSI algorithm is integrated into visual signal analysis and processing to improve the accuracy in real scene.The frequencies of the WT tower and the mode shapes can be obtained via Eq.( 7)-Eq.( 9) in section 2.3.Totally 697 points along the tower barrel (1.2m) are selected for processing, in the practical scenarios, by adjusting the shooting angle and focal length to select the ROI region, the resolution can be fully utilized, at the same time a high frame rate camera can be used to ensure that the wind turbine tower barrel has enough sampling pixels.Multiple points can be selected and analyzed by subspace state, Fig. 13 shows the mode shapes of 7 points, which are randomly selected from base to top.The vibration amplitude of the point on the top of the wind turbine is larger, and the vibration amplitude near the tower base is smaller.Since the camera captures vibrations along the wind direction (fore-aft), the modes derived from visual images are mainly two-dimensional.The above data shows that SSI method can be effectively applied in visual signal monitoring, and the system mode changes can still be obtained under the uncertain external wind excitation, and the modes of each order can be characterized according to the vibration modes.
In addition to the above mode shapes, the damping ratios corresponding to different modes of the system can also be obtained by SSI method (See The damping ratio is important in the stability analysis and shock absorption of the system.
In this experiment, under wind tunnel excitation, there are multiple frequency peaks caused by the harmonic components, SSI can help characterize the accuracy of fault frequency identification in the previous section.The application of SSI in vision can improve the accuracy of the frequency parameters under different variations of WT tower foundation conditions, SSI enables lower measurement uncertainty levels to be reached.In the future, it is necessary to design an optimized state space analysis method according to the practical application requirements.

Conclusion
In this paper, the VDM method is proposed to monitor the variations of WT tower foundation conditions.The multi-point monitoring of the visual method brings an efficient approach of monitoring the condition of large-scale infrastructure.Combined with its easy deployment, remote monitoring, no load effect and other characteristics, it can be more convenient to achieve WT foundation fault diagnosis by monitoring the WT tower.Through comparing with acceleration results and COMSOL dynamic analysis, it shows that VDM can effectively detect early faults such as uneven loosening of WT tower foundation.
SSI algorithm is integrated into visual signal analysis and processing.With the combination of computer technology and improved SSI technology, the accuracy and efficiency of VDM detection will be greatly improved, laying a foundation for the identification of small early faults in complex dynamic scenarios.

Fig. 1 .
Fig. 1.(a)The schematic diagram of tower structure model with distributed foundations stiffness.(b) Mode shape obtained.The early sinking and loosening faults of the foundation are simulated by adjusting the

Fig. 4 .
Fig. 4. Tests solutions of WT tower under variable foundation conditions.

Fig. 7
Fig. 7 shows the vibration in time domain of the two groups of the cases, each group has four cases.

Fig. 7 .
Fig. 7. Vibration process of a single pixel point in the time domain for different foundation cases.

4. 2 Fig. 8 .
Fig. 8. Data in the frequency domain under variations of WT foundation from vision data.

Fig. 11 (Fig. 12 .
Fig. 11(a).The first natural frequency comparations between vision measurement and accelerometer measurement.(b).Harmonic wave peak-peak value between the 1 st pair of frequencies.The difference against the acceleration measurement is that the vision can realize multi-point monitoring via simple deployment and vision method can achieve remote monitoring without any load effect.Visual methods can provide more intuitive, multi-angle monitoring results as shown in Fig.12, in which the energy distribution can be seen directly.When the WT foundation has a non-uniform diagonal variation, its energy is divergent, otherwise, the energy is concentrated.The first 4 cases belong to

Fig. 13 .
Fig. 13.SSI analysis of cases in Group 1 from vision data.

Table 1
. shows the first two order frequencies � 01 , � 02 of WT tower under variable foundation conditions.In columns of schematic drawing, different colors represent different stiffness values.Case 1 ( � 1 = � 2 = � 3 = � 4 = � ) is the normal foundation condition without any fault.There are two types of variations of WT tower foundation conditions, one is the

Table 1 .
The first two frequencies of a WT structure under variable foundation conditions.

Table 2 .
). Case 1~8 represent different elastic supports of the base, that are Floor, Cork, Rubber, Cube material.