Medication Patterns of Traditional Chinese Medicine for Diabetic Cardiomyopathy Based on Data Analysis and Deep Reinforcement Learning

A.DRLs COMBINED WITH DATA ANALYSIS

Diabetic cardiomyopathy is a myocardial disease that occurs in diabetic patients and cannot be explained by other cardiac pathologies [16,17]. There are many TCM treatments for diabetic cardiomyopathy. The use of artificial intelligence methods to analyze the medication patterns of these treatments can reduce the adverse effects of medication. It can also save medical resources by avoiding drug wastage through rational drug administration [18,19]. However, many of the current analysis models have low accuracy for drug dosage analysis and need to be optimized [20]. DRL algorithm can improve its performance by connecting deep learning and reinforcement learning [21]. The specific flow of this algorithm is shown in Fig. 1.

Fig. 1. Basic process of DRL.

In Fig. 1, DRL first needs to acquire high-dimensional observation data and utilize the powerful perceptual ability of deep learning to perceive the specific feature representation of the acquired data. Then it evaluates the value function of each action based on the expected return. Moreover, the relationship between the state and the action is fitted by the neural network in deep learning, and the DRL then selects an action to execute based on the strategy. The surrounding environment reacts to the action and gives the next observation data and reward mechanism based on the action. The execution of the above action repeats until the given condition is satisfied. The DRL then updates the initial strategy based on all the feedback results obtained after the execution of the action. The above process is repeated continuously with trial and error until the DRL algorithm is able to learn the optimal solution to achieve the goal. In this process, the data state value function is calculated as shown in Equation (1):

In Equation (1), v(s) is the value function of the strategy in the current state s. r is the immediate reward obtained after taking the action. r is the discount factor. v(s1) is the value function of the next action state and the expected value. The data action value function is calculated as shown in Equation (2):

In Equation (2), Q(s,a) denotes the action value function. R denotes the reward received after taking an action. a1 denotes the optimal action taken in the next state s1. Whereas, when performing strategy iteration, the optimal strategy is found by calculating the optimal value function. The calculation is shown in Equation (3):

In Equation (3), vbest represents the optimal value function. Qbest represents the optimal action value function. Qbest is calculated as shown in Equation (4):

However, in practical application, DRL algorithm leads to the increase of spatial states due to more features and attributes of data states. This improves the difficulty of computation and data storage, which affects the analysis results [22,23]. The K-means algorithm is a data clustering analysis algorithm for iterative solving [24]. This algorithm is able to divide the state space into different clusters by K-means, and by learning each cluster, it will reduce the amount of computation and improve the learning efficiency [25]. The basic flow of the K-means algorithm is shown in Fig. 2.

Fig. 2. Flowchart of the K-means algorithm.

In Fig. 2, the K-means algorithm first requires initialization of the data. K data points are selected as initial cluster centroids in the collected data. Then the distance of each data point from the center point of each cluster is calculated, and the data points are assigned to the cluster with the closest distance to it. After the data is assigned, the average value of all data points in each cluster is calculated. The calculated value is used as the new cluster center point. The above steps are repeated until the maximum number of iterations is reached or the cluster centroids no longer change. Finally, the clustering results are output to obtain the cluster label of each data point and complete the data clustering. In this regard, the calculation of the clustering center selection is shown in Equation (5):

In Equation (5), ck denotes the kth clustering center. The distance for data point assignment is calculated using Euclidean distance. It is calculated as shown in Equation (6):

In Equation (6), d represents the dimension of the data points. ck1 represents the data in the clustering center of the first class of data. xi1 represents the clustering center of the first class of data. The data points are assigned in the way as shown in Equation (7):

In Equation (7), Ci denotes the data assigned to the cluster center. The cluster center position, that is, the cluster centroid position, is calculated as shown in Equation (8):

In Equation (8), |ck| represents the number of data points contained in the clustering center ck. ∑xiεckxi represents the sum of data points contained in all clustering centers. The above calculations are used to distribute the collected data into different clusters to reduce the computational effort of datasets. The DRL algorithm is optimized using the K-means algorithm. The data points are assigned to different clusters through data point allocation, cluster centroid update, and iteration in the K-means algorithm as a way to reduce the subsequent computational complexity. DRL is then used to analyze the allocated data to reduce the data storage space and improve the calculation speed. The process is shown in Fig. 3.

Fig. 3. K-means-DRL algorithm flowchart.

B.K-MEANS-DRL-BASED MODEL FOR ANALYZING MEDICATION PATTERNS

To solve the problem of inaccurate analysis of drug dosage in the current medication analysis model, this study constructs a medication law analysis model using the K-means-DRL algorithm mentioned in the previous section to analyze the dosage and taking pattern of various drugs. The basic framework of the medication pattern analysis model based on the K-means-DRL algorithm is shown in Fig. 4.

Fig. 4. K-means-DRL medication pattern analysis model.

In Fig. 4, the model consists of five parts: data collection, data preprocessing, model construction, model evaluation, and result interpretation and application. In the data collection section, medication information such as drug use, patient information, patient treatment status, and treatment effects needs to be collected. These data generally come from patients’ clinical records, electronic medical records, and drug management systems. In the data preprocessing section, the collected data are clarified, transformed, and integrated to ensure the accuracy and consistency of the data. Duplicates in the data are removed, and outliers are handled. Afterward, a suitable model is selected for modeling based on the actual solution problem. In this study, the K-means-DRL algorithm is used to construct an analytical model to analyze the dosage and taking pattern of drugs. Then the constructed model is evaluated using historical data to verify the accuracy and reliability of the model, and the model is optimized and adjusted. Finally, the optimal strategy obtained from the analytical model is interpreted, and useful information is extracted to optimize the medication regimen and improve the therapeutic effect. In this process, for the determination of the content of the raw drug, the direct titration method is used. Its calculation formula is shown in Equation (9):

In Equation (9), V represents the volume of titrant consumed for titration. T represents the degree of titration. T represents the concentration correction factor of the titrant. W represents the weighed volume of the pharmaceutical agent specimen. The content determination of the pharmaceutical drug uses the experimental violet spectrophotometric method for calculation. The formula is shown in Equation (10).

In Equation (10), Cx represents the concentration of the pharmaceutical reagent solution. Ax represents the absorbance of the agent test solution. Ar represents the absorbance of the control solution. Cr represents the concentration of the control solution. The compositional content and dosage content of different drugs are obtained by the above calculations. In this medication analysis model, the principle of using K-means-DRL algorithm is shown in Fig. 5.

Fig. 5. K-means-DRL application principle.

In Fig. 5, in the medication analysis model, the K-means-DRL module receives the incoming various medication data. The various data are first processed into clusters in the K-means module. That is, the collected medication data are divided into different clusters based on drug properties or medication time through operations such as data point assignment and cluster center point update. After all the medication data are assigned, these assigned clusters are entered into the DRL module. In this module, the problem to be solved, that is, analyzing the medication pattern of drugs, is first entered. Then the states, actions, and reward mechanisms of DRL are defined. Then the appropriate learning algorithm is selected according to the actual situation. Since this research is about the analysis of drug medication patterns and the medication scenarios are complex, the Q-Learning algorithm, which can learn the optimal strategy in different scenarios, is chosen for model construction. The algorithm is used to process the complex data and represent the characteristics of the data and the relationship between them. Then a data action is randomly executed according to the initial strategy. The surroundings return a reward signal as well as observational data based on that action, and the next action is performed based on the returned reward signal. The process is repeated until all medication data actions are completed. In this process, the Q-value in the Q-Learning algorithm is updated by the formula shown in Equation (11):

In Equation (11), Q1(s1,a1) represents the expected return of action a1 taken under state s1. ∂ represents the learning rate. r1 represents the immediate payoff of action a1 taken under state s1. maxatQ1t(s1t,a1t) represents the maximum expected return of a different action a1t taken under the new state s1t. The medication analysis model is used to analyze the medication pattern in the treatment of diabetic cardiomyopathy. The model is used to analyze the dosing pattern of the disease so as to ensure the efficacy and safety of the drugs, reduce the side effects of the drugs, improve the therapeutic effect, and promote the health of the patients.

A.PERFORMANCE ANALYSIS OF THE K-MEANS-DRL ALGORITHM

To analyze the predictive superiority of the proposed K-means-DRL algorithm, comparative experiments are conducted between the K-means-DRL algorithm and backpropagation algorithm long short-term memory (BP-LSTM), least squares adaptation support vector machine (LS-SVM) algorithm, and sparrow search algorithm variational mode decomposition (SSA-VMD) algorithm. The environmental configuration during the experiment is shown in Table I.

Table I. Experimental environment configuration

For the experiments, the Iris dataset is chosen for the experiments. The Iris dataset contains 150 pieces of data, with a total of 50 pieces of data, each of which contains four features. The prediction performance of the four algorithms is analyzed using the above experimental environment and dataset. The prediction accuracy of the four algorithms is analyzed. The results are shown in Fig. 6.

Fig. 6. Comparison of algorithm prediction accuracy.

In Fig. 6(a), the distribution of the predicted values of the K-means-DRL algorithm are within the range of the true values, and the algorithm has high prediction accuracy. In Fig. 6(b), when the BP-LSTM algorithm predicts the dataset, there are very few predicted values that are not within the range of true values. However, most of the predicted values overlap with the range of true values, and the algorithm’s prediction accuracy is lower than that of the K-means-DRL algorithm. In Figs. 6(c) and 6(d), when the LS-SVM algorithm and the SSA-VMD algorithm predicted the data, the two algorithms had most of the distribution of the predicted values not within the range of the true values. This result shows that the K-means-DRL algorithm has the highest prediction accuracy among the four algorithms. Moreover, the prediction effect is significantly better than the other algorithms to predict the data accurately. Then the prediction speed and prediction error rate of the four algorithms are compared. The results are shown in Fig. 7.

Fig. 7. Comparison of prediction error and prediction time.

In Fig. 7(a), the K-means-DRL algorithm has the lowest prediction error among the four algorithms. After 300 repeated experiments, the average prediction error rate of this algorithm is only 1.2%. The BP-LSTM algorithm has an average prediction error rate of 1.6%. The LS-SVM algorithm and SSA-VMD algorithm have an average prediction error rate of 2.7% and 3.6%, respectively, which is much higher than the K-means-DRL algorithm proposed in the study. In Fig. 7(b), the SSA-VMD algorithm has the slowest speed of only 2.8 bps for data prediction, and the K-means-DRL algorithm has the fastest prediction speed of 8.2 bps. The prediction speeds of the BP-LSTM algorithm and the LS-SVM algorithm are 6.6 bps and 4.5 bps, respectively, which are higher than the SSA-VMD algorithm but lower than the K-means-DRL algorithm. From the above experimental results, it can be concluded that the K-means-DRL algorithm can substantially improve the speed of data prediction, thus reducing the consumption of computational resources and optimizing the allocation of resources. Finally, the cross-entropy of the four algorithms as well as the check-full rate are compared, and the results are shown in Fig. 8.

Fig. 8. Comparison of algorithm recall and cross-entropy.

In Fig. 8(a), among the four algorithms, the one with the highest detection rate is the K-means-DRL algorithm, which has a detection rate of 98.7%. The lowest checking rate is SSA-VMD algorithm, which has a checking rate of only 83.2%. Algorithmic cross-entropy is often used as a loss function in deep learning to indicate the difference between the algorithm’s predicted results and the actual results. The smaller the value, the more stable the algorithm is. In Fig. 8(b), the average values of cross-entropy of K-means-DRL algorithm, BP-LSTM algorithm, LS-SVM algorithm, and SSA-VMD algorithm are 0.53, 1.78, 2.43, and 3.91, respectively. Among the four algorithms, the K-means-DRL algorithm has the lowest cross-entropy. This result indicates that the K-means-DRL algorithm has the best stability among the four algorithms. Overall, the K-means-DRL algorithm proposed in the study has the highest prediction accuracy, the lowest prediction error, and the shortest prediction time. The overall performance of the algorithm is significantly better than other algorithms. Therefore, this study utilizes the algorithm to construct an analytical model with the aim of improving the analytical accuracy and analytical speed of the analytical model by this algorithm.

B.EFFECTIVENESS OF THE APPLICATION OF K-MEANS-DRL ANALYTICAL MODELING

After validating the predictive effectiveness of the algorithm proposed in the study, the performance of the analytical model based on the algorithm is analyzed. The study compares the performance of K-means-DRL analysis model with support vector machine principal components analysis (SVM-PCA), random forest logistic regression analysis (RF-Logistic), and least squares support vector regression (LS-SVR). The clinical records and electronic medical records of 100 patients with diabetic cardiomyopathy in a hospital over a period of 6 months are selected as the experimental dataset. The performance of the models is verified by analyzing the samples in the dataset. The inclusion criteria of patients are as follows: 1. Meet the diagnostic criteria of diabetic cardiomyopathy; 2. The course of diabetic cardiomyopathy ranges from 6 months to several years; 3. The patient is between 30 and 45 years old; 4. The patient’s gender, body mass index, and other basic information are complete; 6. The patient has no family history of genetic disease and other complications. 7. The patient is willing to participate, and the guardians are aware of it. The patient’s acceptance criteria are as follows: 1. The diagnosis of diabetic cardiomyopathy is not clear; 2. The condition is too serious and the stability of the condition is poor; 3. The patient’s treatment record is incomplete; 4. The patient has other serious diseases; 5. The patient is allergic or intolerant to TCM. Selecting 100 patients, the number of books is relatively moderate and easy to manage, and the sample size is too large to extend the research cycle. Selecting 100 patients is helpful to complete the study in a relatively short time. The 100 medical records are obtained through the electronic medical record system in the hospital cooperating with medical institutions, including the basic characteristics such as the patient’s name and age, as well as the characteristics of clinical variables such as the patient’s admission date, discharge date, medical history, treatment plan, rehabilitation variables such as the patient’s recovery time, rehabilitation effect, and resource utilization variables such as the patient’s medical expenses and drug use. In order to ensure the representativeness of the data, cases with different degrees of illness are selected at different time periods in the outpatient department and inpatient department of the hospital. Then the validity of the records is verified by quality assessment of the medical records and checking with the original medical records. In the treatment of patients with diabetic cardiomyopathy, clinical benefits refer to the improvement of patients’ health status through treatment, such as the reduction of symptoms, the recovery of cardiac function, and the improvement of quality of life. Resource utilization refers to the medical expenses, hospitalization time, examination, treatment equipment, and other resources used by patients in the medical process. Patient rehabilitation refers to the recovery of heart function, disappearance of symptoms, improvement of quality of life, etc. The time-consuming analysis and accuracy analysis of the four models are firstly compared. The results are shown in Fig. 9.

Fig. 9. Comparison of model analysis time and analysis accuracy.

In Fig. 9(a), the analysis accuracy of the K-means-DRL analysis model is 99.5% on average. Moreover, the average time consumed by this model to analyze the patients’ medication patterns is only 1.3 s. In Fig. 9(b), the analysis accuracy of SVM-PCA analysis model is 90.2%. Furthermore, the average time consumed for the analysis of the SVM-PCA model is 1.9 s, which is an increase of 0.6 s compared to the time of the K-means-DRL model. In Fig. 9(c) and Fig. 9(d), the analytical accuracy of both RF-Logistic model and LS-SVR model is decreasing. The analysis accuracy of the RF-Logistic model is 81.2% and that of the LS-SVR model is 80.4%. The analysis time consumed by the RF-Logistic model is 2.3 s, and the analysis time of the LS-SVR model reaches 3.2 s. From this result, it can be concluded that the K-means-DRL model has the highest accuracy in analyzing the medication exchange pattern and the shortest time consumed in the analysis. The results of the analysis of the four models for diabetic cardiomyopathy medication are then compared. The results are shown in Table II.

Table II. Comparison of medication results

In Table II, the TCMs for patients with diabetic cardiomyopathy are mainly blood-activating and stasis-removing, qi-tonifying, yin-tonifying, and heat-clearing herbs. Moreover, among the four models, the K-means-DRL analysis model proposed by the study is the most accurate in analyzing the types of herbal medications used in diabetic cardiomyopathy. All of its discrepancies with the actual situation are within 1%. Whereas, the errors of the other models in analyzing the types of Chinese medicines used are all above 1%. Moreover, the largest error value of the LS-SVR model for analyzing the types of Chinese medicines used in patients with diabetic cardiomyopathy reaches 4.2%, and the accuracy of the analysis of the types of medicines is low. Then the changes of medical resource utilization and patient rehabilitation time in diabetic cardiomyopathy are compared after using the four models. The results are shown in Fig. 10.

Fig. 10. Comparison of medical resource utilization and patient rehabilitation time.

In Fig. 10(a), medical resource utilization is only 64.3% before using the analytical models. Whereas, after analyzing the medication pattern of diabetic cardiomyopathy patients using all four analytical models, the medical resource utilization is improving. Among them, the medical resource utilization of K-means-DRL model reaches 98.6%, and the resource utilization rate is improved by 34.3%. The SVM-PCA, RF-Logistic, and LS-SVR models, on the other hand, can improve the resource utilization rate, but their enhancement is smaller. The medical resource utilization of SVM-PCA model, RF-Logistic model, and LS-SVR model are 82.4%, 71.5%, and 68.9% respectively. Compared with before the use of the models, their medical resource utilization increases by 18.1%, 7.2%, and 4.3%, respectively. In Fig. 10(b), the recovery time of patients with diabetic cardiomyopathy is decreasing after using all four models. Among them, the K-means-DRL model is able to reduce the recovery time of patients by 39.3%. While the SVM-PCA model, RF-Logistic model, and LS-SVR model reduce the recovery time of the patients by 30.2%, 22.7%, and 17.5%, respectively, which is much lower than the K-means-DRL model. In order to verify the accuracy of the proposed method again, the study collects patients’ cases from multiple hospitals through medical institutions. Among them, 100 cases of diabetic cardiomyopathy patients in primary hospitals, secondary hospitals, and tertiary hospitals are collected as validation datasets. The variables in this dataset are the same as the test dataset. The K-means-DRL model proposed in the study is used for analysis. The results of medication analysis are shown in Table III.

Table III. Analysis of drug use patterns

It can be seen from Table III that when the K-means-DRL model is used to verify and analyze the medication rules of patients with diabetic cardiomyopathy in different hospitals, the analysis errors of medication types and doses are less than 1%. From the above results, it can be concluded that the K-means-DRL medication pattern analysis model proposed in the study has the highest accuracy and the smallest error in analyzing the medication pattern of patients with diabetic cardiomyopathy. Furthermore, the utilization rate of medical resources is the highest, and the patient rehabilitation time is reduced the most after using this model. The practical application of the model is optimal.