Intelligent Text Sentiment Analysis of Emotional Beauty in the English Translation: Based on TB-LSTM Algorithm

A.INTEGRATION OF THEMATIC AND LEXICAL INFORMATION

Topic modeling is one of the popular methods for learning text representation, which has been frequently used in natural language processing fields such as text clustering, text partitioning, and sentiment analysis in recent years. It is used as an external auxiliary knowledge to improve the model complexity and achieve better classification results. LDA is a well-known topic model in text modeling. LDA can mine topic and is unsupervised, not relying on training samples. It does not have domain transfer issues and has good domain adaptability. Therefore, this study utilizes the probability computing power of LDA to obtain the topic information of the document [18–20]. LDA assumes that the total number of documents in a document set is D, with k topics and v non-repeating words. Figure 2 shows the principle of LDA. It takes text as input and obtains the probability of each text appearing in the topic and the probability distribution of each word appearing in each topic through statistical analysis of the text. LDA is a complete Bayesian probability graph model. When inferring parameters, it is necessary to first obtain a posterior distribution and then use Gibbs sampling method to estimate its parameters until it converges. The optimal distribution of text themes and keywords is ultimately determined.

Fig. 2. LDA topic model.

Firstly, it should transform the topic vocabulary distribution matrix W extracted from the LDA topic model into a low-dimensional topic vocabulary embedding matrix T through a fully connected layer. Thus, the topic information representation vector dk with the same dimension as the lexical hidden semantic information representation vector in equation (1) is obtained.

In equation (1), tanh is a function. Pβ is the probability distribution of the topic vocabulary. Then, a low-dimensional topic vocabulary embedding matrix is set as M. The implicit semantic information matrix of the vocabulary is N. The relationship ∂ between a single vocabulary and the topic is calculated. Based on the ∂, the matrix M is reconstructed to obtain the embedding matrix Q for vocabulary topics. Finally, N and Q are connected together to obtain a lexical level semantic representation, denoted as K. In this process, the first step is to preprocess the input document and organize it into a sequence of d=(v1,v2,v3,…,vt,…,vT) form words. Then, the vocabulary in the sequence is processed into word vectors through Word2vec and then input into Bi-LSTM to obtain the matrix N.

LSTM is a variant of recurrent neural networks. It overcomes the shortcomings of traditional methods, which can handle long sequence data [21]. Figure 3 shows the basic units of LSTM. It is a long-term cellular state that consists of three components, namely three gates. By controlling the “gate,” it achieves an organic fusion of short-term memory and long-term memory, to some extent overcoming the loss. When information passes through the forgotten gate, it determines which information to ignore and activates the function to indicate whether the output can be maintained. Then, the information that needs to be saved for the subsequent calculations. Next, the previous data are integrated to form new data, and the data that have already been used in the previous layer are deleted. Then, the output information is filtered using the Sigmoid function. Finally, the Tanh function is used to execute the output [22].

Fig. 3. Basic unit of LSTM.

LSTM can effectively overcome the shortcomings of traditional recurrent neural networks such as information loss and excessive gradients. However, it can only capture the content mentioned earlier well, while ignoring the content in the following text [23]. Therefore, this study uses Bi-LSTM as a benchmark to train it and improve its processing ability for complex text data, while overcoming its inherent shortcomings. Figure 4 shows the structure of Bi-LSTM, consisting of a forward LSTM and a backward LSTM. This algorithm utilizes two unrelated LSTMs to perform data prediction in both forward and backward directions, and it combines the forward and reverse input sequences at the network output layer to obtain the final output value [24–26].

Fig. 4. Bi-LSTM network.

Fig. 5. Sentence sequence processing.

When using Bi-LSTM to obtain matrix N, equation (2) is the forward hidden state hF→ of the vocabulary calculated by the network.

In equation (2), hF represents the hidden semantic information vector of the vocabulary, representing all information centered around the vocabulary. LSTMF(vt)→ represents the forward LSTM unit. t is the t-th vocabulary. Equation (3) represents the backward hidden state hF← of vocabulary.

In equation (3), LSTMB(vt)← represents the backward LSTM unit. The final output is ht=[hF→,hB←]. Then, the relationship ∂t,k between the topic information vector and the hidden state vector for each word can be obtained in equation (4).

Then, the topic feature vector qt of vocabulary is calculated in equation (5).

In equation (5), tk represents the low-dimensional topic embedding. Then, qt and ht are concatenated to obtain a representation vector ct that integrates topic information and vocabulary level semantics, which is calculated in equation (6).

B.SENTENCE SEQUENCE PROCESSING BASED ON DOUBLE-LAYER ATTENTION MECHANISM

In the text, emotions can be expressed through the selection of words, the use of tone, sentence structure, and modifiers. Positive emotions may include words such as “like,” “happy,” and “beautiful,” while negative emotions may include words such as “sad,” “angry,” and “painful.” Traditional sentiment analysis methods often convert all the text in the sample set into word sequences and then train the entire word sequence as a sample. This processing method ignores an important component of the discourse: sentence. Its implicit information cannot be ignored [27]. In response to this issue, this study intends to adopt a hierarchical sequence pattern combined with attention mechanism, using text as units by sentence and inputting it into a neural network for learning. Attention mechanisms are adopted at the levels of words and sentences, assigning different weights to words and sentences. Word-level topic information and sequence-level attention mechanisms are combined with LDA topic model and Bi-LSTM. At the lexical level, potential thematic information is introduced into the semantic representation at the lexical level to improve the effectiveness of lexical representation and apply it to multidimensional sentiment classification.

The steps for sentence sequence processing are shown in Figure 5. First, the input document is divided into sentence sequences using punctuation marks as segmentation points. Then, word segmentation is performed, and sentence’s all vocabularies are represented as word vectors using Word2vec. Sentence’s word vectors are input into Bi-LSTM to get the hiding state vector matrix N of vocabulary. Then, for each word in each sentence, the topic vector matrix T is reconstructed to obtain the word topic embedding matrix for each word in each sentence. Afterward, the word topic embedding vector qlt is obtained by repeating equations (4) and (5). The concatenation of qlt and word hidden state vectors results in the t-th vocabulary’s word-level semantic vector clt in the l-th sentence that integrates topic information.

The organized document is represented by equation (7).

In equation (7), l is the maximum number of sentences. Each sentence is segmented to remove useless stop words. The sentence sequence is further organized into a word sequence of equation (8).

However, not every word has equal importance in representing the sentence’s meaning. Therefore, a sequence-level attention mechanism is used afterward to further integrate text semantics into the sequence dimension, thereby constructing text sentiment representation. Firstly, the sentence is expressed as a vector sl, which is represented by equation (9).

In equation (9), wlt is the contribution weight of a vocabulary to the corresponding sentence, which is calculated in equation (10).

In equation (10), av is the parameter that the model needs to learn. eltT is a hidden vector representation of klt obtained using a single-layer feed-forward network, calculated in equation (11).

In equation (11), bv is the parameter that needs to be learned. Similarly, for documents, not every sentence contributes equally to the representation of the document’s meaning. Sentence sequence representation is integrated into text representation by using attention mechanism, and document’s vector representation F is finally obtained through equation (12).

In equation (12), φl represents the contribution of a single sentence in the entire document, calculated in equation (13).

In equation (13), as is the learning parameter of the model. el is a hidden representation obtained from a sentence sequence through a single-layer feed-forward network, calculated in equation (14).

In equation (14), bs and Ws are learning parameters. The sentiment classifier uses the document sentiment semantic vector F to derive the sentiment label y, and then maps it to a probabilistic form using the Sigmoid function. The training of this research model adopts an end-to-end backpropagation training method, with the objective function being the cross-entropy cost function. Assuming y as the known classifying category and y¯ as the predicted one, it aims to make the cross-entropy of y and y¯ minimized, as displayed in equation (15).

Based on the above research, this study constructs TB-LSTM in Fig. 6. It has four parts: LDA, classifier, sequence-level dual layer attention mechanism, and word-level topic information. Based on multi-level features of text, this model improves the expression efficiency of vocabulary level by integrating topic information extracted by LDA [28]. By introducing attention mechanisms, deep mining at the vocabulary and sequence levels can be implemented, improving the recognition accuracy and generalization ability of the model. Based on a text dictionary, LDA mines topics and analyzes the implicit topics to obtain a potential topic vector matrix. On this basis, the thematic information is transmitted to the lexical level thematic information attention mechanism to expand the expression at the lexical level. Then, the text is structured and transformed into a word sequence. Relying on the Word2vec software, the word sequence is transformed into a word vector matrix, which is then introduced into Bi-LSTM to obtain an implicit representation matrix of the word. Then, under the attention mechanism of topic information, semantic information expression at the word level is obtained based on the association between topic information and implicit representations. Finally, this attention mechanism at the word level is introduced into topic information sentence’s semantic representation, forming a semantic representation at the semantic level. On this basis, a sentiment classifier is used to infer sentiment labels for the text.

Fig. 6. TB-LSTM model.

When analyzing the emotion of English text sentences, the same emotion word may express different emotional tendencies in different contexts, that is, there exists the problem of emotion ambiguity of emotion words in different sentences. Therefore, the formula (16) is designed to calculate the contribution value of each word to the affective tendency of the sentence, which is represented by the score value α, whose value range is [0,1].

Since α is related to the context of the target word, context vector A is introduced, and A is uniformly initialized during the training process. At last, weighted sum of αt and ht is used to obtain a new feature representing cL, which is the final vector of the attention layer. The calculation formula is shown as equation (17).

For example, the word “short” in the sentence “That battery has a very short life.” expresses the writer’s negative emotion toward “That battery.” However, the sentence “Customer service solved the problem in a very short time.” expresses the author’s positive emotion toward “Customer service.” According to the calculation result of equation (17), the final output feature vector contains the emotional feature information of the word and the calculated weight, which is used to judge the importance of the word “short.”

A.EXPERIMENTAL ENVIRONMENT SETTING

The development and construction of the models require an environment that can support the completion of experiments. Table I shows the experimental and development environments for this experiment.

Table I. Experimental environments and development environments

Before the experiment began, text preprocessing operations were performed on all input documents, including text segmentation, data cleaning, and removing stop words. Because the number of topics k in the LDA topic model has a crucial impact on the proposed model, the first step is to conduct parameter validation experiments. k is set to 25, 30, and 35, respectively. The performance of the model under various indicators is observed to select the optimal parameter value. Then a comparative experiment is conducted to compare the research model with the methods in references [29–31]. The evaluation indicators are Precision (P), Recall (R), and F1-score.

Dataset 1 selects IMDB English movie reviews, and dataset 2 selected CCF-BDCI Chinese news. IMDB is a two-dimensional dataset that includes both positive and negative emotions. After filtering, 24000 compliant label data were obtained, including 13000 negative evaluations and 12000 positive evaluations. 20000 pieces were used as training data, and 4000 pieces were used as testing data. CCF-BDCI contained a total of 5000 eligible label data, and its emotional dimension was three-dimensional, including positive, neutral, and negative emotions. Among them, there were 2300 negative evaluations, 2000 positive evaluations, and 700 neutral evaluations. 4000 pieces were used as training data, and 1000 pieces were used as testing data. Dataset 3 was selected from the twelve-dimensional Twitter Weibo English on GitHub, which contains 4000 text data with twelve emotional tags. 3000 were trained and 1000 were tested. Dataset 4 selects a seven-dimensional self-built dataset from Sina Weibo, which contains 8000 text data with seven emotional labels. 5000 were trained and another 3000 were tested. Table II shows the training parameters of the research model.

Table II. Parameter setting

The ultimate research objective is to apply sentiment analysis to English translation. Therefore, sentiment analysis technology is embedded into translation software to combine the emotional labels output by the model with the semantic understanding module of the translation software to better handle contextual emotional expressions. The sentiment analysis module can use natural language processing techniques to identify information such as emotional polarity (positive, negative, or neutral) and emotional intensity in the text. The translation software is prioritized to process key emotional information. Based on these emotional analysis results, the key emotional information in the original text is determined and made the focus of the translation task. The translation software selected is translator v6.2.620 developed by Google. The accuracy of the evaluation indicators for translation results is selected.

B.MODEL TRAINING RESULTS

Figure 7 shows the training results of parameter k. When k=20, the F1-score, P, and R of the research model were all the highest. Therefore, topic selection can affect the effectiveness of sentiment classification directly. This experiment selected 20 topics to construct a model to achieve the best performance.

Fig. 7. Training results of the parametersk.

Figure 8 shows the training results of each model. The research model could converge quickly, reaching the convergence state in about 100 iterations, and achieving the expected accuracy. The model in reference [29] had the slowest convergence speed, reaching convergence slowly after 250 iterations. The model in reference [30] converged after 160 iterations. The convergence speed of the model in reference [31] was similar to that of the research model, but it did not achieve the expected accuracy. Relatively speaking, the training performance of research models was better.

Fig. 8. The training results of each model.

C.MODEL TEST RESULTS

Figure 9 shows the classification results of each model on the four datasets. The P, R, and F1-score of the research model on dataset 1 were 0.73, 0.65, and 0.69, respectively. Its P, R, and F1-score on dataset 2 were 0.63, 0.72, and 0.7, respectively. Its P, R, and F1-score on dataset 3 were 0.64, 0.6, and 0.63, respectively. Its P, R, and F1-score on dataset 4 were 0.58, 0.65, and 0.61, respectively. Compared with other models, the research model had the highest values for all indicators, proving its best classification performance.

Fig. 9. Classification results for each model on the four datasets.

Figure 10 shows the classification of the research model in different dimensions. Labels 1–12 in Fig. 10(c) represent 12 emotions, including sadness, excitement, worry, surprise, disgust, happiness, deep love, consideration, boredom, anger, emptiness, and neutrality. Labels 1–7 in Fig. 10(d) represent seven emotions: joy, anger, sadness, fear, love, evil, and shock. The research model had the best classification performance on the two-dimensional text data and the worst classification performance on the twelve-dimensional text dataset. The increase in emotional dimensions increased the difficulties in emotion classification. However, the seven-dimensional sentiment classification still maintained good classification results, and the misclassifications were not significantly different from the three-dimensional classification results.

Fig. 10. Classification of the model in this study under different dimensions.

Each dataset is translated and compared with references [32–34]. Figure 11 shows the translation accuracy of the embedded model. From the figure, the translation accuracy of the research model for datasets 1 and 2 was around 0.85, while the translation accuracy of datasets 3 and 4 was relatively low, which were 0.78 and 0.8 respectively. However, compared with the other three models, the translation accuracy of the proposed model was optimal in most cases. On the whole, the translation accuracy of the proposed model is relatively stable and has little fluctuation.

Fig. 11. The translation accuracy of each model.