Sentiment Analysis: iPhone Reviews
Analyzing customer feedback is crucial for a company’s revenue and profitability as customer reviews can greatly impact a business. By studying customer feedback, companies can identify areas that require improvement in their branding, products, and services. This helps in making informed decisions that lead to a reduction in churn rate. An important business intelligence tool that can be used is SENTIMENT ANALYSIS also referred to as Opinion mining.
Sentiment analysis refers to the process of identifying and categorizing opinions conveyed in a given text, with the aim of ascertaining whether the customer’s stance is positive, negative, or neutral.
Case Study
For this analysis, the goal is to assess the overall satisfaction of customers with Apple’s iPhone, utilizing the data from the DataDNA iPhone Reviews Dataset Challenge.
Dataset
The dataset, containing over 5,000 reviews of the Apple iPhone from Amazon.com, offers valuable insights and extensive opinions that can aid in gauging the present customer sentiment towards the product. Through exceptional review ratings and elaborative text reviews, readers can gain an understanding of why customers appreciated or disliked the product, providing crucial feedback for the market, including details on what worked (or didn’t work) well.
Analysis
I will walk you through the complete process of conducting sentiment analysis on a substantial volume of data. While I utilized a Jupyter Notebook for all of the analysis and visualization, any Python IDE will suffice. Note that all packages have been previously imported.
Step 1: Read the Data frame
import pandas as pd
df = pd.read_csv("DataDNA March 2023.csv")
df.head()Checking the first five rows of the data frame:
It is evident from the data frame that it comprises various details related to products, users, and their reviews. The crucial data for our analysis will be “review_title”, “review_text”, and “review_rating.”
review_text — The text of the review. (String)
review_rating — The rating given to the product by the reviewer. (Integer)
review_title — The title of the review. (String)
Step 2: Data Analysis
We will examine the “review_rating” variable and determine whether the majority of customer ratings are positive or negative.
The Plotly library needs to be installed for the analysis.
# Product Scores
fig = px.histogram(df, x="review_rating", category_orders={"review_rating": ["1.0", "2.0", "3.0", "4.0", "5.0"]})
fig.update_traces(marker_color="turquoise",marker_line_color='rgb(8,48,107)',
marker_line_width=1.5)
fig.update_layout(title_text='Product rating')
fig.show()
It is evident from here that the majority of customer ratings are positive. This suggests that the reviews will likely be predominantly positive as well, which will be analyzed shortly.
We can now generate word clouds to visualize the most commonly used words in the reviews.
import nltk
from nltk.corpus import stopwords
from wordcloud import WordCloud,STOPWORDS# Create stopword list:
stopwords = set(STOPWORDS)
stopwords.update(["br", "href"])
textt = " ".join(review for review in df.review_text)
wordcloud = WordCloud(stopwords=stopwords).generate(textt)
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.savefig('wordcloud11.png')
plt.show()
From the reviews, common terms such as “good,” “awesome,” “best,” “amazing,” and “Amazon” are prevalent. These terms predominantly convey a positive sentiment, suggesting that a majority of the reviews in the dataset express positivity.
Step 3: Classifying Reviews
The next phase involves categorizing the reviews as either “positive” or “negative” for training our sentiment classification model.
Reviews with a “Score” > 3 will be labeled as +1, indicating positive sentiment. On the other hand, reviews with a “Score” < 3 will be labeled as 0, indicating a negative sentiment.
Since reviews with a “Score” of 3 are neutral, we will exclude them from our analysis. The focus of our model is to differentiate between positive and negative reviews i.e. customers generally satisfied and dissatisfied with Apple’s iPhone.
df['review_rating'] = df['review_rating'].astype(str).astype(float)
df = df[df['review_rating'] != 3.0]
df['sentiment_val'] = df['review_rating'].apply(lambda rating : +1
if rating > 3 else 0)We can now see a new column called ‘sentiment_val’ in the data frame:
Step 4: More Analysis
Let’s generate word clouds for positive and negative reviews after categorizing them.
To begin with, we will generate two separate data frames, one for positive reviews and another for negative reviews.
# split df - positive and negative sentiment:
positive = df[df['sentiment_val'] == 1]
negative = df[df['sentiment_val'] == 0]Wordcloud — Positive Sentiment
stopwords = set(STOPWORDS)
stopwords.update(["br", "href","good", "iPhone", "phone", "product",
"Amazon", "Apple"]) pos = " ".join(review for review in positive.review_text)
wordcloud2 = WordCloud(stopwords=stopwords).generate(pos)
plt.imshow(wordcloud2, interpolation='bilinear')
plt.savefig('wordcloud31.png')
plt.axis("off")
plt.show()
Wordcloud — Negative Sentiment
neg = " ".join(review for review in negative.review_text)
wordcloud3 = WordCloud(stopwords=stopwords).generate(neg)
plt.imshow(wordcloud3, interpolation='bilinear')
plt.axis("off")
plt.savefig('wordcloud33.png')
plt.show()
As observed earlier, the positive sentiment word cloud contained a plethora of optimistic words, including “excellent,” “awesome,” “great,” “best,” and “amazing.”
The negative sentiment word cloud was filled with mostly negative words, such as “problem,” and “issue.”
Lastly, we can examine the distribution of reviews with sentiment across the dataset:
df['sentiment'] = df['sentiment_val'].replace({0 : 'negative'})
df['sentiment'] = df['sentiment'].replace({1 : 'positive'})
fig = px.histogram(df, x="sentiment")
fig.update_traces(marker_color="indianred",marker_line_color='rgb(8,48,107)',
marker_line_width=1.5)
fig.update_layout(title_text='Product Sentiment')
fig.show()
Based on the above observation, it can be inferred that a larger proportion of customers are generally satisfied with Apple’s iPhone.
Step 5: Model Building
The sentiment analysis model can now be built. Its purpose is to classify reviews as either positive or negative based on the input received. To achieve this, a simple logistic regression model will be trained.
To make predictions using the review_text data, the first step is to eliminate all punctuation marks from the data.
def remove_punctuation(review_text):
final = "".join(u for u in review_text if u not in ("?", ".", ";", ":",
"!",'"'))
return finaldf['review_text'] = df['review_text'].apply(remove_punctuation)
#df = df.dropna(subset=['review_title'])
df['review_title'] = df['review_title'].apply(remove_punctuation)To split the data frame, a new data frame with only two columns — “review_title” and “sentiment_val” containing the target variable should be created.
dfNew = df[['review_title','sentiment_val']]
dfNew.head()
We will now split the data frame into train and test sets. 80% of the data will be used for training, and 20% will be used for testing.
# random split train and test data
index = df.index
df['random_number'] = np.random.randn(len(index))
train = df[df['random_number'] <= 0.8]
test = df[df['random_number'] > 0.8]Create a bag of words
Next, we will use a count vectorizer from the Scikit-learn library.
To enable the logistic regression algorithm to comprehend the text, we must convert the text in our data frame into a bag-of-words model, which creates a sparse matrix of integers. This model counts the number of occurrences of each word and prints it.
# count vectorizer:
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(token_pattern=r'\b\w+\b')
train_matrix = vectorizer.fit_transform(train['review_title'])
test_matrix = vectorizer.transform(test['review_title'])Import Logistic Regression
# Logistic Regression
from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()Split target and independent variables
#Split target and independent variables
X_train = train_matrix
X_test = test_matrix
y_train = train['sentiment_val']
y_test = test['sentiment_val']Fit model on data and Make predictions
#Fit model on data
lr.fit(X_train,y_train)#Make predictions
predictions = lr.predict(X_test)We have successfully generated predictions using the model.
Step 6: Testing
Now, we can test the accuracy of our model!
# find accuracy, precision, recall:
from sklearn.metrics import confusion_matrix,classification_report
new = np.asarray(y_test)
confusion_matrix(predictions,y_test)print(classification_report(predictions,y_test))The classification report:
The model’s test data accuracy stands at approximately 94%, which is quite impressive, given that there was no feature extraction or extensive preprocessing performed.
