Twitter Text mining/scraping with R

As a data analyst one should be able to analyse the current trends in social media. This is useful in decision making and insight on people's views. Currently there is teachers strike in Kenya which has lasted for four weeks. Schools are  closed and  the strike is still as per the time of updating this blog. I won't go to the logistic about the strike but I can use my skills to determine people's views, correlation between the strike and other things. Take for example the president's name, the opposition leaders, the teachers and the children, all of them are affected from the  analysis. Check out the ggplot product below to gain more insight.

The data sets are just tweets about  the teachers strike. The bar graph shows more insight in that among the tweets users mention different words during tweets and the frequency of these words is what I have used to plot this graph. Also check out the word cloud below.

From the word cloud it is clear that strike was the word of the tweets. Followed by other words like teachers, the impeachment thereat from the opposition, UKenyatta the current president who is facing impeachment threats, Raila is the opposition leader, Ruto is the deputy president, Mutahi ngunyi is a well known political analyst who is a staunch supporter of President Kenyatta. There are so many words from the tweets but the word cloud only had specific frequency I had set.
For more of this kind of analysis follow me on twitter @oscar_doe.

Twitter text mining is a very well documented area of data mining. I will therefore explain little of it here just to make this article comprehensive. In twitter text mining I will be using the package twitter and tm in R. The main function will be searchTwitteR{} because of its ability to data mine with specific details such as location, dates, language and many more.

#Load packages

required(tm)

required (twitteR)

Libarary(wordcloud)

#Authorize twitter API

reqURL<-"https://api.twitter.com/oauth/request_token"

access_token<-" refer from the twitter app"

access_token_secret<-" refer from the twitter app"

apiKey<-" refer from the twitter app"

apiSecret<-" refer from the twitter app"

setup_twitter_oauth(apiKey,apiSecret,access_token,access_token_secret)

#Scrape twitter using searchTwitter function

#This function picks tweets with Brexit

#The language is null to accommodate other languages

#Date has been set from 23-june-2016

#location is actually worldwide

tweets<-searchTwitter("Brexit",n=1000,lang = NULL,since = "2016-06-23",until=NULL,locale=NULL,geocode=NULL,sinceID=NULL,maxID=NULL)

tweets_text<-sapply(tweets,function(x)x$getText())

Now we have the tweets but there other things that is contained in the tweets that we do not need. We do not need emojis, stop words like, the, why and many more. So we now proceed to removing this unwanted sets in our data.

#removing emojis in r

tweets1<-iconv(tweets_text,'UTF-8','ASCII')

Bulid a corpus and save the tweets as data frame

tweets_corpus=Corpus(VectorSource(tweets1))

tdm=TermDocumentMatrix(

  tweets_corpus,

  control=list(

    removePunctuation=TRUE,stopwords=c("England","Wales","Scotland",stopwords("english")),

    removeNumbers=TRUE,tolower=TRUE)

)

m=as.matrix(tdm)

# get word counts in decreasing order

word_freqs=sort(rowSums(m),decreasing=TRUE)

# create a data frame with words and their frequencies

dm=data.frame(word = names(word_freqs),freq=word_freqs)

wordcloud(dm$word,dm$freq,random.order=FALSE,colors=brewer.pal(8,"Dark2"))

returning the number of tweets scrapped

length(tweets)

Market Research

Market research is the process of gathering, analyzing and interpreting information about a specific market, product or service to be offered for sale in the future. It also helps to explore the past, present and potential customers for the product or service, it research into characteristics, spending habits, location and needs of your business’s market, the industry and the competitors.

We all believe that accurate and thorough information is the foundation of all successful business ventures because it provides a wealth of information about prospective and existing customers, the competition, and the industry.

At Unitary Analytics we take market research as a key factor in maintaining competitiveness over the potential competitors. Our experience experts will conduct a market research for you, therefore providing you with important information to identify and analyze the market need, market size and competition. This will allow prospective and business owners to determine the feasibility of the business before committing substantial resources to that venture and therefore provides relevant data to help solve challenges that the business will face.