Introduction and Background
Social media platforms produce vast quantities of data on a daily basis. Users of social media are able to share their thoughts and opinions of various topics easily and conveniently through these platforms. Twitter is one of the world’s leading social media platforms that is used by millions of users. By analyzing Twitter data, interesting insights regarding the dynamics of online communities may be discovered.
However, Twitter data can not be easily processed due to the sheer amount of data generated daily by its users. A problem is also faced when identifying meaningful tweets as sometimes user-generated content may be nonsensical and it can be difficult to extract meaning from certain tweets without context.
Problem Statement
The project focused on building a system that could aid researchers in analyzing large amounts of Twitter data in usable time. The system should be able to find frequent relationships in tweets relatively fast and be able to visualize these results in a way such that meaningful information can be extracted from the dataset provided.
Project Goals and Significance
Social media has become an integral part of society, and generates massive amounts of data on a daily basis. By employing data mining techniques on these datasets, researchers and businesses can gain valuable insight into the dynamics of online communities and individual users. The detection of interesting associations between topics and words is one of these insights. Unfortunately, the analysis of large amounts of social media data requires significant amounts of processing power, which can often be costly to both researchers and businesses. Maintaining a cluster of your own, or renting a cluster on a permanent basis from providers such as Amazon Web Services (AWS) is an expensive exercise. Additionally, extracting meaningful information from the results of the analysis can be a time-consuming and tedious process.The goal of this project was to build a system that can utilize a cluster to do a scalable analysis of large amounts of Twitter data, as well as present the results in a useful manner.
Project Breakdown
The project was divided into three systems that make up the SANCTUM system (Scalable Analysis on a Cluster of Twitter-Data Using Mining).
- A Web-based front-end interface that is able to specify and schedule jobs. The interface is also able to visualize the results of completed jobs in meaningful ways.
- An information retrieval module that is able to index the data and provide a smaller subset of the data that adheres to a searched term or phrase.
- A rule mining module that makes use of association rule mining to generate relationships on frequently occuring words and topics in the data.