Crisis informatics is a new area of research that studies the use of technology in and around disaster events by members of the public. In particular, crisis informatics studies the digital behaviors of people accessing social media websites as they collectively generate, consume and disseminate information about a crisis or disaster event. A key challenge for crisis informatics researchers is capturing these activities, as the information can appear without warning and is often ephemeral. For instance, on the Twitter micro-blogging service, only the last 3200 status updates of a user are stored, older updates are simply deleted. Unfortunately, crisis informatics researchers have observed users who generate more than 3200 updates during a single event, increasing the importance of being able to capture their information while it is being generated.

This thesis examines the issues surrounding the design and development of a data collection mechanism for social media websites with the constraints of crisis informatics research in mind. The system is designed to capture data while it is being generated, storing it in a persistent database for later analysis. It is designed to scale to multiple machines to ensure that many websites can be monitored at once or to ensure that all the data being generated from a single website or service (such as Twitter) can be captured without loss.

While our data collection service provides a generic framework for searching social media websites, for the purposes of this thesis, we focused on building modules for searching Twitter. Our Twitter integration allows for two types of search, one based on keyword and one based on gathering the tweets of users who contributed to the result set of a keyword search. We call the latter search a contextual search as it grabs the tweets that surround the ones that match the given keyword search.

Our data collection service provides search capabilities on top of the results that it retrieves, allowing crisis informatics researchers the ability to verify the accuracy of their search terms.

This system has seen production use in gathering data across a major disaster event that occurred in January of 2010. The data gathered by our service provided data sets with known collection-related parameters, allowing them to be useful for later empirical and ethnographic research activities by crisis informatics researchers.