Big data is the current trendy phrase that covers many different areas.  Big data describes equally well having a huge volume of data generated in a short period of time (like molecular simulations of DNA), having a huge volume of data that needs to be indexed and archived (like PubMed or Web of Science), or wanting to analyze different types of data that wasn’t collected for a given purpose (the CI-BER project uses a variety of data types collected over the years to study a neighborhood in Asheville, NC).

The Virtual School of Computational Science and Engineering (VSCSE) exists to fill the gap in people’s training.  Many people having big data situations were trained as physical scientists, life scientists, or social scientists, not computer science folks.  Even those trained as computer science folks, librarians, or archivists were not trained to deal with this volume of data if that training was completed more than a couple years ago.  VSCSE sponsors workshops and short courses in targeted areas for motivated people who know they need more education, but are not going to go to a formal college or graduate degree program. 

The virtual part refers to the fact that these folks are extremely comfortable with technology.  The Data Intensive summer school held 8 July-10 July 2013, for example, was being teleconferenced to more than 400 participants at 15 listed sites.  I sat in a room at the National Center for Supercomputing Applications in Urbana, IL, but the speakers have been broadcasting from their home sites (e.g., San Diego, Chicago, North Carolina) and questions/comments come from all the sites.  As one speaker put it, it’s kind of weird to speak to people you can’t always see, but I’m ok with that.

Three main problems exist when dealing with big data: storage, interaction, and analysis.  The summer school covered all three aspects.  I will report here on things that may help others who did not sit through the summer school.  I do not have answers to all the questions I encountered, but I am going to share some interesting questions that have made me think recently. 

Storage

Several of the talks touched on just the scope of having too much data.  Even with terabyte drives being cheap and readily available, the storage problems are still the same as when I was in school: data can be generated far faster than one can analyze it and will exceed local storage if one is not careful.  Even a petabyte can fill quickly with multiple users who are each generating terabytes. 

In fact, the volume problem can be worse than it used to be because of the size of individual files, the number of files, or both.  Ending up with several thousand files that are each tens, if not hundreds of gigabytes, of data is becoming a common problem for people who do the kinds of simulations that I do.  Worse is that those files are probably generated on one supercomputing cluster, but need to be analyzed or stored somewhere else because that’s how things were done years ago.  The system admins become downright rude about people filling up scratch space, but, in terms of researcher time management, leaving generated files on the scratch drives for a few weeks is often the best solution.  Dropbox, cloud storage, and Google documents are great for people who only need to share a couple of reports or presentations, but it’s not at all useful for the researchers who are routinely dealing with thousands of files that are each huge.

The good folks in the Chicagoland area (Argonne, University of Chicago) have made Globus.  Globus is a way to efficiently transfer files for supercomputer and similar cluster users that takes into account how much babysitting the big files need to transfer.  I have not used it, but I will try it the next time I am in the situation of having my data being generated offsite and need to transfer to storage.

Interaction

Several of the talks at the summer school described the difficulty of interacting with huge amounts of data.  Automated searches are nice, but how are things indexed in the first place?  What about data sets that consist of widely varying things that are hard to describe?  For example, many databases are great with formatted records, but what about data that is generated and is unformatted, free-form, or wildly different formats (not just Excel, plain text, and Kaleidagraph, but movies, pictures, maps, pointers to books or physical objects)?  Limiting oneself to things that are easily described and indexed using traditional means missing out on perhaps as much as 70-85% of the data currently in electronic form. 

I was particularly taken by one speaker who mentioned that many of the easily dealt with data are misleading.  For example, a numerical poll may tell you much less about what people are thinking than the free-form responses.  However, tabulating a million numerical poll results is computationally easy and dealing with a thousand free-form written responses is not.  Searching something like PubMed by keyword is easy, if you know the right keywords, but is extremely difficult if the words you want were not what someone else considered important enough to index, especially if the records are not searchable text.  The novel Mr. Penumbra's 24-Hour Bookstore has a very nice example of the protagonist searching a database for the record of a physical object that must be in storage somewhere and coming up empty because he can't figure out the proper keywords.  I will not share the spoiler on how the protagonist solved the problem by hacking the carbon interface instead of the silicon interface, but it is very clever and a good example of how computers are not as smart as people are even now.

I still have more questions than answers, because that is the state of the field, but I look forward to trying some things for myself and reading/talking with people who are pushing hard to find solutions for their mixed huge data set problems.

Analysis

People who have data usually want to perform analysis to turn data into information.  This is also far trickier than one might expect if one is doing anything other than simply scaling up an analysis to a bigger data set of coherently formatted records.  Even simply scaling up an analysis can become tricky when machine memory will be exceeded or a program has a hard limit on the number of entries.  For example, I have overwhelmed Excel by having too many rows of data to read in and I am told limits exist for many other programs.  Once people start talking about databases with millions of entries and tens of thousands of possible keys for every entry, the unwieldiness is evident.  That is indeed big data without good solutions and we haven't even gotten to what do you do when a data set is not in a formatted way that can be automated and you aren't quite sure how to use it anyway.

 In the summer school, we were introduced to some functions in R that might help people plot and learn something.  My favorite is get_map, which allows you to use the data from Google maps (a coherent description with examples is here).  Another interesting tool was the Google motion chart, which I suspect is how Michael Marder makes his interesting plots on what matters for science education.

 More flexible tools in R were in the package ggplot2 (a nice overview of the functionality is here).  If you want slick looking, complex plots, then this is for you.  I am continuing to experiment with facet_grid because I often have multiple small plots to make to look at slices through my data by extra variables.  Having a program that will do it for me saves me a lot of time.

Summary

Big data is the latest buzzword that affects some of us (you do know about the requirement for a data management plan for your next NSF proposal, don't you?), but it is real problem for people doing cutting-edge science.  If you find yourself overwhelmed with your data, then I highly recommend that you attend a workshop or seminar series to share ideas, even with those who are far outside your field.  It's time well spent.