Simple tips to deal with Big Data In Bioinformatics
Here is something I came across recently, a relatively old article which I found on Bioinformatics Zen by Michale Barton. Simple, practical and worldly wise. Bioinformatics usually involves shuffling data into the right format for plotting or statistical tests. I prefer to use a database to store and format data as I think this make projects easier to maintain compared with using just scripts. I find a dynamic language like Ruby and libraries for database manipulation like ActiveRecord makes using a database relatively simple. Using a database however stops being simple when you have to deal with very large amounts of data. Here I’m outlining my experience of analysing gigabytes of data with millions of data points and how tried to improve my software’s performance when manipulating this data. I’ve ordered each approaches with I think is the most pragmatic first. The simple things Obvious but sometimes overlooked. 1. Use a bigger computer Using a faster computer might seem like a la...