One of the interesting things I get to work on at my job is a technology called Hadoop. Hadoop is a set of open source software that includes (among many other things) a distributed file system, a software framework written in Java for distributed processing, and a high-level data flow language called Pig that allows for simple queries of large datasets.In layman's terms, Hadoop allows for the storage, data processing, and data mining of ginormous sets of data (on the order of Petabytes*). I'm part of a team that helps write jobs that process advertising data on Hadoop to allow customers to run useful analytics and research experiments. The clusters that our jobs run on are made up of over 3000 different servers. A single job can run on almost all of them! Yeah, I'm nerdy.
If any of you are ambitious and want to try out Hadoop for yourself, you can install and run it in local mode (on your own home computer) by following these instructions.
* A Petabye is equivalent to 1024 Terabytes. A Terabyte is equivalent to 1024 Gigabytes.
No comments:
Post a Comment