Mobile devices, remote sensing technologies, software logs, cameras, microphones, radio-frequency identification, wireless sensors, weather satellites and sensors, scientific experiments , social networks, internet text and documents, Internet search indexing, call detail records, astronomy, atmospheric science, genomics, biogeochemical, biological, and other complex and often interdisciplinary scientific research, military surveillance, medical records, photography archives, video archives, and large-scale e-commerce, all contribute. As more and more sensors, mobile devices, cameras etc are added into the network/system as more number of people share photos, music etc, as more number of netizens join social networks the size increases. A few examples of some systems and the amount of data they generate:
CERN:-The Large Hadron Collider project in CERN produced 22PB of data this year after accepting only 1% of the data produced which is about 100MB per sec.
FLICKR:-More than 4 billion queries per day, ~35M photos in squid cache (total), ~2M photos in squid’s RAM, ~470M photos, 4 or 5 sizes of each, 2 PB raw storage
FACEBOOK: - As of July 2011 750 million worldwide users uploaded approximately 100 terabytes of data every day to the social media platform. Extrapolated against a full year, that’s enough data to manage the U.S. Library of Congress’ entire print collection—3,600 times over.
Not only is this but the per-capita capacity to store information also responsible for this huge data explosion. Data storage was very expensive about quarter of a century back, as prices of storage came down more and more data got stored and currently as per estimates per-capita capacity to store information has roughly doubled every 40 months since the 1980
Is that the end of the story as far as the source and rational behind the growth of data? No, please include enterprise ‘structured’ data to the list too, which can provide wonderful insights. Metadata, data about data, which is increasing twice as fast as the digital data growth, also adds to the list.
Asked In: Many Interviews |