The Wall Fell. The Data Flood Began.
In the early 1990s, the end of the Cold War created an existential crisis for the Intelligence Community, compounded by a new technological threat: the "worldwide information explosion."
Cost of 1TB of Storage
$2,000,000
1993
~$20
2025
The MDDS Vision
20 Petabytes
The goal was to manage 200MB of data on "every man, woman, and child on earth"—a breathtakingly ambitious goal in 1993.
The Email That Sparked Big Data
From: Tatu Ylonen
Date: Fri, 14 Jul 1995 12:34:56 +0300
Subject: CFP: Massive Digital Data Systems
The Intelligence Community is sponsoring an unclassified workshop on Massive Digital Data Systems (MDDS)...
The IC is being challenged to provide timely intelligence... from massive volumes of data. Issues include: scalability to petabytes, federated architectures, data fusion, query optimization, and information retrieval.
...
Crypto relevance?
What was CMS?
The Community Management Staff was the IC's "corporate headquarters," coordinating policy and budget across agencies like the CIA and NSA.
What was MITRE?
A non-profit Federally Funded R&D Center (FFRDC) that served as the technical brain trust for the government and the IC.
Match the IC's 1993 Wish List to Modern Tech
A Tale of Two Grants
Public Funding Stream
The Stanford Digital Library Project, funded by NSF/DARPA/NASA, provided the foundational infrastructure and institutional legitimacy.
Intelligence-Linked Stream
The MDDS program, managed for the CIA/NSA, provided targeted "seed-funding" for Brin's research on data mining and query optimization.
Brin’s 1998 paper, "What can you do with a Web in your Pocket?", contains a critical footnote...
PageRank: More Than a Search Tool
They Came to Watch Google Grow
IC Visits Stanford
Program manager Bhavani Thuraisingham confirms they would visit Stanford "every few months to see Brin and monitor his progress."
Brin Demos "Google"
In these briefings, "Brin did present to us on the query flocks research, and also demonstrated to us versions of the Google search engine."
September 1998
The final demonstration for the IC occurred in the very same month that Sergey Brin and Larry Page officially incorporated Google, Inc.
What would you have thought in 1998?
From MDDS to the Utah Data Center
1993: Petabyte Projections
The MDDS program sets a goal to manage data on a scale previously unheard of.
MDDS Goal: 20 Petabytes
2004: Google Expands
Just a decade later, Google's index surpasses the IC's initial dream by orders of magnitude.
Google Index: ~100 Petabytes
2013: PRISM Leaks
Revelations show the deep, ongoing relationship between the IC and the tech world.
Direct Access to Tech Giants
2014: Utah Data Center
The physical manifestation of the MDDS vision, built to store global data.
Capacity: 3-12 Exabytes
The Most Important Question You Never Noticed
Crypto relevance?
Tatu Ylonen's question in 1995 was prescient. The tools built for knowledge are indistinguishable from tools for control. MDDS wasn’t evil—it was a strategic necessity from the IC's perspective. But now, we all live inside the machine it helped design.
Understanding this history is essential to navigating the complex ethical and political challenges of the age of Big Data.