Big data refers to datasets so large and complex that traditional database tools cannot process them effectively. Three characteristics define big data: volume (the sheer amount), velocity (the speed at which it arrives), and variety (the different formats it comes in). Understanding big data is part of the GCSE Computer Science topic on the impact of technology.
What are the three Vs of big data?
The three Vs framework is the standard way to characterise what makes a dataset "big data" rather than simply a large database:
Volume
The sheer quantity of data is enormous — often measured in petabytes (1 petabyte = 1,000 terabytes) or exabytes. Example: Facebook processes more than 500 terabytes of new data every day from likes, posts, photos, and videos. A traditional relational database on a single server cannot store or query this quantity efficiently.
Velocity
The speed at which data arrives and must be processed. Some big data must be processed in real time or near-real time:
- Stock market trades: millions per second, each requiring instant analysis
- Traffic sensors: updating navigation apps with road conditions every few seconds
- Social media: thousands of posts per second during a major event
Traditional batch processing — collecting data and analysing it later — is too slow for these applications.
Variety
Big data comes in many formats, not just neat rows and columns. It may be:
- Structured: traditional database tables (numbers, dates, fixed fields)
- Semi-structured: JSON or XML files, emails with metadata
- Unstructured: images, videos, audio, natural language text, social media posts
Traditional databases handle structured data well but struggle with unstructured formats.
Some definitions add a fourth V: Veracity — the accuracy and trustworthiness of the data, since big datasets often contain errors, missing values, and inconsistencies.
Where does big data come from?
| Source | Examples |
|---|---|
| Social media | Posts, likes, shares, hashtags, images — billions of interactions daily |
| Internet of Things (IoT) | Smart meters, fitness trackers, traffic sensors, weather stations |
| Financial transactions | Every card payment, stock trade, and bank transfer |
| Healthcare | Medical scans, electronic health records, genomic data |
| Search engines | Every search query, click, and scroll |
| Retail | Every purchase, return, and browsing session on e-commerce sites |
| Scientific research | Telescope data, climate models, particle accelerator readings |
How is big data processed?
Traditional databases run on a single server, which is not powerful enough for big data. Instead, big data systems use distributed computing: the data is split across many computers (a cluster), and each computer processes its share in parallel. The results are then combined.
The most influential early system was Hadoop (open source), which uses an approach called MapReduce:
- Map phase: each computer in the cluster processes its portion of the data independently, outputting intermediate key-value pairs.
- Reduce phase: the intermediate results from all computers are combined to produce the final answer.
For example, to count how many times each word appears in billions of web pages: the Map phase counts words in each chunk; the Reduce phase sums the counts for each word across all chunks.
Modern systems such as Apache Spark process data faster by keeping more in RAM rather than writing to disc between steps.
What are the benefits of big data?
| Domain | Benefit | Example |
|---|---|---|
| Healthcare | Identifying disease patterns across millions of patients | NHS using patient data to predict hospital admissions |
| Transport | Optimising traffic flow in real time | Google Maps routing millions of users to avoid jams |
| Retail | Personalising recommendations | Amazon suggesting products based on purchase history |
| Science | Accelerating research | Analysing genomic data to identify disease-linked genes |
| Crime prevention | Predicting high-risk areas for policing resources | Analysis of crime pattern data |
| Energy | Reducing waste | Smart grids balancing electricity supply with demand |
What are the risks and ethical concerns?
| Risk | Description |
|---|---|
| Privacy invasion | Vast personal data collection without meaningful consent |
| Data breaches | Large, centralised datasets are high-value targets for hackers |
| Algorithmic bias | Biased training data produces biased predictions (e.g. in hiring or loan decisions) |
| Surveillance | Governments or corporations tracking individual behaviour at scale |
| Data ownership | Unclear who owns data about an individual — the person, the company, or the government |
| Environmental cost | Storing and processing petabytes of data consumes enormous energy |
The General Data Protection Regulation (GDPR) requires organisations to have a lawful basis for processing personal data, to store only what is necessary, and to protect it appropriately. Big data's appetite for data sits in tension with GDPR's principle of data minimisation.
Frequently asked questions
Is big data the same as machine learning?
No, but they are closely connected. Big data refers to the large, fast, varied datasets. Machine learning is a technique for finding patterns in data automatically. Machine learning systems are more powerful when trained on larger datasets — big data provides the raw material. For example, a speech recognition system trained on millions of hours of audio (big data) is far more accurate than one trained on thousands of hours.
Can schools use student data as big data?
Potentially, at a large scale. Aggregated data from thousands of students can reveal patterns in learning — which explanations work best, which question types predict exam performance — informing pedagogy. This is the basis of educational data mining and learning analytics. The ethical requirements are significant: student data is sensitive, and schools must comply with GDPR, be transparent about data use, and ensure data is not used in ways that could harm students.
What is data mining?
Data mining is the process of discovering useful patterns, correlations, and insights in large datasets using statistical and computational techniques. Examples include: finding which products are frequently bought together (used by supermarkets to arrange shelves), identifying credit card fraud patterns (unusual purchase behaviour), and segmenting customers into groups with similar buying habits (used in marketing). Data mining is how organisations turn big data into actionable intelligence.
How does big data differ from a regular database?
A regular relational database (such as a school's student records system) stores structured data in tables, can be queried with SQL, and runs on a single server or a small cluster. It is designed for consistency and accuracy. Big data systems sacrifice some consistency guarantees in exchange for the ability to handle vastly larger volumes, higher ingestion rates, and diverse data formats across clusters of hundreds or thousands of servers. The tools, architectures, and engineering challenges are fundamentally different.
Explore the social and ethical dimensions of big data with Professor Turing's discussion-based approach at aitutors.me.