Large Datasets: Features, Collection and Uses

A dataset is simply an organised collection of related data. A dataset becomes a large dataset when it is too big to make sense of by hand. Scientists describe largeness using three features that all start with V. The first is volume, the sheer number of records: not 28 student heights but billions of weather readings. The second is variety, the number of different things measured: a weather dataset records temperature, rainfall, wind speed, wind direction, humidity and pressure, all at once. The third is velocity, the speed at which new data arrives: sensors can stream fresh readings every second, so the dataset never stops growing.

Large datasets also come in two forms. Structured data fits neatly into a table of rows and columns, like a spreadsheet of dates and temperatures. Unstructured data does not, it includes things like photos, satellite images, written text and audio. Modern science deals with both, but structured data is the easiest to count, sort and graph, which is why so much of data science starts there.

Nobody types billions of numbers by hand. Large datasets are gathered automatically and continuously. Automated sensors and instruments measure things like temperature, light or movement and log them without a person watching. Satellites photograph the whole planet, building enormous image datasets of land, ocean and atmosphere. Big surveys and the census collect answers from millions of people at once. Transaction and log records are created automatically every time you tap a card, ride a train or load a web page.

People help too. Citizen science projects invite thousands of volunteers to record bird sightings, classify galaxies or photograph plants, adding to datasets no single team could build alone. And our everyday online activity, searches, clicks and posts, produces a constant stream of data. Each method has trade-offs: sensors are reliable but only measure what they are built for, while surveys capture rich human detail but cost time and money.

Large datasets matter because of what they let us do. In weather forecasting, the Bureau of Meteorology feeds billions of readings into models that predict rain and heatwaves days ahead. In public health and medicine, datasets of millions of patient records help spot which treatments work and how diseases spread. In astronomy, telescope surveys photograph billions of stars and galaxies, far too many to study one by one. In ecology, the Atlas of Living Australia tracks where species live and how their ranges shift over time.

Government planning leans on the Australian Bureau of Statistics census to decide where to build schools and hospitals. Even sport now runs on data: clubs record every pass, sprint and shot to find patterns players cannot see unaided. In every case, the dataset is too large for a person to read, so we use computers and statistics to turn it into something useful.

Large datasets are powerful because they reveal patterns invisible in small samples, give reliable averages that are not thrown off by one odd result, and let us compare across groups and over time. But they have real limits. They need huge storage and processing power. Their value depends on data quality, since errors copied billions of times are still errors. They raise privacy and ethics questions, because data about people must be protected. And bias in collection can quietly distort the whole picture, as the city-only bird survey showed.

To use a dataset you must read its structure. Each row is a record, one observation such as one day at one weather station. Each column is a variable, one measured thing such as temperature. Numbers come with units (degrees Celsius, millimetres) and the whole table is described by metadata, which tells you when, where and how the data was gathered. Knowing this anatomy is the key to everything you will do with data in the rest of this unit.