Using a Large Dataset to Develop and Test a Question

A large dataset is a big table of records that someone has already collected, often thousands or millions of rows. On its own it answers nothing, it is just data waiting for a question. The skill in this lesson is the secondary-data workflow: a clear sequence that takes you from a wall of numbers to a tested answer. The steps are explore the dataset, notice a pattern or gap, form an investigable question, identify the variables, clean the data, analyse it, then answer with evidence.

This connects two earlier skills. In Lesson 1 you learned what makes a question investigable: a variable you can measure, a variable that changes, and feasibility. In Lesson 10 you met large datasets and where they come from. Here you put them together: the dataset suggests the question, and the columns you have decide whether that question is actually investigable.

Primary data is data you collect yourself, by measuring or counting things in your own investigation. Secondary data is data that someone else has already collected, which you then reuse. When you work with a large dataset from the Bureau of Meteorology or the Australian Bureau of Statistics, you are using secondary data. The big advantage is speed and scale: you instantly have thousands of records that would take years to collect yourself.

The trade-off is that you did not control how it was collected, so you must check it carefully. Who collected it, and are they reliable? When and where was it collected? What does each column actually measure, and in what units? This information is called metadata, and reading it is part of responsible data science. Never assume, always check the source before you trust the numbers.

Every dataset is organised the same way. Each row is one record, also called an observation, such as the weather on a single day. Each column is one variable, such as date, maximum temperature, or rainfall. Where a row and a column meet is one value, like 31.4 degrees. The header at the top names each column and usually states its units, for example degrees Celsius or millimetres.

Reading a dataset means understanding this structure before you do anything else. How many rows are there (how much data)? What does each column measure, and in what units? Are there any columns you do not understand? You answer these from the header row and the metadata. Only once you know what variables you have can you ask a question the dataset is actually able to answer.

From Lesson 1 you know a question must be investigable: it needs a variable that changes and a variable you can measure. With secondary data there is an extra rule, the question must match the columns you actually have. You cannot ask "Does humidity affect ice-cream sales?" if there is no humidity column. The dataset sets the boundary of what you can ask.

So the move is: scan the columns, pick two that might be related, and write a question linking them. With our table you could ask "Is there a relationship between daily maximum temperature and the number of ice-creams sold?" Both variables are columns you have, both are measurable, and the data already exists, so it is investigable and feasible. A good dataset question names the two columns and asks how one relates to the other.

Once your question links two columns, decide which column is the independent variable (the one you think causes or drives the change) and which is the dependent variable (the one you expect to respond). In a dataset you do not physically change anything, but you still sort the columns this way so your analysis and graph make sense. By habit, the independent variable goes on the horizontal axis and the dependent on the vertical axis.

For "Is there a relationship between maximum temperature and ice-creams sold?", temperature is the independent variable (we think temperature drives sales) and ice-creams sold is the dependent variable (sales respond to temperature). The date column is neither, it just labels each record. Sorting columns into independent, dependent and label is how you get a dataset ready to analyse.

Real datasets are messy. Before analysing, you clean the data, which means finding and dealing with problems so the numbers can be trusted. The main things to look for are missing values (blank cells where a reading was not recorded), outliers or errors (impossible values like a temperature of 200 degrees, often a typing mistake), wrong units (one row recorded in a different unit), and duplicates (the same record entered twice).

Cleaning matters because a single bad value can ruin a result. If one day's temperature is wrongly typed as 310 instead of 31, the average jumps and your conclusion is wrong. You do not just delete numbers you dislike, you correct clear errors, remove duplicates, and decide carefully what to do with missing values. Cleaning is honest housekeeping that protects the answer, not a way to force the result you wanted.

Let us run the whole workflow on the small extract from Card 4. Explore: the columns are date, max temperature (degrees Celsius), rainfall (mm) and ice-creams sold. Notice: the hottest days (35.1 and 31.4 degrees) have the highest sales (210 and 182), while the coolest day (22.6) has the lowest (64). That looks like a pattern worth testing.

Question: "Is there a relationship between daily maximum temperature and the number of ice-creams sold?" Variables: independent is temperature, dependent is ice-creams sold, and date is just a label. Clean: check for blanks, impossible values and duplicates, and fix any found. Analyse: plot temperature on the horizontal axis and sales on the vertical axis, then look at whether higher temperature lines up with higher sales. Answer: in this extract, as maximum temperature rises, ice-cream sales also tend to rise, so the evidence supports a positive relationship. You have turned a table of numbers into a clear, evidence-based answer.