Significance of data cleaning
Describe the significance of data cleaning
In this lesson
- Describing how the quality of training data affects a model's performance
- Describing techniques for outliers, duplicate, incorrect, irrelevant, improperly formatted and missing data
- Describing normalisation and standardisation, and when each is preferred
LearnWhy clean data matters
LearnProblems and how to treat them
LabClean the data, then test the model
A model predicts house prices from floor area, distance to the centre and age. Treat each problem and watch its error on new sales.
| ID | Area (m²) | Distance (km) | Age | Price | Seller phone |
|---|---|---|---|---|---|
| 1 | 127 | 11.2 | 6 | 233 | 07334419231 |
| 2 | 118 | 13.9 | 13 | 189 | 07824340488 |
| 3 | 109 | 12 | 33 | 176 | 07586784381 |
| 4 | 165 | 15.1 | 78 | €263k | 07721219083 |
| 5 | 162 | 3.7 | 68 | 313 | 07101024837 |
| 6 | 54 | 6.1 | 40 | 114000 | 07835788778 |
| 7 | blank | 9.1 | 37 | 306 | 07110215761 |
| 8 | 160 | 7.6 | 67 | 300 | 07194892853 |
| 9 | 145 | 18.8 | 68 | 172,000 | 07456920652 |
| 10 | 188 | 1.2 | 70 | 346 | 07703847391 |
| 11 | 69 | 3 | 45 | 156 | 07429572755 |
| 12 | 194 | 2.8 | -57 | 395 | 07157949000 |
Rows used for training: 48 of 55 · Columns: 5 · 3 prices stored as text could not be used.
Typical prediction error on 200 new, clean sales: €2,733k — the model is useless.
A price typed in euros (not thousands) and an area with an extra zero are still in the data, and they dominate the model.
The 15 extra copies give three unusual sales six times their proper weight, pulling the model towards them.
Treat every problem to complete this activity. For missing data, compare the options: deletion, imputation and prediction all give similar results here. (Predicting a missing area from the price works only for training data: a new house’s price is what the model must predict.)
LearnNormalisation and standardisation
Features measured in different units have very different ranges. Many algorithms (k-NN, clustering, gradient descent) are affected by those ranges, and the result would even change if a feature were recorded in different units (metres instead of kilometres). So features are rescaled first.
LabDoes scaling matter?
A k-nearest-neighbours model sorts houses into above or below the median price, using floor area (m²) and distance to the centre (m).
| Feature | min | max | mean | standard deviation |
|---|---|---|---|---|
| Floor area | 50.00 | 197 | 123 | 39.65 |
| Distance | 1,200 | 19,900 | 10,566 | 5,377 |
k-NN accuracy (each house classified from its 3 most similar houses): 59%
Distances in metres run into the thousands while floor areas are in the hundreds, so “similar” houses are chosen almost only by distance; floor area, the most important feature, is nearly ignored.
Try all three options, and both scaling methods with the extreme value, to complete this activity.
CheckQuick check
A feature has the values 20, 50 and 80. What is 50 after min-max normalisation to the range 0–1?
ExamWorked exam answer
A gym's attendance data contains each visit twice for members who used both the front and side doors. Describe how these duplicate records could affect a model that predicts busy times, and how they should be treated.
[2]ReviewMistakes examiners see
Removing every outlier automatically. A rare but real value (a very expensive house) is information; only errors should be removed.
Mixing up the two scalings: normalisation sets a fixed range (0 to 1); standardisation sets mean 0 and standard deviation 1.
For “Describe” questions, name the technique, say what it does to the data, and give its effect on the model in the scenario.
Exam practiceExam-style questions
4 questions · 12 marks. Answer each one first, then mark it yourself against the markscheme. Your answers are saved in this browser only.
A cycling app predicts how long a ride will take from its distance (km), total climb (m) and the weather. It trains on rides recorded by thousands of users' phones. Some rides have no climb recorded, and a few phones recorded impossible speeds.
A few rides show average speeds of over 200 km/h because of GPS errors. Describe how these outliers could be identified and treated.
Describe two ways of dealing with the rides that have no climb recorded.
The model uses k-nearest neighbours. Distances range from 1 to 150 km and climbs from 0 to 3000 m, and a few rides cross high mountain passes. Explain why the features should be scaled, and why standardisation may be preferred to normalisation here.
Almost all of the app's users are experienced cyclists. Describe how this could affect the model's predictions.
0 of 4 questions marked. Mark every question to see your total.
Exam-style questions written for Praxis IB in the format of the 2027 specimen papers. They are not IB past-paper questions.