Machine learning: match the validation unit to the prediction task
A validation score answers a question about what was withheld and what must be predicted. This route keeps the observation unit, training rule and deployment target visible before discussing model complexity. A perfect score can be correct for repeated observations of known devices and irrelevant to predicting a new device.
Course topic: Machine Learning and Artificial Intelligence
Study route and exercise: University Reading List editorial project. How we curate · Sources checked:
On this page
Scope and responsibility
For learners with basic statistics, regression, Python reading and distance calculations. ISLP offers the accessible applied spine; Murphy needs stronger linear algebra, probability and calculus; AIMA broadens the question to agents and decision problems. Every device, feature and label below is synthetic, with no population-performance or causal claim.
Choose your starting point
I want free applied statistical learning
Use the Python ISL volume for concepts paired with labs. Begin with prediction error and resampling before comparing increasingly flexible methods.
I need a probabilistic foundation
Use Murphy after the mathematical prerequisites. Ask what joint distribution and decision rule a model assumes, not merely which package implements it.
I need the wider AI map
Use AIMA to distinguish learning a predictor from search, planning and agent decisions. Select learning chapters for the worksheet rather than treating every AI topic as supervised classification.
Compare the three books
An Introduction to Statistical Learning: with Applications in Python
Gareth James; Daniela Witten; Trevor Hastie; Robert Tibshirani; Jonathan Taylor · 2023
Free applied spine
- Choose it when
- You can read basic Python and statistics and want conceptual explanations paired with executable labs.
- Use another route when
- You need every derivation or a first Python course before beginning the statistical questions.
- Selected reading
- 2023 Python volume: Statistical Learning, Classification and Resampling Methods, then selected model-selection discussions.
- Limits and trade-offs
- It is distinct from the 2013/2021 R volumes. Authors provide a free PDF; the publisher’s print/e-book formats and notebook dependencies remain separate objects.
Try before choosing
Write two legitimate validation questions for the same repeated-device table and choose a split unit for each.
Check your work: Withholding a row is not the same as withholding the device that generated it.
Probabilistic Machine Learning: An Introduction
Kevin P. Murphy · 2022
Free mathematical companion
- Choose it when
- You want to connect prediction to probability, optimization and explicit decision criteria.
- Use another route when
- Linear algebra, multivariable calculus and conditional probability are still unfamiliar.
- Selected reading
- 2022 introductory volume: mathematical background, supervised learning and model assessment by title, before specialized transfer or deep-learning topics.
- Limits and trade-offs
- The selected hardcover ISBN 9780262046824 identifies the 2022 book; the author’s free draft was updated in 2025 under noncommercial/no-derivatives terms. It is not the Advanced Topics sequel.
Try before choosing
State whether a future query is conditional on a known device or drawn from a new device, then name what independence your evaluation needs.
Check your work: Identical feature rows from one group do not supply independent evidence about an unseen group.
Artificial Intelligence: A Modern Approach
Stuart Russell; Peter Norvig · Edition: 4 · 2020
Broad AI reference
- Choose it when
- You need to situate prediction within agents, search, reasoning and action.
- Use another route when
- You want a short statistical-learning lab sequence without selecting chapters.
- Selected reading
- Fourth US edition: Intelligent Agents and Learning from Examples, then probabilistic/deep/reinforcement learning as relevant.
- Limits and trade-offs
- The US and Global contents differ. Public exercises and code complement the commercial book; they do not make the complete book or all instructor solutions free.
Try before choosing
Describe an agent that uses the classifier, distinguishing its percept, predicted label, action and evaluation criterion.
Check your work: A classification accuracy is not automatically an agent’s utility or a safety/decision guarantee.
Check your preparation
What is fitted?
The worksheet fixes 1-nearest-neighbor and its tie rule before evaluation; there is no hyperparameter selection or preprocessing to tune on withheld rows.
What is the target unit?
It may be another reading of an already represented device or a device absent from training. The correct withholding unit depends on that target.
What does a score establish?
An exact result on the supplied finite packet. Estimating future-population performance requires a justified sampling/deployment relation and uncertainty analysis beyond the table.
A six-step study plan
Steps completed: 0 / 6
These are your own study marks, not an assessment. They are not sent to this site.
-
Define the target
Selected reading: ISLP statistical-learning introduction.
What to do: Write what a future observation represents and what information training already contains about it.
Check your work: Make known-device and new-device targets separate sentences.
-
Specify the rule
Selected reading: Classification reading.
What to do: State feature, distance,1-neighbor vote and deterministic tie handling.
Check your work: Device ID is used to form folds, not as an extra prediction feature.
-
Audit row withholding
Selected reading: Resampling discussion and the original table.
What to do: List the training neighbor used for each withheld row.
Check your work: Record whether a same-device twin remains in training.
-
Audit group withholding
Selected reading: The same model-assessment reading.
What to do: Remove both rows of a device and calculate its prediction.
Check your work: Keep the algorithm fixed so the comparison isolates the withholding unit.
-
Change deployment assumptions
Selected reading: The solved follow-up.
What to do: Prove a nearest-neighbor margin for bounded new readings of known devices.
Check your work: A conditional robustness proof does not rescue an estimate for new devices.
-
Report without leaking a claim
Selected reading: Murphy model assessment or AIMA evaluation discussion.
What to do: Report split unit, target, predictions and exact finite scores together.
Check your work: Avoid selecting the best-looking score after seeing answers or promoting it to an unsupported generalization guarantee.
Worked exercise
Original repeated-device table: two different validation questions
Four synthetic devices A, B, C, D have feature values 0,10,20,30 and stable labels 0,1,0,1. Each appears in two identical rows. Use 1-nearest-neighbor on the scalar feature, absolute distance, and choose the lower feature value on a tie. Compare leaving out one row at a time with leaving out both rows of a device.
Assumptions
The eight rows are complete and exact. Device ID defines grouping but is not a prediction feature. Train only on the retained rows, with the same fixed rule in every fold. Scores are counts of correct withheld-row predictions, not estimates with supplied sampling guarantees.
- Row-wise withholding leaves the identical same-device twin at distance 0. Every withheld row receives its correct label, giving 8/8=100%.
- Withhold device A: its nearest retained feature 10 has label 1, wrong for A 0. Withhold D: nearest 20 has label 0, wrong for D 1.
- Withhold B: features 0 and 20 tie, both label 0, so its prediction is wrong. Withhold C:10 and 30 tie, both label 1, also wrong. Both rows of every held-out device fail:0/8=0%.
- Neither arithmetic score is broken. They withhold different information. The row score measures this packet’s already-represented-device task, while the group score removes device-specific representation. Neither alone proves a population accuracy.
Answer and interpretation
Row withholding 100%; whole-device withholding 0% on these eight invented rows. A split unit must match the intended prediction question.
Test the result
List all eight predictions under both schemes and the neighbor used. Do not infer that grouped validation always produces zero or that row validation is always inappropriate.
Change an assumption
Changed target: bounded readings of known devices
Now deploy only on the same four known devices, retain all eight training rows, and stipulate each query’s label stays its device label. Its feature differs from that device’s original value by at most ε. First set ε=1, then find a sufficient uniform radius and test ε=5.
- At ε=1 the same-device training feature is at distance at most 1. Any other device’s feature is at least 10−1=9 away, so every allowed query is correctly classified.
- For general 0≤ε<5, the own-device distance is at most ε while every other-device distance is at least 10−ε>ε. This proves a uniform guarantee under the stated target and stable-label assumptions.
- At ε=5 the guarantee fails: device B can produce query 5 with true label 1. Features 0 and 10 tie; the lower-feature tie rule selects 0, predicting label 0. One valid counterexample is enough to refute a guarantee for all devices.
Answer and interpretation
All allowed known-device queries are correct at ε=1, and uniformly for ε<5. Equality 5 does not give the same all-device guarantee. This is a conditional distance proof, not a claim about unknown devices.
Test the result
Check the strict inequality and the B-at 5 counterexample. If labels drift, devices are new, or the distance rule changes, the supplied proof no longer establishes the result.
Common mistakes
Row independence assumed
Repeated rows can retain exactly the device-specific information the deployment task would withhold.
Prediction confused with action quality
A score for labels does not specify losses, decisions or outcomes for an agent.
Finite packet promoted to a guarantee
Exact calculations establish these rows or the explicit distance assumptions, not an unspecified future distribution.
Before moving on
- I can define the target unit and construct the matching folds.
- I can reproduce both prediction ledgers and the strict distance bound.
- I can distinguish a finite score, a conditional robustness proof and a population claim.
Sources and edition checks
Sources checked:
- Authors: free ISL Python volume and edition distinctions
The authors distinguish Python 2023 from the R editions and provide a free applied reading route.
- Springer:2023 Python book and five authors
The publisher identifies the five-author Python volume and its prediction/resampling scope.
- Author: introductory volume and updated free draft
The free updated introductory draft and its permissions are distinct from the printed publication date.
- MIT Press:2022 introductory hardcover
The 2022 hardcover and Advanced Topics sequel are different works.
- Authors: fourth US edition, agents and learning contents
The fourth US edition supplies a broad agent-and-learning comparison, not a substitute for the same statistical-lab sequence.
These are proposed study activities, not official course assignments or a claim that every book has been read cover to cover.