Multiple Imputation and its Application
, by Carpenter, James R.; Bartlett, Jonathan W.; Morris, Tim P.; Wood, Angela M.; Quartagno, Matteo; Kenward, Michael G.
- ISBN: 9781119756088 | 1119756081
- Cover: Hardcover
- Copyright: 7/24/2023
Multiple Imputation and its Application
The most up-to-date edition of a bestselling guide to analyzing partially observed data
In this comprehensively revised Second Edition of Multiple Imputation and its Application, a team of distinguished statisticians delivers an overview of the issues raised by missing data, the rationale for multiple imputation as a solution, and the practicalities of applying it in a multitude of settings.
With an accessible and carefully structured presentation aimed at quantitative researchers, Multiple Imputation and its Application is illustrated with a range of examples and offers key mathematical details. The book includes a wide range of theoretical and computer-based exercises, tested in the classroom, which are especially useful for users of R or Stata. Readers will find:
- A comprehensive overview of one of the most effective and popular methodologies for dealing with incomplete data sets
- Careful discussion of key concepts
- A range of examples illustrating the key ideas
- Practical advice on using multiple imputation
- Exercises and examples designed for use in the classroom and/or private study
Written for applied researchers looking to use multiple imputation with confidence, and for methods researchers seeking an accessible overview of the topic, Multiple Imputation and its Application will also earn a place in the libraries of graduate students undertaking quantitative analyses.
James R. Carpenter is Professor of Medical Statistics at the London School of Hygiene & Tropical Medicine and Programme Leader in Methodology at the MRC Clinical Trials Unit at UCL, UK.
Jonathan W. Bartlett is a Professor of Medical Statistics at the London School of Hygiene & Tropical Medicine, UK.
Tim P. Morris is Principal Research Fellow in Medical Statistics at the MRC Clinical Trials Unit at UCL, UK.
Angela M. Wood is Professor of Health Data Science in the Department of Public Health and Primary Care, University of Cambridge, UK.
Matteo Quartagno is Senior Research Fellow in Medical Statistics at the MRC Clinical Trials Unit at UCL, UK.
Michael G. Kenward retired in 2016 after sixteen years as GlaxoSmithKline Professor of Biostatistics at the London School of Hygiene & Tropical Medicine, UK.
Preface
Data acknowledgments
Glossary
I Foundations 1
1 Introduction 2
1.1 Reasons for missing data . . . . . . . . . . . . . . . . . . . . . 5
1.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Patterns of missing data . . . . . . . . . . . . . . . . . . . . . 8
1.3.1 Consequences of missing data . . . . . . . . . . . . . . . 10
1.4 Inferential framework and notation . . . . . . . . . . . . . . . . 13
1.4.1 Missing Completely At Random (MCAR) . . . . . . . . 15
1.4.2 Missing At Random (MAR) . . . . . . . . . . . . . . . . 16
1.4.3 Missing Not At Random (MNAR) . . . . . . . . . . . . 22
1.4.4 Ignorability . . . . . . . . . . . . . . . . . . . . . . . . . 27
1.5 Using observed data to inform assumptions about the missingness mechanism . .. . . . . . . 28
1.6 Implications of missing data mechanisms for regression analyses 32
1.6.1 Partially observed response . . . . . . . . . . . . . . . . 33
1.6.2 Missing covariates . . . . . . . . . . . . . . . . . . . . . 37
1.6.3 Missing covariates and response . . . . . . . . . . . . . . 40
1.6.4 Subtle issues I: the odds ratio . . . . . . . . . . . . . . . 40
1.6.5 Implication for linear regression . . . . . . . . . . . . . . 43
1.6.6 Subtle issues II: sub sample ignorability . . . . . . . . . 44
1.6.7 Summary: when restricting to complete records is valid 45
1.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
1.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2 The Multiple Imputation Procedure and Its Justification 52
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.2 Intuitive outline of the MI procedure . . . . . . . . . . . . . . 54
2.3 The generic MI Procedure . . . . . . . . . . . . . . . . . . . . . 61
2.4 Bayesian justification of MI . . . . . . . . . . . . . . . . . . . . 64
2.5 Frequentist Inference . . . . . . . . . . . . . . . . . . . . . . . 66
2.6 Choosing the number of imputations . . . . . . . . . . . . . . . 73
2.7 Some simple examples . . . . . . . . . . . . . . . . . . . . . . . 75
2.8 MI in More General Settings . . . . . . . . . . . . . . . . . . . 84
2.8.1 Proper imputation . . . . . . . . . . . . . . . . . . . . . 84
2.8.2 Congenial imputation and substantive model . . . . . . 85
2.8.3 Uncongenial imputation and substantive models . . . . 87
2.8.4 Survey Sample Settings . . . . . . . . . . . . . . . . . . 94
2.9 Constructing congenial imputation models . . . . . . . . . . . . 95
2.10 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
2.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
1.6.3 Missing covariates and response . . . . . . . . . . . . . . 40
1.6.4 Subtle issues I: the odds ratio . . . . . . . . . . . . . . . 40
1.6.5 Implication for linear regression . . . . . . . . . . . . . . 43
1.6.6 Subtle issues II: sub sample ignorability . . . . . . . . . 44
1.6.7 Summary: when restricting to complete records is valid 45
1.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
1.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2 The Multiple Imputation Procedure and Its Justification 52
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.2 Intuitive outline of the MI procedure . . . . . . . . . . . . . . 54
2.3 The generic MI Procedure . . . . . . . . . . . . . . . . . . . . . 61
2.4 Bayesian justification of MI . . . . . . . . . . . . . . . . . . . . 64
2.5 Frequentist Inference . . . . . . . . . . . . . . . . . . . . . . . 66
2.6 Choosing the number of imputations . . . . . . . . . . . . . . . 73
2.7 Some simple examples . . . . . . . . . . . . . . . . . . . . . . . 75
2.8 MI in More General Settings . . . . . . . . . . . . . . . . . . . 84
2.8.1 Proper imputation . . . . . . . . . . . . . . . . . . . . . 84
2.8.2 Congenial imputation and substantive model . . . . . . 85
2.8.3 Uncongenial imputation and substantive models . . . . 87
2.8.4 Survey Sample Settings . . . . . . . . . . . . . . . . . . 94
2.9 Constructing congenial imputation models . . . . . . . . . . . . 95
2.10 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
2.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
II Multiple imputation for simple data structures 104
3 Multiple imputation of quantitative data 105
3.1 Regression imputation with a monotone missingness pattern . . 105
3.1.1 MAR mechanisms consistent with a monotone pattern . 107
3.1.2 Justification . . . . . . . . . . . . . . . . . . . . . . . . 109
3.2 Joint modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
3.2.1 Fitting the imputation model . . . . . . . . . . . . . . 111
3.2.2 Adding covariates . . . . . . . . . . . . . . . . . . . . . 115
3.3 Full conditional specification . . . . . . . . . . . . . . . . . . . 118
3.3.1 Justification . . . . . . . . . . . . . . . . . . . . . . . . . 119
3.4 Full conditional specification versus joint modelling . . . . . . . 121
3.5 Software for multivariate normal imputation . . . . . . . . . . . 121
3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
3.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
4 Multiple imputation of binary and ordinal data 125
4.1 Sequential imputation with monotone missingness pattern . . 125
4.2 Joint modelling with the multivariate normal distribution . . . 127
4.3 Modelling binary data using latent normal variables . . . . . . 130
4.3.1 Latent normal model for ordinal data . . . . . . . . . . 137
4.4 General location model . . . . . . . . . . . . . . . . . . . . . . 141
4.5 Full conditional specification . . . . . . . . . . . . . . . . . . . 142
4.5.1 Justification . . . . . . . . . . . . . . . . . . . . . . . . . 143
4.6 Issues with over-fitting . . . . . . . . . . . . . . . . . . . . . . 144
4.7 Pros and cons of the various approaches . . . . . . . . . . . . . 150
4.8 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
4.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
4.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
5 Imputation of unordered categorical data 156
5.1 Monotone missing data . . . . . . . . . . . . . . . . . . . . . . 157
5.2 Multivariate normal imputation for categorical data . . . . . . 158
5.3 Maximum indicant model . . . . . . . . . . . . . . . . . . . . . 159
5.3.1 Continuous and categorical variable . . . . . . . . . . . 162
5.3.2 Imputing missing data . . . . . . . . . . . . . . . . . . . 164
5.4 General location model . . . . . . . . . . . . . . . . . . . . . . 165
5.5 FCS with categorical data . . . . . . . . . . . . . . . . . . . . 169
5.6 Perfect prediction issues with categorical data . . . . . . . . . . 170
5.7 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
5.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
5.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
III Multiple imputation in practice 175
6 Non-linear relationships, interactions, and other derived variables 176
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
6.1.1 Interactions . . . . . . . . . . . . . . . . . . . . . . . . . 178
6.1.2 Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
6.1.3 Ratios . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
6.1.4 Sum scores . . . . . . . . . . . . . . . . . . . . . . . . . 181
6.1.5 Composite endpoints . . . . . . . . . . . . . . . . . . . . 182
6.2 No missing data in derived variables . . . . . . . . . . . . . . . 184
6.3 Simple methods . . . . . . . . . . . . . . . . . . . . . . . . . . 186
6.3.1 Impute then transform . . . . . . . . . . . . . . . . . . . 187
6.3.2 Transform then impute / just another variable . . . . . 187
6.3.3 Adapting standard imputation models and passive imputation .. . . . . . . . . . . . . . . . . . . . . . 190
6.3.4 Predictive mean matching . . . . . . . . . . . . . . . . . 191
6.3.5 Imputation separately by groups for interactions . . . . 195
6.4 Substantive-model-compatible imputation . . . . . . . . . . . . 200
6.4.1 The basic idea . . . . . . . . . . . . . . . . . . . . . . . 200
6.4.2 Latent-normal joint model SMC imputation . . . . . . . 207
6.4.3 Factorised conditional model SMC imputation . . . . . 209
6.4.4 Substantive model compatible fully conditional specification . . . . . . . . . . . . . . . . . . . . . . . . . 212
6.4.5 Auxiliary variables . . . . . . . . . . . . . . . . . . . . . 213
6.4.6 Missing outcome values . . . . . . . . . . . . . . . . . . 214
6.4.7 Congeniality vs. compatibility . . . . . . . . . . . . . . . 214
6.4.8 Discussion of SMC . . . . . . . . . . . . . . . . . . . . . 216
6.5 Returning to the problems . . . . . . . . . . . . . . . . . . . . . 217
6.5.1 Ratios . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
6.5.2 Splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
6.5.3 Fractional polynomials . . . . . . . . . . . . . . . . . . . 218
6.5.4 Multiple imputation with conditional questions or ‘skips’223
6.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
7 Survival data 231
7.1 Missing covariates in time to event data . . . . . . . . . . . . . 231
7.1.1 Approximately compatible approaches . . . . . . . . . . 232
7.1.2 Substantive model compatible approaches . . . . . . . . 241
7.2 Imputing censored survival times . . . . . . . . . . . . . . . . . 245
7.3 Non-parametric, or ‘hot deck’ imputation . . . . . . . . . . . . 248
7.3.1 Non-parametric imputation for survival data . . . . . . 251
7.4 Case–cohort designs . . . . . . . . . . . . . . . . . . . . . . . . 254
7.4.1 Standard analysis of case–cohort studies . . . . . . . . . 254
7.4.2 Multiple imputation for case-cohort studies . . . . . . . 255
7.4.3 Full-cohort . . . . . . . . . . . . . . . . . . . . . . . . . 256
7.4.4 Intermediate approaches . . . . . . . . . . . . . . . . . . 257
7.4.5 Substudy approach . . . . . . . . . . . . . . . . . . . . . 257
7.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
7.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
8 Prognostic models, missing data and multiple imputation 265
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
8.2 Motivating example . . . . . . . . . . . . . . . . . . . . . . . . 266
8.3 Missing data at model implementation . . . . . . . . . . . . . 267
8.4 Multiple imputation for prognostic modelling . . . . . . . . . . 268
8.5 Model building . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
8.5.1 Model building with missing data . . . . . . . . . . . . . 268
8.5.2 Imputing predictors when model building is to be performed . . . . . . . . . . . . . . . . . . . . . . . . . 270
8.6 Model performance . . . . . . . . . . . . . . . . . . . . . . . . 271
8.6.1 How should we pool MI results for estimation of performance? . . . . . . . . . . . . . . . . . . . . . . . 271
8.6.2 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . 272
8.6.3 Discrimination . . . . . . . . . . . . . . . . . . . . . . . 273
8.6.4 Model performance measures with clinical interpretability273
8.7 Model validation . . . . . . . . . . . . . . . . . . . . . . . . . . 274
8.7.1 Internal model validation . . . . . . . . . . . . . . . . . 274
8.7.2 External model validation . . . . . . . . . . . . . . . . . 275
8.8 Incomplete data at implementation . . . . . . . . . . . . . . . 276
8.8.1 MI for incomplete data at implementation . . . . . . . . 276
8.8.2 Alternatives to multiple imputation . . . . . . . . . . . 278
8.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
9 Multilevel multiple imputation 283
9.1 Multilevel imputation model . . . . . . . . . . . . . . . . . . . 284
9.1.1 Imputation of level 1 variables . . . . . . . . . . . . . . 287
9.1.2 Imputation of level 2 variables . . . . . . . . . . . . . . 291
9.1.3 Accommodating the substantive model . . . . . . . . . . 296
9.2 MCMC algorithm for imputation model . . . . . . . . . . . . . 297
9.2.1 Checking model convergence . . . . . . . . . . . . . . . 305
9.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
9.3.1 Cross-classification and 3-level data . . . . . . . . . . . 307
9.3.2 Random level 1 covariance matrices . . . . . . . . . . . 308
9.3.3 Model fit . . . . . . . . . . . . . . . . . . . . . . . . . . 310
9.4 Other imputation methods . . . . . . . . . . . . . . . . . . . . 311
9.4.1 1-step and 2-step FCS . . . . . . . . . . . . . . . . . . . 312
9.4.2 Substantive model compatible imputation . . . . . . . . 313
9.4.3 Non-parametric methods . . . . . . . . . . . . . . . . . 314
9.4.4 Comparisons of different methods . . . . . . . . . . . . 314
9.5 Individual participant data meta-analysis . . . . . . . . . . . . 315
9.5.1 When to apply Rubin’s rules . . . . . . . . . . . . . . . 318
9.5.2 Homoscedastic vs heteroscedastic imputation model . . 320
9.6 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
9.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
9.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
10 Sensitivity analysis: MI unleashed 326
10.1 Review of MNAR modelling . . . . . . . . . . . . . . . . . . . 328
10.2 Framing sensitivity analysis: Estimands . . . . . . . . . . . . . 331
10.3 Pattern mixture modelling with MI . . . . . . . . . . . . . . . 335
10.3.1 Missing covariates . . . . . . . . . . . . . . . . . . . . . 341
10.3.2 Sensitivity with multiple variables: the NAR FCS procedure . . . . . . .. . . . . . . . . . . . . . . . . . . 344
10.3.3 Application to survival analysis . . . . . . . . . . . . . . 346
10.4 Pattern mixture approach with longitudinal data via MI . . . . 351
10.4.1 Change in slope post-deviation . . . . . . . . . . . . . . 353
10.5 Reference based imputation . . . . . . . . . . . . . . . . . . . . 356
10.5.1 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
10.5.2 Information Anchoring . . . . . . . . . . . . . . . . . . 368
10.6 Approximating a selection model by importance weighting . . 372
10.6.1 Weighting the imputations . . . . . . . . . . . . . . . . 375
10.6.2 Stacking the imputations and applying the weights . . . 376
10.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
10.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
11 Multiple imputation for measurement error and misclassification 392
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
11.2 Multiple imputation with validation data . . . . . . . . . . . . 394
11.2.1 Measurement error . . . . . . . . . . . . . . . . . . . . . 396
11.2.2 Misclassification . . . . . . . . . . . . . . . . . . . . . . 397
11.2.3 Imputing assuming error is non-differential . . . . . . . 399
11.2.4 Non-linear outcome models . . . . . . . . . . . . . . . . 400
11.3 Multiple imputation with replication data . . . . . . . . . . . . 401
11.3.1 Measurement error . . . . . . . . . . . . . . . . . . . . . 403
11.3.2 Misclassification . . . . . . . . . . . . . . . . . . . . . . 408
11.4 External information on the measurement process . . . . . . . 409
11.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
11.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
12 Multiple imputation with weights 416
12.1 Using model based predictions in strata . . . . . . . . . . . . . 417
12.2 Bias in the MI Variance Estimator . . . . . . . . . . . . . . . . 418
12.3 MI with weights . . . . . . . . . . . . . . . . . . . . . . . . . . 422
12.3.1 Conditions for consistency of θbMI . . . . . . . . . . . . 422
12.3.2 Conditions for the consistency of Vb MI . . . . . . . . . . 424
12.4 A multilevel approach . . . . . . . . . . . . . . . . . . . . . . . 426
12.4.1 Evaluation of the multilevel multiple imputation approach for handling survey weights . . . 429
12.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
12.5 Further topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
12.5.1 Estimation in Domains . . . . . . . . . . . . . . . . . . 437
12.5.2 Two-stage analysis . . . . . . . . . . . . . . . . . . . . 437
12.5.3 Missing values in the weight model . . . . . . . . . . . . 438
12.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
12.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439
13 Multiple imputation for causal inference 443
13.1 Multiple imputation for causal inference in point exposure studies444
13.1.1 Randomised trials . . . . . . . . . . . . . . . . . . . . . 445
13.1.2 Observational studies . . . . . . . . . . . . . . . . . . . 446
13.2 Multiple imputation and propensity scores . . . . . . . . . . . . 450
13.2.1 Propensity scores for confounder adjustment . . . . . . 450
13.2.2 Multiple imputation of confounders . . . . . . . . . . . . 452
13.2.3 Imputation model specification . . . . . . . . . . . . . . 456
13.3 Principal stratification via multiple imputation . . . . . . . . . 457
13.3.1 Principal strata effects . . . . . . . . . . . . . . . . . . 458
13.3.2 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 459
13.4 Multiple imputation for instrumental variable analysis . . . . . 461
13.4.1 Instrumental variable analysis for non-adherence . . . . 461
13.4.2 Instrumental variable analysis via multiple imputation . 464
13.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467
13.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
14 Using multiple imputation in practice 472
14.1 A general approach . . . . . . . . . . . . . . . . . . . . . . . . 473
14.2 Objections to multiple imputation . . . . . . . . . . . . . . . . 477
14.3 Reporting of analyses with incomplete data . . . . . . . . . . . 482
14.4 Presenting incomplete baseline data . . . . . . . . . . . . . . . 483
14.5 Model diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . 486
14.6 How many imputations? . . . . . . . . . . . . . . . . . . . . . . 487
14.6.1 Using the jack-knife estimate of the Monte-Carlo standard error . . . . . . . . . . . . . . . . . . . . 490
14.7 Multiple imputation for each substantive model, project or
dataset? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
14.8 Large datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 493
14.8.1 Large datasets and joint modelling . . . . . . . . . . . 494
14.8.2 Shrinkage by constraining parameters . . . . . . . . . . 496
14.8.3 Comparison of the two approaches . . . . . . . . . . . . 499
14.9 Multiple Imputation and record linkage . . . . . . . . . . . . . 500
14.10Setting random number seeds for multiple imputation analyses 502
14.11Simulation studies including multiple imputation . . . . . . . . 503
14.11.1Random number seeds for simulation studies including
multiple imputation . . . . . . . . . . . . . . . . . . . . 503
14.11.2Repeated simulation of all data or only the missingness
mechanism? . . . . . . . . . . . . . . . . . . . . . . . . 504
14.11.3How many imputations for simulation studies? . . . . . 505
14.11.4Multiple imputation for data simulation . . . . . . . . . 507
14.12Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
A Markov Chain Monte Carlo 512
B Probability distributions 517
B.1 Posterior for the multivariate normal distribution . . . . . . . 521
C Overview of multiple imputation in R, Stata 524
C.1 Basic multiple imputation using R . . . . . . . . . . . . . . . . 524
C.2 Basic MI using Stata . . . . . . . . . . . . . . . . . . . . . . . . 526
Bibliography 530
Index 555
What is included with this book?
The New copy of this book will include any supplemental materials advertised. Please check the title of the book to determine if it should include any access cards, study guides, lab manuals, CDs, etc.
The Used, Rental and eBook copies of this book are not guaranteed to include any supplemental materials. Typically, only the book itself is included. This is true even if the title states it includes any access cards, study guides, lab manuals, CDs, etc.
Please wait while the item is added to your bag...
