Eliciting and Analyzing Expert Judgment

Note: Supplemental materials are not guaranteed with Rental or Used book purchases.

ISBN: 9780898714746 | 0898714745
Cover: Hardcover
Copyright: 5/1/2001

Rent

(Recommended)

$91.36

Term

Due

Price

Semester

Dec 13

$101.52

Quarter

Aug 19

$96.44

Short Term

Jul 26

$91.36

*This item is part of an exclusive publisher rental program and requires an additional convenience fee. This fee will be reflected in the shopping cart.
Buy New

Special Order: 1-2 Weeks

$138.70

Expert judgment is invaluable for assessing products, systems, and situations for which measurements or test results are sparse or nonexistent. Eliciting and Analyzing Expert Judgment: A Practical Guide takes the reader step by step through the techniques of eliciting and analyzing expert judgment, with special attention given to helping the reader develop elicitation methods and tools adaptable to a variety of unique situations and work areas. The analysis procedures presented in the book may require a basic understanding of statistics and probabilities, but the authors have provided detailed explanations of the techniques used and have taken special care to define all statistical jargon. Originally published in 1991, this book is designed so that those familiar with the use of expert judgment can quickly find the material appropriate for their advanced background.

Preface to ASA-SIAM Edition

xxi

Preface

xxiii

List of Figures

xxv

List of Tables

xxvii

List of Examples

xxix

PART I: INTRODUCTION TO EXPERT JUDGMENT

Introduction

(14)

What Is Expert Judgment?

(6)

When Expert Judgment Is Used

(2)

General Attributes of Expert Judgment

(1)

Expert Judgment Covered in This Book

(2)

How Expert Judgment Is Elicited

(2)

Philosophy Guiding the Elicitation

(1)

Philosophy Guiding the Analysis

(1)

How To Use This Book

(4)

Common Questions and Pitfalls Concerning Expert Judgment

(18)

Summary of Common Questions and Pitfalls

(2)

List of Common Questions

(1)

List of Pitfalls

(1)

Questions

(8)

What Does It Mean When the Experts Disagree?

(1)

Is Expert Judgment Valid Data?

(1)

Is the Gathering and Analyzing of Expert Judgment Scientific?

(2)

Are Experts Bayesian?

(1)

Do Experts Provide Better Data?

(2)

Can Experts Be Fully Calibrated?

(1)

Pitfalls

(8)

Interviewers, Knowledge Engineers, and Analysts Can Introduce Bias

(2)

Experts Are Limited in the Number of Things That They Can Mentally Juggle

(1)

The Level of Detail in the Data (Granularity) Affects the Analyses

(2)

The Conditioning Effect Poses Difficulties in Gathering and Analyzing Experts Data

(3)

Background on Human Problem Solving and Bias

(22)

Why Is It Necessary to Have an Understanding of Human Problem Solving?

(1)

What Is Involved in Solving Problems and in Responding?

(3)

The Four Cognitive Tasks

(2)

Two Models of Human Memory and Information Processing

(1)

Bias

(19)

Two Views of Bias

(1)

Potential Impact of Bias

(1)

Sources of Bias

(1)

Motivational Bias

(2)

Cognitive Bias

(1)

Program for Handling Bias

(3)

Determining Which Steps to Apply

(1)

The Reason for Focusing on Bias

(3)

The Selection of the Definition of Bias, Motivational or Cognitive

(1)

Interest in Particular Types of Bias

(6)

PART II: ELICITATION PROCEDURES

Selecting the Question Areas and Questions

(12)

Steps Involved in Selecting the Questions

(4)

Illustrations of the Variation in Project Goals, Question Areas, and Questions

(1)

Sources of Variation

(2)

Executing the Steps with the Assistance of Sponsors, Project Personnel, and Experts

(2)

Determining in Which Steps the Advisory or External Experts Will Assist

(2)

The External Experts Do Not Participate in Question Area Selection, Question Identification, or Refinement, Except Minimally Just Before Having Their Judgments Elicited

(1)

The External Experts Receive the Question Areas and Develop the Questions and Refine Them, or They Receive the Questions and Refine Them

(1)

The External Experts Help Select the Question Areas and Assist in the Identification and Refinement of the Questions

(1)

Checklist for Selected Questions

(4)

Common Difficulties---Their Signs and Solutions

(1)

Difficulty: Sponsor Cannot Provide Clear Information on the Project's Goal, the Data to Be Gathered, or the Question Areas

(1)

Difficulty: The Question Developed From the Question Area Remains Too Broad

(1)

Difficulty: Too Many Questions Have Been Selected for the Amount of Time Available

(1)

Refining the Questions

(16)

Reasons for Structuring the Questions

(2)

Techniques for Structuring the Questions

(9)

Presentation of the Question Information

(1)

Types of Question Information Needed

(1)

Background

(1)

Assumptions

(1)

Definitions

(1)

Ordering of Information

(1)

Roles of Project Personnel and Experts

(1)

Decomposition of the Question

(1)

Considerations in Question Decomposition

(1)

Roles of Project Personnel and Experts

(1)

Representation of the Question

(1)

Considerations in Representation

(2)

Roles of Project Personnel and Experts

(1)

Question Phrasing

(1)

Considerations in Question Phrasing

(1)

Roles of Project Personnel and Experts

(1)

When the Refinement of the Questions Should Be Preceded by the Selection of the Experts

(5)

Common Difficulties---Their Signs and Solutions

(1)

Difficulty: The External Experts Did Not Provide Enough Input in Refining the Question

(1)

Difficulty: The Question Decomposition Becomes Too Complicated or Too Detailed

(1)

Difficulty: The Questions are III Defined or Open to Differing Interpretations

(3)

Selecting and Motivating the Experts

(14)

For Applications Whose Data Will Be the Expert's Answers

(8)

Who Is Considered an Expert?

(1)

What Constitutes Expertise?

(1)

When Expertise Matters

(1)

Additional Considerations in Selecting Experts

(1)

Multiple and Diverse Experts

(1)

Number of Experts

(1)

Selection Schemes

(1)

Motivating Experts to Participate

(1)

Motivating the Experts Through Pay

(1)

Motivating the Experts Through Communication of Intrinsic Aspects of the Project

(3)

For Applications Whose Data Will Be Problem-Solving Processes

(6)

What Is Needed in an Expert

(1)

Method-Driven Selection

(1)

Motivating the Expert

(2)

Common Difficulties---Their Signs and Solutions

(1)

Difficulty: The Experts Do Not Wish to Participate

(1)

Difficulty: Everyone, Including Nonexperts, Wishes to Participate

(1)

Difficulty: The Real Experts Are Too Busy to Participate at the Needed Level

(1)

Difficulty: The System for Selecting Experts Is Criticized

(1)

Difficulty: A Conflict Exists Between Those Wanting to Identify the Expert's Data and Those Wanting to Preserve Anonymity

(1)

Selecting the Components of Elicitation

(24)

Determining Which of the Five Components Are Needed---A Checklist

100

(2)

Selecting From Within Each Component

102

(21)

Selecting From Elicitation Situations

102

(1)

Interactive Group

103

(1)

Delphi

103

(1)

Individual Interview

104

(1)

Selecting From Modes of Communication

104

(1)

Face to Face

104

(1)

Telephone

105

(1)

Mail

105

(1)

Selecting From Elicitation Techniques

105

(1)

Verbal Report

106

(1)

Verbal Probe

106

(1)

Ethnographic Technique

107

(1)

Selecting From Response Modes

107

(1)

Estimate of Physical Quantity

108

(1)

Probability Estimate

109

(1)

Odds Ratio

110

(1)

Probability Distribution

110

(1)

Continuous Scales

111

(1)

Pairwise Comparisons

112

(1)

Ranks or Ratings

113

(2)

Bayesian Updating

115

(1)

Selecting From Dispersion Measures

116

(1)

Ranges

116

(1)

Volunteered Ranges

117

(1)

Percentiles

117

(1)

Variances and Standard Deviations

118

(1)

Selecting the Type of Aggregation

118

(1)

Behavioral Aggregation

118

(1)

Mathematical Aggregation

119

(1)

Selecting From Methods for Documentation

119

(1)

Answer-Only Documentation

120

(1)

Answer Plus Problem-Solving Documentation

120

(1)

Summary Documentation

120

(1)

Detailed Verbatim Documentation

121

(1)

Detailed Structured Documentation

121

(1)

A Common Difficulty---Its Signs and Solutions

121

(1)

Difficulty: The Literature on the Different Methods Is Either Scarce or Conflicing

121

(2)

Designing and Tailoring the Elicitation

123

(28)

Considerations in Designing the Elicitation

123

(28)

Logistics and Costs of Different Elicitation Situations and Modes of Communication

123

(1)

Tailoring the Elicitation Situations

124

(1)

Tailoring the Modes of Communication

125

(1)

Face to Face

125

(1)

Telephone

125

(1)

Mail

126

(1)

Reasons for Gathering the Experts Together

126

(1)

Expenses for Gathering the Experts Together

127

(1)

Structuring the Elicitation Process

127

(1)

Why Structuring Is Done

128

(1)

Degrees of Structuring

128

(1)

Structuring Options Applied to the Stages of Elicitation

128

(2)

General Rules in Structuring the Elicitation

130

(1)

Handling Bias---A Starting Point

131

(1)

Anticipate Which Biases Are Likely to Occur in the Planned Elicitation---Step 1

132

(7)

Redesign the Planned Elicitation to Make It Less Prone to the Anticipated Biases---Step 2

139

(1)

Documentation During and After the Elicitation Sessions

139

(1)

What Documentation Can Include

139

(2)

Logistics of Who Will Record and When

141

(5)

Sample Documentation Formats

146

(1)

Presentation of the Question---A Quick Check

146

(1)

Common Difficulties---Their Signs and Solutions

147

(1)

Difficulty: Realizing that the Time Allotted to Planning the Elicitation Is Not Enough

147

(1)

Difficulty: Ignoring the Possibility of Bias

147

(4)

Practicing the Elicitation and Training the Project Personnel

151

(12)

Practicing the Elicitation

151

(6)

What Needs to Be Practiced?

151

(2)

What Requires Pilot Testing?

153

(1)

How to Pilot Test

153

(1)

Sample Sizes and Selection

153

(1)

Sequence for Pilot Testing

154

(1)

How to Conduct Intensive Pilot Tests

155

(1)

Intensive Pilot Test---Part 1

155

(1)

Intensive Pilot Test---Part 2

156

(1)

How to Conduct Limited Pilot Tests

156

(1)

Training Project Personnel

157

(6)

When Training Is Needed

157

(1)

How to Train

158

(1)

Common Difficulties---Their Signs and Solutions

159

(1)

Difficulty: Pilot Tests Show that the Sample Experts Experience Significant Difficulty with the Response Mode

159

(1)

Difficulty: Pilot Tests Indicate that the Elicitation Will Likely Need More Time Than Is Available

159

(1)

Difficulty: Project Personnel Resist the Training

160

(1)

Difficulty: The Rehearsal Shows that the Documentation Scheme Does Not Meet the Project's Other Requirements

160

(1)

Difficulty: The Documentation of the Experts' Problem Solving Has Been Done Differently or at Different Levels

161

(2)

Conducting the Elicitation

163

(26)

Scheduling the Elicitations

163

(2)

Scheduling the Meetings

163

(1)

Confirming the Scheduled Meetings

164

(1)

Preparing for and Conducting the Elicitations

165

(24)

Tips on Preparing for the Elicitations

165

(1)

How to Prepare for an Individual Interview

165

(1)

How to Prepare for an Interactive Group Situation

166

(1)

How to Prepare for a Delphi Situation

167

(1)

Tips on Conducting the Elicitations

168

(1)

Introducing the Experts to the Elicitation Process

168

(1)

Make the Experts Aware of the Potential Intrusion of Particular Biases and Familiarize Them with the Elicitation Procedures---Step 3

168

(1)

How to Introduce the Experts to an Individual Interview Situation

169

(1)

How to Introduce the Experts to an Interactive Group Situation

170

(1)

How to Introduce the Experts to a Delphi Situation

170

(1)

Monitor the Elicitation for the Occurrence of Bias---Step 4

171

(1)

Gathering and Recording Expert Data

172

(1)

Using the Individual Interview, Interactive Group, and Delphi Situations

172

(1)

Using the Three Techniques for Eliciting Problem-Solving Data

172

(5)

Adjust, in Real Time, to Counter the Occurrence of Biases---Step 5

177

(4)

Common Difficulties---Their Signs and Solutions

181

(1)

Difficulty: The Experts Resist the Elicitation Process or Resist Giving Judgments Under Uncertainty

181

(1)

Difficulty: The Experts Appear Confused About What They Should Do or How They Should Do It

182

(1)

Difficulty: The Final Statement/Representation of the Question or the Expert's Latest Data Were Not Documented

183

(1)

Difficulty: Bias May Have Occurred but Its Presence Was Not Monitored During the Elicitations

183

(1)

Difficulty: There Is Wide Disagreement Between the Experts' Data

184

(5)

PART III: ANALYSIS PROCEDURES

Introducing the Techniques for Analysis of Expert Judgment Data

189

(46)

Notation for Part III

189

(3)

Description of Some Basic Statistical Concepts

192

(9)

Random Variables and Probability Distributions

192

(2)

Estimators and Significance Tests

194

(1)

Relationship of Estimators and Distributions

194

(2)

Multivariate Estimators

196

(2)

Significance Testing of Estimators

198

(3)

Descriptions and Uses of Simulation Techniques

201

(10)

Monte Carlo Techniques

201

(5)

Advantages and Disadvantages

206

(1)

Uses for Monte Carlo Simulation

207

(1)

Bootstrap Sampling and Estimation

207

(1)

Advantages and Disadvantages

208

(1)

Uses for the Bootstrap

208

(1)

How to Implement the Bootstrap

208

(3)

Descriptions and Uses of Data Analysis Techniques

211

(19)

Multivariate Techniques

211

(1)

Regression Analysis

211

(3)

Cluster Analysis

214

(4)

Factor Analysis

218

(2)

Discriminant Analysis

220

(2)

Analysis of Variance

222

(3)

Saaty's Technique for Pairwise Data Analysis

225

(4)

Advantages and Disadvantages of Saaty's Method

229

(1)

Uses for Saaty's Method

230

(1)

Descriptions and Uses of Bayesian Techniques

230

(5)

The Bayesian Philosophy

230

(2)

Advantages, Disadvantages, and Uses of Bayesian Methods

232

(3)

Initial Look at the Data---The First Analyses

235

(14)

What Data Has Been Gathered?

235

(1)

Overview of the Data

236

(2)

Establishing Granularity

236

(1)

Establishing Conditionality

237

(1)

How to Quantify

238

(8)

When Is It Necessary to Quantify?

239

(1)

Quantification Schemes

240

(1)

Dummy Variable

240

(1)

Dichotomous Quantification

240

(1)

Rank or Rating Quantification

241

(2)

Number Line Quantification

243

(2)

Categorical Variables

245

(1)

Description Variables

246

(1)

Forming a Data Base of Information

246

(3)

Understanding the Data Base Structure

249

(20)

Conditionality---Examining Relationships Between Answers and Ancillary Data

250

(2)

Correlations

250

(1)

Graphs

251

(1)

Analyzing the Answer Data

252

(8)

Investigating Multimodality

253

(5)

Investigating Between/Within Variation Structure

258

(2)

Analyzing the Ancillary Data

260

(5)

Analyzing the Ancillary Data with the Answer Data

265

(4)

Correlation and Bias Detection

269

(24)

Defining Correlation and Dependence

269

(2)

Bias and Correlation Relationships

271

(1)

Detecting Correlation in the Analysis

272

(16)

Using Granularity

273

(1)

Using Hypothesized Sources of Correlation

273

(2)

Using Correlation Analysis

275

(2)

Using Multivariate Analysis

277

(2)

Using Analysis of Variance

279

(1)

Using Simulation Techniques

280

(5)

Using Elicitation Methods

285

(1)

Using Assumptions

286

(2)

Analysis Summary and Conclusions

288

(5)

Model Formation

293

(22)

General Linear Models

294

(7)

Full-Scale General Linear Models

294

(2)

Combination Models

296

(2)

Anchoring and Adjustment Scores

298

(1)

Cumulative Scores

298

(1)

Collapsing Variables

299

(2)

Other Multivariate Models

301

(7)

Factors from Factor Analysis

301

(2)

Discriminant Analysis

303

(4)

Cluster Analysis

307

(1)

Conditional Models

308

(5)

Saaty

309

(3)

Decomposition Diagrams

312

(1)

Model Selection Suggestions and Cautions

313

(2)

Combining Responses---Aggregation

315

(38)

Aggregation Scheme Selection

315

(27)

Aggregation Estimators

315

(3)

Determining Weights

318

(1)

Data-Based Determinations

319

(1)

Saaty Weight Determinations

320

(2)

Model-Based Determinations

322

(1)

Conditional Modeling

322

(4)

GLM Modeling

326

(1)

Direct Estimation

327

(2)

Equal Weights

329

(2)

Aggregation Distributions

331

(1)

Using Bayesian Methods

331

(2)

Using Assumed Distributions

333

(3)

Using Empirical Distributions

336

(3)

Using Monte Carlo Simulation

339

(3)

Application Environments

342

(10)

Decision Maker and One Expert

343

(2)

Decision Maker and n Experts

345

(6)

Analyst and n Experts

351

(1)

Aggregation and Uncertainty Analysis

352

(1)

Characterizing Uncertainties

353

(20)

Living with Uncertainties

353

(1)

Obtaining Uncertainty Measures

354

(3)

Using Elicitation

354

(1)

Error Bars

354

(1)

Variances or Standard Deviations

354

(1)

Percentiles

355

(1)

Ranges

355

(1)

Using Post-Elicited Data

355

(2)

Modeling Uncertainties

357

(12)

Bayesian Methods

357

(1)

Using One Prior

357

(3)

Using Multiple Priors

360

(2)

Simulation Methods

362

(1)

Monte Carlo Simulation

362

(3)

Bootstrap Simulation

365

(2)

Decision Analytic Methods

367

(2)

Comparison of the Methods

369

(4)

Making Inferences

373

(14)

What Inferences Can Be Made

373

(3)

Improving the Inference Process

376

(8)

Design-Based Improvements

376

(1)

Synergism of Elicitation and Analysis

376

(1)

Granularity

377

(2)

Quantification

379

(1)

Conditionality

380

(1)

Analysis-Based Improvements

381

(1)

Cross Validation and Redundancy

381

(2)

Simulation

383

(1)

Inferences With Modeling, Aggregation, and Uncertainties

384

(1)

Final Comments

385

(2)

Appendices

387

(36)

Appendix A---SAATY

387

(4)

Appendix B---MCBETA

391

(14)

Appendix C---EMPIRICAL

405

(10)

Appendix D---BOOT

415

(8)

Glossary of Expert Judgment Terms

423

(18)

References

441

(10)

Index

451

Please wait while the item is added to your bag...

FREE SHIPPING

Amazon no longer offers textbook rentals. We do!

Eliciting and Analyzing Expert Judgment

Summary

Table of Contents

Supplemental Materials