The Value of Learning How Organizations Capture Value and ROI and Translate It into Support, Improvement, and Funds
, by Phillips, Jack J.; Phillips , Patricia P.
- ISBN: 9780787985325 | 0787985325
- Cover: Hardcover
- Copyright: 7/27/2007
The Value of Learning is a hands-on guide for the implementation of learning and development programs that can be applied across all types of programs, ranging from leadership development to basic skills training for new employees. In this book, Patti Phillips and Jack J. Phillips offer a proven approach to measurement and evaluation for learning and development that can be replicated throughout an organization, enable comparisons of results from one program to another, and ultimately improve ROI.
Patricia Pulliam Phillips is president and CEO of the ROI Institute, Inc., the leading source of ROI competency building, implementation support, networking, and research. An expert in measurement and evaluation, she provides support to organizations around the world that want to prove the value of their programs.
Jack J. Phillips is a world-renowned expert on accountability, measurement, and evaluation. He provides consulting services for Fortune 500 companies and major global organizations.
| List of Exhibits, Figures, and Tables | p. xxi |
| Preface | p. xxvii |
| Acknowledgments | p. xxxi |
| Building a Comprehensive Evaluation Process | p. 1 |
| Challenges | p. 2 |
| Global Evaluation Trends | |
| Measurement and Evaluation Challenges | |
| Benefits of Measurement and Evaluation | |
| The Myths of Measurement and Evaluation | |
| Key Steps and Issues | p. 13 |
| Stakeholders | |
| Levels and Steps | |
| Chain of Impact | |
| ROI Process Model | |
| Objectives | |
| Evaluation Planning | |
| Data Collection | |
| Analysis | |
| Isolation of the Effects of Learning and Development | |
| Conversion of Data to Monetary Values | |
| The Cost of Programs | |
| The Return on Investment Calculation | |
| Intangible Benefits | |
| Data Reporting | |
| Operating Standards | |
| Implementation Issues | |
| Final Thoughts | p. 31 |
| Defining Needs and Objectives: Ensuring Business Alignment | p. 32 |
| The Challenge | p. 32 |
| Business Alignment Issues | |
| Begin with the End in Mind | |
| Required Discipline | |
| The Needs Analysis Dilemma | |
| Payoff Needs | p. 36 |
| Key Questions | |
| Obvious vs. Not So Obvious | |
| The Reasons for New Programs | |
| Determining Costs of the Problem | |
| The Value of Opportunity | |
| To Forecast or Not to Forecast | |
| Business Needs | p. 44 |
| Determining the Opportunity | |
| Defining the Business Measure-Hard Data | |
| Defining the Business Need-Soft Data | |
| Using Tangible vs. Intangible-A Better Approach | |
| Finding Sources of Impact Data | |
| Identifying All the Measures | |
| Exploring "What If...?" | |
| Job Performance Needs | p. 53 |
| Analysis Techniques | |
| Taking a Sensible Approach | |
| Learning Needs | p. 58 |
| Subject-Matter Experts | |
| Job and Task Analysis | |
| Observations | |
| Demonstrations | |
| Tests | |
| Management Assessment | |
| Preference Needs | p. 61 |
| Key Issues | |
| Impact Studies | |
| Levels of Objectives for Programs | p. 65 |
| Reaction and Planned Action | |
| Learning Objectives | |
| Application and Implementation Objectives | |
| Business Impact Objectives | |
| ROI Objectives | |
| The Importance of Specific Objectives | |
| Final Thoughts | p. 73 |
| Measuring Inputs and Indicators | p. 74 |
| Measuring Input and Indicators | p. 75 |
| Defines the Input | |
| Reflects Commitment | |
| Facilitates Benchmarking | |
| Explains Coverage | |
| Highlights Efficiencies | |
| Provides Cost Data | |
| Tracking Participants | p. 78 |
| Tracking Hours | p. 80 |
| Tracking Coverage by Jobs and Functional Areas | p. 81 |
| Tracking Topics and Programs | p. 82 |
| Tracking Requests | p. 84 |
| Tracking Delivery | p. 85 |
| Tracking Costs | p. 86 |
| Pressure to Disclose All Costs | |
| The Danger of Costs Without Benefits | |
| Sources of Costs | |
| Learning Program Steps and Costs | |
| Prorated Versus Direct Costs | |
| Employee Benefits Factor | |
| Major Cost Categories | |
| Cost Reporting | |
| Tracking Efficiencies | p. 94 |
| Tracking Outsourcing | p. 95 |
| Tracking for the Scorecard | p. 96 |
| Defining Key Issues | p. 97 |
| Input Is Not Results | |
| Reports to Executives Should Be Minimized | |
| The Data Represent Operational Concerns | |
| This Data Must Be Automated | |
| Final Thoughts | p. 98 |
| Measuring Reaction and Planned Action | p. 100 |
| Why Measure Reaction and Planned Action? | p. 101 |
| Customer Service | |
| Early Feedback Is Essential | |
| Making Adjustments and Changes | |
| Predictive Capability | |
| For Some, This Is the Most Important Data | |
| Comparing Data with Other Programs | |
| Creating a Macro Scorecard | |
| Sources of Data | p. 106 |
| Participants | |
| Participants' Managers | |
| Internal Customers | |
| Facilitators | |
| Sponsors/Senior Managers | |
| Areas of Feedback | p. 107 |
| Content vs. Non-Content | |
| The Deceptive Feedback Cycle | |
| Key Areas for Feedback | |
| Timing of Data Collection | p. 114 |
| Early, Detailed Feedback | |
| Pre-Assessments | |
| Collecting at Periodic Intervals | |
| For Long Programs with Multiple Parts | |
| Data Collection with Questionnaires and Surveys | p. 115 |
| Questionnaire/Survey Design | |
| Intensities | |
| Questionnaire/Survey Response Rates | |
| Sample Surveys | |
| Data Collection with Interviews and Focus Groups | p. 123 |
| Improving Reaction Evaluation | p. 123 |
| Keep Responses Anonymous | |
| Have a Neutral Person Collect the Forms | |
| Provide a Copy in Advance | |
| Explain the Purpose of the Feedback and How It Will Be Used | |
| Explore an Ongoing Evaluation | |
| Consider Quantifying Course Ratings | |
| Collect Information Related to Improvement | |
| Allow Ample Time for Completing the Form | |
| Delayed Evaluation | |
| Ask for Honest Feedback | |
| Using Data | p. 127 |
| Building the Macro-Level Scorecard | |
| Shortcut Ways to Measure Reaction and Planned Action | p. 129 |
| Final Thoughts | p. 130 |
| Measuring Learning and Confidence | p. 132 |
| Why Measure Learning and Confidence? | p. 132 |
| The Importance of Intellectual Capital | |
| The Learning Organization | |
| The Learning Transfer Problem | |
| The Compliance Issue | |
| The Use and Development of Competencies | |
| The Role of Learning in Programs | |
| The Chain of Impact | |
| Certification | |
| Consequences of an Unprepared Workforce | |
| The Challenges and Benefits of Measuring Learning | p. 137 |
| The Challenges | |
| The Benefits | |
| Measurement Issues | p. 140 |
| Objectives | |
| Typical Measures | |
| Timing | |
| Cognitive | |
| Levels of Bloom's Taxonomy | |
| Data Collecton Methods | p. 144 |
| Questionnaires/Surveys | |
| Objective Tests | |
| Criterion-Referenced Tests | |
| Performance Tests | |
| Technology and Task Simulations | |
| Case Studies | |
| Role Playing/Skill Practice | |
| Assessment Center Method | |
| Exercises/Activities | |
| Informal Assessments | |
| Administrative Issues | p. 157 |
| Reliability and Validity | |
| Consistency | |
| Monitoring | |
| Pilot Testing | |
| Readability | |
| Scoring | |
| Reporting | |
| Confronting Test Failures | |
| Using Learning Data | p. 161 |
| Final Thoughts | p. 162 |
| Measuring Application and Implementation | p. 163 |
| Why Measure Application and Implementation? | p. 163 |
| The Value of Information | |
| A Key Transition Time | |
| The Key Focus of Many Programs | |
| The Chain of Impact | |
| Barriers and Enablers | |
| Reward Those Who Are Most Effective | |
| Challenges of Measuring Application and Implementation | p. 166 |
| Linking Application with Learning | |
| Designing Data Collection into Programs | |
| Applying Serious Effort to Level 3 Evaluation | |
| Including Level 3 in the Needs Assessment | |
| Developing ROI with Application Data | |
| Key Issues | p. 168 |
| Methods | |
| Objectives | |
| Topics to Explore | |
| Sources | |
| Timing | |
| Responsibilities | |
| The Use of Questionnaires | p. 172 |
| Progress with Objectives | |
| Relevance/Importance of the Program | |
| Knowledge/Skill Use | |
| Changes with Work/Action Items | |
| Improvements/Accomplishments | |
| Define the Measure | |
| Provide the Change | |
| Monetary Value | |
| Total Impact | |
| List of Other Factors | |
| Improvements Linked with the Program | |
| Perceived Value | |
| Links with Output Measures | |
| Success of the Program Team | |
| Barriers and Enablers | |
| Management Support | |
| Appropriateness of Program and Suggestions for Improvement | |
| Checklist | |
| Improving Response Rates | |
| Data Collection with Interviews | p. 189 |
| Types of Interviews | |
| Interview Guidelines | |
| Data Collection with Focus Groups | p. 191 |
| Applications for Focus Group Evaluation | |
| Guidelines | |
| On-the-job Observation | p. 193 |
| Guidelines for Effective Observation | |
| The Use of Action Plans and Follow-Up Assignments | p. 196 |
| Developing the Action Plan | |
| Successful Use of Action Plans | |
| Action Plan Advantages and Disadvantage | |
| The Use of Performance Contracts | p. 203 |
| Transfer of Learning | p. 204 |
| Developing ROI for Level 3 | p. 206 |
| Data Use | p. 209 |
| Final Thoughts | p. 211 |
| Measuring and Isolating the Impact of Programs | p. 212 |
| Why Measure Business Impact? | p. 213 |
| Higher-Level Data | |
| Breaking the Chain of Impact | |
| A Business Driver for Many Programs | |
| Show Me the Money Data | |
| Easy to Measure | |
| Common Data Types | |
| Types of Impact Measures | p. 215 |
| Hard Versus Soft Data | |
| Tangible Versus Intangible | |
| Scorecards | |
| Specific Measures Linked to Programs | |
| Business Performance Monitoring | p. 219 |
| Identify Appropriate Measures | |
| Convert Current Measures to Usable Ones | |
| Developing New Measures | |
| The Use of Action Plans to Develop Business Impact Data | p. 221 |
| Set Goals and Targets | |
| Define the Unit of Measure | |
| Place a Monetary Value on Each Improvement | |
| Implement the Action Plan | |
| Provide Specific Improvements | |
| Isolate the Effects of the Program | |
| Provide a Confidence Level for Estimates | |
| Collect Action Plans at Specified Time Intervals | |
| Summarize the Data and Calculate the ROI | |
| Advantages of Action Plans | |
| Use of Performance Contracts to Measure Business Impact | p. 227 |
| The Use of Questionnaires to Collect Business Impact Data | p. 228 |
| When You Don't Have a Clue | |
| When the Measure Is a Defined Set | |
| When the Measure Is Known | |
| Response Rates | |
| Selecting the Appropriate Data Collection Method for Each Level | p. 235 |
| Isolating the Effects of the Program | p. 237 |
| Identifying Other Factors: A First Step | |
| Using Control Groups | |
| Using Trend-Line Analysis | |
| Forecasting | |
| Using Estimates | |
| Calculating the Impact of Other Factors | |
| Use of the Techniques | p. 255 |
| Final Thoughts | p. 255 |
| Identifying Benefits and Costs, and Calculating ROI | p. 257 |
| Why Calculate Monetary Benefits? | p. 258 |
| Value Equals Money | |
| Impact Is More Understandable | |
| Money Is Necessary for ROI | |
| Monetary Value Is Needed to Understand Problems | |
| Key Steps to Convert Data to Money | |
| Standard Monetary Values | p. 262 |
| Converting Output Data to Money | |
| Calculating the Cost of Quality | |
| Converting Employee Time Using Compensation | |
| Finding Standard Values | |
| Data Conversion When Standard Values Are Not Available | p. 271 |
| Using Historical Costs from Records | |
| Using Input from Experts to Convert Soft Data | |
| Using Values from External Databases | |
| Linking with Other Measures | |
| Using Estimates from Participants | |
| Using Estimates from the Management Team | |
| Using Staff Estimates | |
| Technique Selection and Finalizing the Values | p. 282 |
| Use the Technique Appropriate for the Type of Data | |
| Move from Most Accurate to Least Accurate | |
| Consider the Resources | |
| When Estimates Are Sought, Use the Source with the | |
| Broadest Perspective on the Issue | |
| Use Multiple Techniques When Feasible | |
| Apply the Credibility Test | |
| Review the Client's Needs | |
| Is This Another Project? | |
| Consider a Potential Management Adjustment | |
| Consider the Short-Term/Long-Term Issue | |
| Consider an Adjustment for the Time Value of Money | |
| Why Monitor Costs? | p. 288 |
| Why Measure ROI? | p. 289 |
| Fundamental Cost Issues | p. 290 |
| Monitor Costs, Even If They Are Not Needed | |
| Cost Will Not Be Precise | |
| Disclose All Costs | |
| Fully Loaded Costs | |
| Reporting Costs Without Benefits | |
| Cost-Tracking Issues | p. 293 |
| Prorated Versus Direct Costs | |
| Employee Benefits Factor | |
| Major Cost Categories | p. 294 |
| Initial Analysis and Assessment | |
| Development of Solutions | |
| Acquisition Costs | |
| Application and Implementation Costs | |
| Maintenance and Monitoring | |
| Support and Overhead | |
| Evaluation and Reporting | |
| Cost Accumulation and Estimation | p. 296 |
| Basic ROI Issues | p. 296 |
| Definition | |
| Annualized Values: A Fundamental Concept | |
| BCR/ROI Calculations | p. 297 |
| Benefit/Cost Ratio | |
| ROI Formula | |
| ROI Targets | |
| ROI Is Not for Every Program | |
| Other ROI Measures | p. 303 |
| Payback Period | |
| Discounted Cash Flow | |
| Final Thoughts | p. 304 |
| Measuring the Hard to Measure and the Hard to Value: Intangible Benefits | p. 307 |
| Why Intangibles Are Important | p. 308 |
| Intangibles Are the Invisible Advantage | |
| We Are Entering the Intangible Economy | |
| More Intangibles Are Converted to Tangibles | |
| Intangibles Drive Programs | |
| Measurement and Analysis of Intangibles | p. 310 |
| Measuring the Intangibles | |
| Converting to Money | |
| Identifying Intangibles | |
| Analyzing Intangibles | |
| Customer Service | p. 316 |
| Team Effectiveness | p. 319 |
| Cooperation/Conflict | |
| Decisiveness/Decision Making | |
| Communication | |
| Innovation and Creativity | p. 320 |
| Innovation | |
| Creativity | |
| Employee Attitudes | p. 324 |
| Employee Satisfaction | |
| Organizational Commitment | |
| Employee Engagement | |
| Employee Capability | p. 326 |
| Experience | |
| Knowledge | |
| Learning | |
| Competencies | |
| Educational Level | |
| Attention | |
| Leadership | p. 333 |
| 360-Degree Feedback | |
| Leadership Inventories | |
| Leadership Perception | |
| Job Creation and Acquisition | p. 335 |
| Productivity Versus Job Growth | |
| Importance of Job Creation and Growth | |
| Recruitment Sourcing and Effectiveness | |
| Recruitment Efficiency | |
| Stress | p. 339 |
| Networking | p. 343 |
| Final Thoughts | p. 345 |
| Reporting Results | p. 348 |
| Why the Concern About Communicating Results? | p. 348 |
| Communication Is Necessary to Make Improvements | |
| Communication Is Necessary to Explain Contributions | |
| Communication Is a Politically Sensitive Issue | |
| Different Audiences Need Different Information | |
| Principles of Communicating Results | p. 350 |
| Communication Must Be Timely | |
| Communication Should Be Targeted to Specific Audiences | |
| Media Should Be Carefully Selected | |
| Communication Should Be Unbiased and Modest | |
| Communication Must Be Consistent | |
| Testimonials Are More Effective Coming from Respected Individuals | |
| The Audience's Opinion of the Program Will Influence the Communication Strategy | |
| The Process for Communicating Results | p. 352 |
| The Need for Communication | p. 354 |
| Planning the Communications | p. 355 |
| The Audience for Communications | p. 356 |
| Basis for Selecting the Audience | |
| Information Development: The Impact Study | p. 359 |
| Communication Media Selection | p. 361 |
| Meetings | |
| Interim and Progress Reports | |
| Routine Communication Tools | |
| E-Mail and Electronic Media | |
| Program Brochures and Pamphlets | |
| Case Studies | |
| Presenting Information | p. 364 |
| Routine Feedback on Program Progress | |
| The Presentation of Results to Senior Management | |
| Streamlining the Communication | |
| Building Scorecards | |
| Reactions to Communication | p. 372 |
| Using Evaluation Data | p. 372 |
| Final Thoughts | p. 373 |
| Implementing and Sustaining a Comprehensive Evaluation System | p. 375 |
| Why the Concern About Implementing and Sustaining Evaluation? | p. 375 |
| Resistance Is Always Present | |
| Implementation Is Key | |
| Consistency Is Needed | |
| Efficiency Is Necessary | |
| Implementing the Process: Overcoming Resistance | p. 377 |
| Assessing the Climate | p. 378 |
| Developing Roles and Responsibilities | p. 379 |
| Identifying a Champion | |
| Developing the Evaluation Leader | |
| Establishing a Task Force | |
| Assigning Responsibilities | |
| Establishing Goals and Plans | p. 382 |
| Setting Evaluation Targets | |
| Developing a Timetable for Implementation | |
| Revising or Developing Policies and Guidelines | p. 383 |
| Preparing the L&D Team | p. 384 |
| Involving the L&D Team | |
| Using Measurement and Evaluation as a Learning Tool | |
| Teaching the L&D Team | |
| Initiating Evaluation Studies | p. 385 |
| Selecting the Initial Program | |
| Developing the Planning Documents | |
| Reporting Progress | |
| Establishing Discussion Groups | |
| Preparing the Sponsors and Management Team | p. 387 |
| Removing Obstacles | p. 388 |
| Dispelling Myths | |
| Delivering Bad News | |
| Monitoring Progress | p. 390 |
| Final Thoughts | p. 390 |
| How Results-Based Are Your Workplace | p. 391 |
| Learning and Performance Programs? An Assessment for the L&D Staff | p. 391 |
| Glossary | p. 403 |
| Index | p. 407 |
| About the Authors | p. 423 |
| Table of Contents provided by Ingram. All Rights Reserved. |
What is included with this book?
The New copy of this book will include any supplemental materials advertised. Please check the title of the book to determine if it should include any access cards, study guides, lab manuals, CDs, etc.
The Used, Rental and eBook copies of this book are not guaranteed to include any supplemental materials. Typically, only the book itself is included. This is true even if the title states it includes any access cards, study guides, lab manuals, CDs, etc.
Please wait while the item is added to your bag...
