Service Availability Principles and Practice

Maria Toeroe, Ericsson, Canada
Maria Toeroe has been in the telecommunications industry for over 10 years conducting research and standardisation in the area of dependable software, service availability and fault tolerance utilizing the expertise she gained in her prior work in the academia related to formal description techniques and their use in protocol engineering. Having represented Ericsson in the Service Availability Forum since 2005, Maria has led several standardization efforts in different chair positions of the SA Forum Technical Work Group.? Within Ericsson she is also responsible for university collaborations related to her fields. She has been an Adjunct Associate Professor with the Department of Electrical and Computer Engineering of Concordia University since 2007. Maria holds a Diploma degree from the Institute of Engineering Economics Kharkov, a University Doctorate and a PhD from the Faculty of Electrical Engineering and Informatics of the Budapest University of Technology and Economics.

Francis Tam, Nokia, Finland
Francis Tam has conducted research, development and standardisation in the dependable software area of service availability and fault tolerance in the telecommunications industry for more than 16 years. He has served as industrial adviser to three European Commission funded projects on dependable computer systems and frequently gives lectures on highly available systems to the Master of Engineering students at INSA Lyon in France. Francis holds a BSc in Computer Science from Queen Mary College (University of London), an MSc in Computing Science from North Staffordshire Polytechnic and a Doctor of Science (Technology) in Software Systems from Tampere University of Technology in Finland.

Foreword xv

Preface xix

Acknowledgments xxv

List of Abbreviations xxvii

Part I INTRODUCTION TO SERVICE AVAILABILITY

1 Definitions, Concepts, and Principles 3
Francis Tam

1.1 Introduction 3

1.2 Why Service Availability? 4

1.2.1 Dossier on Unavailability of Service 4

1.2.2 Issues and Challenges 5

1.3 Service Availability Fundamentals 6

1.3.1 System, Behavior, and Service 6

1.3.2 Dependable Computing Concepts 8

1.3.3 The Meaning of Availability 10

1.4 Achieving Service Availability 13

1.4.1 Following the Framework of Fault Tolerance 13

1.4.2 Redundancy is a Requisite 14

1.4.3 Dealing with Failures 16

1.4.4 Upgrade Matters 19

1.5 Conclusion 20

2 The Birth of the Service Availability Forum 23
Francis Tam

2.1 Introduction 23

2.2 Technology Environment 23

2.3 Business Environment 24

2.3.1 Ecosystem 25

2.3.2 COTS and Open Systems 26

2.4 The Service Availability Forum Era 27

2.5 Concluding Remarks 28

Part II THE SA FORUM SYSTEM: SERVICES AND FRAMEWORKS

3 Overview of the Service Availability Architecture 33
Dave Penkler

3.1 Introduction 33

3.1.1 Background and Business Context 33

3.1.2 Goals and Requirements 34

3.1.3 Service Availability Architecture Scope and Presentation 36

3.2 HA Concepts Applied 39

3.2.1 To Be or Not to Be High Availability Aware 39

3.2.2 HA Aware Application Perspective 42

3.3 Architecture 43

3.3.1 Basic Architectural Model 43

3.3.2 The AIS Services and Frameworks Architecture 47

3.3.3 Service Dependencies 58

3.4 Open Issues 59

3.4.1 The Optional Features Issue 60

3.4.2 Integrated AIS Service API 60

3.4.3 Common Low Level Communication Facility Interface 60

3.4.4 Common Distributed Process Management Interface 61

3.4.5 System Trace Service 61

3.4.6 Diagnostics Framework 61

3.4.7 Overload Control Framework 61

3.5 Conclusion 62

4 The SA Forum Information Model: The Heart of Control and Monitoring 63
Maria Toeroe

4.1 Introduction 63

4.2 Background 64

4.2.1 Management Models Out There 64

4.2.2 The SA Forum Needs 65

4.3 The SA Forum Information Model 67

4.3.1 Overview of the SA Forum Solution 67

4.3.2 Administrative and Management Aspects 80

4.3.3 Application Information Models 81

4.3.4 Open Issues and Recommendations 81

4.4 Conclusion 83

5 Consistent and High Level Platform View 85
Maria Toeroe

5.1 Introduction 85

5.2 Hardware Platform Interface 86

5.2.1 Background 86

5.2.2 Overview of the Hardware Platform Interface 87

5.2.3 The HPI Model 88

5.2.4 HPI Capability Discovery 93

5.2.5 Error Handling and Administrative Operations 94

5.2.6 Open Issues and Conclusions 95

5.3 Platform Management Service 96

5.3.1 The Conception of PLM 96

5.3.2 Overview of the SA Forum Platform Management 97

5.3.3 The PLM Information Model 98

5.3.4 Tracking of PLM Entities 107

5.3.5 Administrative and Management Aspects 110

5.3.6 Service Interaction 118

5.3.7 Open Issues and Conclusions 120

5.4 Cluster Membership Service 121

5.4.1 Background 121

5.4.2 Overview of the Cluster Membership Service 122

5.4.3 CLM Configuration: The Bootstrap Trap 125

5.4.4 Are You a Member? 126

5.4.5 Administrative and Management Aspects 127

5.4.6 Service Interaction 129

5.4.7 Open Issues 130

5.4.8 Recommendation 131

5.5 Conclusion 131

6 Model Based Availability Management: The Availability Management Framework 133
Maria Toeroe

6.1 Introduction 133

6.2 Background 134

6.2.1 Error Detection and Repair 134

6.2.2 Fault Zones and Error Escalation 135

6.2.3 Separation of Services from Serving Entities 136

6.2.4 Service Provisioning Roles 136

6.2.5 Delicacies of Service State Replication 137

6.3 The Availability Management Framework 138

6.3.1 Overview of the SA Forum Solution 138

6.3.2 Components and Component Service Instances 139

6.3.3 The AMF Information Model 148

6.3.4 Redundancy Models 167

6.3.5 The AMF Administrative Interface 176

6.3.6 Interactions Between AMF and Other AIS Services 187

6.3.7 Open Issues 190

6.3.8 Recommendation 191

6.4 Conclusion 191

7 Communication and Synchronization Utilities 193
Maria Toeroe and Sayandeb Saha

7.1 Introduction 193

7.2 Event Service 194

7.2.1 Background: Event Service Issues, Controversies, and Problems 194

7.2.2 Overview of the SA Forum Event Service 195

7.2.3 Event Service Architecture and Model 196

7.2.4 User Perspective 200

7.2.5 Administrative and Management Aspects 201

7.2.6 Service Interactions 201

7.2.7 Open Issues and Recommendations 202

7.3 Message Service 202

7.3.1 Need for Reliability and Load Distribution 202

7.3.2 Overview of the SA Forum Message Service 203

7.3.3 Message Service Architecture and Model 205

7.3.4 User Perspective 207

7.3.5 Administrative and Management Aspects 210

7.3.6 Service Interaction 210

7.3.7 Open Issues and Recommendations 211

7.4 Checkpoint Service 212

7.4.1 Background: Why Checkpoints 212

7.4.2 Overview of the SA Forum Checkpoint Service 213

7.4.3 Checkpoint Service Model 215

7.4.4 User Perspective 217

7.4.5 Administrative and Management Aspects 220

7.4.6 Service Interaction 221

7.4.7 Open Issues 222

7.4.8 Recommendation 222

7.5 Conclusion 223

7.5.1 Common Issue: Entity Names 223

7.5.2 Conclusion 223

8 Services Needed for System Management 227
Maria Toeroe

8.1 Introduction 227

8.2 Log Service 228

8.2.1 Background: Data, Data, and More Data 228

8.2.2 Overview of the SA Forum Solution 229

8.2.3 The LOG Information Model 231

8.2.4 User Perspective 232

8.2.5 Administrative and Management Aspects 233

8.2.6 Service Interaction 233

8.2.7 Open Issues and Recommendations 235

8.3 Notification Service 236

8.3.1 Background: Issues, Controversies, and Problems 236

8.3.2 Overview of the SA Forum Notification Service 237

8.3.3 User Perspective 239

8.3.4 Correlation of Notifications 241

8.3.5 Administrative and Management Aspects 243

8.3.6 Service Interaction 244

8.3.7 Open Issues and Recommendation 246

8.4 Information Model Management Service 247

8.4.1 Background: Issues, Controversies, and Problems 247

8.4.2 Overview of the SA Forum IMM Solution 249

8.4.3 The Object Manager API 251

8.4.4 The Object Implementer API 255

8.4.5 IMM XML File 258

8.4.6 Administrative and Management Aspects 258

8.4.7 Service Interaction 258

8.4.8 Open Issues 260

8.4.9 Recommendation 261

8.5 Conclusion 262

9 Model-Based Software Management: The Software Management Framework 265
Maria Toeroe

9.1 Introduction 265

9.2 Background 266

9.3 Software Management a la Carte 268

9.3.1 Overview of the SA Forum Solution 268

9.3.2 Entity Types File: Is It Eaten or Drunk by SMF? 271

9.3.3 The Upgrade Campaign and Its Specification 273

9.3.4 Upgrade Campaign Execution Status and Failure Handling 279

9.3.5 Administrative and Management Aspects 285

9.3.6 User Perspective 288

9.3.7 Service Interaction 289

9.3.8 Open Issues 291

9.3.9 Recommendation 292

9.4 Conclusion 294

10 Combining the Services 297
Maria Toeroe

10.1 Introduction 297

10.2 Application Design and Development 297

10.3 Application Platform Design 299

10.4 Operation and Maintenance 301

Part III SA FORUM MIDDLEWARE IN ACTION

11 SA Forum Programming Model and API Conventions 305
Francis Tam

11.1 Introduction 305

11.2 Programming Model 306

11.2.1 AIS Area Service Interfaces 306

11.2.2 Real-Time Support 306

11.2.3 Naming Conventions and Type Definitions 308

11.2.4 Usage Model and Library Life Cycle 309

11.2.5 Tracking 311

11.3 Making Sense of the API Specifications 312

11.3.1 Structure of Service API Specification 314

11.3.2 Administration API 315

11.4 Practical Topics 316

11.4.1 Interacting with POSIX 316

11.4.2 Allocating and Freeing Memory 319

11.4.3 Handling Pointers 319

11.4.4 Finding Out Implementation Limits 320

11.4.5 When an Area Service is Unavailable 321

11.4.6 Backward Compatibility 322

11.5 Concluding Remarks 322

12 SA Forum Java Mappings: Specifications, Usage, and Experience 325
Robert Hyerle and Jens Jensen

12.1 Introduction 325

12.2 Background 325

12.2.1 Early Exploration of Java Mappings in Hewlett–Packard 325

12.2.2 Java in Ericsson 326

12.2.3 The SA Forum Java Mapping Initiative 327

12.3 Understanding the Java Mappings 328

12.3.1 Java Application Integration Architecture 328

12.3.2 Naming 329

12.3.3 Package Structure 330

12.3.4 The Underlying Objects 330

12.3.5 Types 331

12.3.6 Parameters, Exceptions, and Method Signatures 332

12.3.7 Factories, Callbacks, and Life-cycles 333

12.3.8 Callbacks and the Selection Object in Java 334

12.4 Using the Java Mappings 335

12.4.1 Integrating AIS Services with Java Applications 335

12.4.2 Integrating AIS Services with Containerized Java Applications 342

12.4.3 AIS Services in Mixed Language and Mixed Implementation Environments 343

12.5 Going Further 343

12.5.1 The Java Mapping Roadmap 343

12.5.2 Related Java Standards and Other References 344

13 SA Forum Middleware Implementations 347
Mario Angelic and Ulrich Kleber

13.1 Introduction 347

13.1.1 OpenHPI 347

13.1.2 OpenSAF 348

13.2 The OpenHPI Project 348

13.2.1 Overview of the OpenHPI Solution 348

13.2.2 User Perspective 351

13.2.3 OpenHPI Tools 353

13.2.4 Open Issues and Recommendations 354

13.3 The OpenSAF Project 355

13.3.1 Background 355

13.3.2 OpenSAF Architecture 356

13.3.3 SA Forum Compliant Services 360

13.3.4 OpenSAF Infrastructure Services 364

13.3.5 Managing OpenSAF 365

13.3.6 Deploying OpenSAF 367

13.4 Conclusion 368

14 Integration of the VideoLAN Client with OpenSAF: An Example 371
Anik Mishra and Ali Kanso

14.1 Introduction 371

14.2 Going Under the Hood: The VLC Workflow 372

14.3 Integrating VLC with OpenSAF 373

14.3.1 Nonproxied-Non-SA-Aware Integration 374

14.3.2 SA-Aware VLC Integration 379

14.3.3 SA-Aware VLC with Service Continuity 384

14.4 Summary and Conclusion 387

15 Migration Paths for Legacy Applications 391
Mario Angelic

15.1 Introduction 391

15.2 Reasons for Migration 392

15.2.1 Benefits for System Owners 392

15.2.2 Benefits for ISVs 392

15.3 Integration Criteria 393

15.3.1 Main Factors 393

15.3.2 Easy Management 394

15.3.3 Streamlined Architecture 396

15.3.4 Code Quality 397

15.3.5 Integration Levels 397

15.4 How to Migrate 399

15.4.1 Availability Integration 399

15.4.2 Manageability Integration 409

15.5 Open Issues 413

15.6 Conclusion 413

16 Overcoming Complexity: Formal Modeling Techniques at the Rescue 415
Maria Toeroe and Ferhat Khendek

16.1 Introduction 415

16.2 Background 416

16.2.1 The Model-Based Approach 416

16.2.2 Starting Points in the Specifications 417

16.3 Model-Based Software Management 419

16.3.1 Configuration Model 419

16.3.2 Configuration Generation 420

16.3.3 Upgrade Campaign Generation 424

16.3.4 Analytical Models and How They Can Help 427

16.4 Conclusion 428

17 Conclusion 431

17.1 Summary 431

17.2 The Future 433

References 435

Index 443

The New copy of this book will include any supplemental materials advertised. Please check the title of the book to determine if it should include any access cards, study guides, lab manuals, CDs, etc.

The Used, Rental and eBook copies of this book are not guaranteed to include any supplemental materials. Typically, only the book itself is included. This is true even if the title states it includes any access cards, study guides, lab manuals, CDs, etc.