VMware Software-Defined Storage A Design Guide to the Policy-Driven, Software-Defined Storage Era

VMware Software-Defined Storage, A Guide to the Policy Driven, Software-Defined Storage Era presents the most in-depth look at VMware's next-generation storage technology to help solutions architects and operational teams maximize quality storage design. Written by a double VMware Certified Design Expert, this book delves into the design factors and capabilities of Virtual SAN and Virtual Volumes to provide a uniquely detailed examination of the software-defined storage model. Storage-as-a-Service (STaaS) is discussed in terms of deployment through VMware technology, with insight into the provisioning of storage resources and operational management, while legacy storage and storage protocol concepts provide context and demonstrate how Virtual SAN and Virtual Volumes are meeting traditional challenges. The discussion on architecture emphasizes the economies of storage alongside specific design factors for next-generation VMware based storage solutions, and is followed by an example in which a solution is created based on the preferred option identified from a selection of cross-site design options.

Storage hardware lifecycle management is an ongoing challenge for IT organizations and service providers. VMware is addressing these challenges through the software-defined storage model and Virtual SAN and Virtual Volumes technologies; this book provides unprecedented detail and expert guidance on the future of storage.

• Understand the architectural design factors of VMware-based storage
• Learn best practices for Virtual SAN stretched architecture implementation
• Deploy STaaS through vRealize Automation and vRealize Orchestrator
• Meet traditional storage challenges with next-generation storage technology

Virtual SAN and Virtual Volumes are leading the way in efficiency, automation, and simplification, while maintaining enterprise-class features and performance. As organizations around the world are looking to cut costs without sacrificing performance, availability, or scalability, VMware-based next-generation storage solutions are the ideal platform for tomorrow's virtual infrastructure. VMware Software-Defined Storage provides detailed, practical guidance on the model that is set to transform all aspects of vSphere data center storage.

Martin Hosken, VCDX is a Global Cloud Architect and part of the Cloud Provider Software Business Unit at VMWare. One of only a small number of double VMWare Certified Design Experts in the world, he has been awarded the annual VMWare vExpert title for the past four years. Martin specializes in the design, implementation, and integration of Cisco, EMC, IBM, HP, Dell, and VMWare systems into enterprise environments and cloud service provider infrastructure.

Introduction xix

Chapter 1 • Software-Defined Storage Design 1

Software-Defined Compute 2

Software-Defined Networking 2

Software-Defined Storage 3

Designing VMware Storage Environments 4

Technical Assessment and Requirements Gathering 5

Establishing Storage Design Factors 6

The Economics of Storage 10

Calculating the Total Cost of Ownership for Storage Resources 11

Information Lifecycle Management 13

Implementing a Software-Defined Storage Strategy 15

Software-Defined Storage Summary 16

Hyper-Converged Infrastructure and Virtual SAN 18

Virtual Volumes 18

Classic and Next-Generation Storage Models 19

Chapter 2 • Classic Storage Models and Constructs 21

Classic Storage Concepts 21

RAID Sets 25

Virtual Provisioning 44

Storage Tiering 49

Storage Scalability Design 54

Storage Management Tools 57

Multitenanted Storage Design 58

Quality of Service 59

Data Deduplication and Data Compression 60

Storage Device Security 61

Hardware High Availability 61

Storage Array–Based Disaster Recovery and Backups 62

Storage Array Snapshots and Clones in a Classic Storage Environment 63

vSphere Metro Storage Cluster 65

All-Flash Disk Arrays 65

vSphere Storage Technologies 67

Virtual Disks 68

Virtual Machine Storage Controllers (vSCSI Adapters) 71

Datastore 73

Raw Device Mapping 79

When to Use RDMs over VMFS or NFS? 81

Storage vMotion and Enhanced vMotion Operations 81

Datastore Clusters 82

Storage Distributed Resource Scheduler 83

Storage I/O Control 85

Classic Storage Model—vStorage APIs for Array Integration 89

Classic Storage Model—VASA 1.0 90

VADP and VAMP 91

Boot from SAN 92

Classic Storage Model—vSphere Storage Policies 94

Tiered Storage Design Models in vSphere 95

Sub-LUN System Access 98

Chapter 3 • Fabric Connectivity and Storage I/O Architecture 101

Fibre Channel SAN 102

Fibre Channel Protocol 102

Fibre Channel Topologies 115

Switch-Based Fabric Architecture 117

Security and Traffic-Isolation Features 125

N_Port Virtualization and N_Port ID Virtualization 131

Boot from SAN 132

Fibre Channel Summary 132

iSCSI Storage Transport Protocol 135

iSCSI Protocol Components 135

iSCSI Traffic Isolation 137

Jumbo Frames 138

iSCSI Device-Naming Standards 138

CHAP Security 139

iSCSI Network Adapters 140

Virtual Switch Design 143

iSCSI Boot from SAN 148

iSCSI Protocol Summary 148

NFS Storage Transport Protocol 149

Comparing NAS and SAN 149

NFS Components 149

NAS Implementation 152

Single Virtual Switch / Single Network Design 157

Single Virtual Switch / Multiple Network Design 159

vSphere 6 NFS Version 4.1 Limitations 161

NFS Protocol Summary 161

Fibre Channel over Ethernet Protocol 161

Fibre Channel over Ethernet Protocol 163

Fibre Channel over Ethernet Physical Components 165

Fibre Channel over Ethernet Infrastructure 167

Fibre Channel over Ethernet Design Options 167

Fibre Channel over Ethernet Protocol Summary 170

Multipathing Module 170

Pluggable Storage Architecture 174

iSCSI Multipathing 177

NAS Multipathing 178

Direct-Attached Storage 180

Evaluating Switch Design Characteristics 182

Fabric Connectivity and Storage I/O Architecture Summary 184

Chapter 4 • Policy-Driven Storage Design with Virtual SAN 187

Challenges with Legacy Storage 187

Policy-Driven Storage Overview 190

VMware Object Storage Overview 191

Virtual SAN Overview 192

Virtual SAN Architecture 194

Virtual SAN Disk Groups 194

Comparing Virtual SAN Hybrid and All-Flash Models 200

All-Flash Deduplication and Compression 202

Data Locality and Caching Algorithms 205

Virtual SAN Destaging Mechanism 206

Virtual SAN Distributed Datastore 206

Objects, Components, and Witnesses 207

On-Disk Formats 212

Swap Efficiency / Sparse Swap 214

Software Checksum 215

Virtual SAN Design Requirements 216

Host Form Factor 216

Host Boot Architecture 217

Virtual SAN Hardware Requirements 222

Virtual SAN Network Fabric Design 236

vSphere Network Requirements 236

Physical Network Requirements 240

Virtual SAN Storage Policy Design250

Storage Policy–Based Management Framework 250

Virtual SAN Rules 251

Virtual SAN Rule Sets 253

Default Storage Policy 267

Application Assessment and Storage-Policy Design 268

Virtual SAN Datastore Design and Sizing 271

Hosts per Cluster 273

Storage Capabilities 275

Configuring Multiple Disk Groups 276

Endurance Flash Sizing 278

Objects, Components, and Witness Sizing 279

Datastore Capacity Disk Sizing 281

Capacity Disk Size 282

Designing for Availability 287

Designing for Hardware Component Failure 289

Host Cluster Design and Planning for Host Failure 292

Quorum Logic Design and vSphere High Availability 302

Fault Domains 302

Virtual SAN Internal Component Technologies 308

Reliable Datagram Transport 308

Cluster Monitoring, Membership, and Directory Services 308

Cluster-Level Object Manager 310

Distributed Object Manager 310

Local Log-Structured Object Manager 310

Object Storage File System 311

Storage Policy–Based Management 312

Virtual SAN Integration and Interoperability 312

Chapter 5 • Virtual SAN Stretched Cluster Design 315

Stretched Cluster Use Cases 317

Fault Domain Architecture 318

Witness Appliance 318

Network Design Requirements 320

Distance and Latency Considerations 322

Bandwidth Requirements Calculations 325

Stretched Cluster Deployment Scenarios 327

Default Gateway and Static Routes 327

Stretched Cluster Storage Policy Design 327

Preferred and Nonpreferred Site Concepts 329

Stretched Cluster Read/Write Locality 329

Distributed Resource Scheduler Configurations 332

High Availability Configuration 335

Stretched Cluster WAN Interconnect Design 339

Evaluating WAN Platforms for Stretched Clusters 339

Deploying Stretched VLANs 347

WAN Interconnect High Availability 353

Secure Communication 353

Data Center Interconnect Design Considerations Summary 354

Stretched Cluster Solution Architecture Example 356

Cisco vPC over DWDM and Dark Fiber 358

OTV over DWDM and Dark Fiber 360

Cisco LISP Configuration Overview 363

Stretched Cluster Failure Scenarios 363

Stretched Cluster Interoperability 365

Support Limitations 365

Chapter 6 • Designing for Web-Scale Virtual SAN Platforms 367

Scale-up Architecture 368

Scale-out Architecture 370

Designing vSphere Host Clusters for Web-Scale 372

Building-Block Clusters and Scale-out Web-Scale Architecture 372

Scalability and Designing Physical Resources for Web-Scale 373

Leaf-Spine Web-Scale Architecture 377

Chapter 7 • Virtual SAN Use Case Library 381

Use Cases Overview 383

Two-Node Remote Office / Branch Office Design 386

Horizon and Virtual Desktop Infrastructure 392

Virtual SAN File Services 395

Solution Architecture Example: Building a Cloud Management Platform with Virtual SAN 395

Introduction and Conceptual Design 395

Customer Design Requirements and Constraints 398

Cluster Configuration 404

Network-Layer Design 408

Storage-Layer Design 412

Cloud Management Platform Security Design 423

Chapter 8 • Policy-Driven Storage Design with Virtual Volumes 429

Introduction to Virtual Volumes Technology 430

Virtual Volumes Component Technology Architecture 434

Virtual Volumes Object Architecture 434

Management Plane 436

VASA 2.0 Specification 436

VASA Provider 436

Data Plane 437

Storage Container 437

Protocol Endpoints 440

Binding Operations 442

Storage Policy–Based Management with Virtual Volumes 444

Published Capabilities 446

Storage Capabilities 448

Storage Capabilities Summary 449

Benefits of Designing for Virtual Volumes 449

Enhanced Performance 450

Greater Application Control 450

Operational Simplification 450

Reduced Wasted Capacity 450

Virtual Volumes Key Design Requirements 450

vSphere Storage Feature Interoperability 451

VAAI and Virtual Volumes 451

Virtual Volumes Summary 451

Chapter 9 • Delivering a Storage-as-a-Service Design 453

STaaS Service Definition 457

Cloud Platforms Overview 458

Cloud Management Platform Architectural Overview 461

vRealize Automation Cloud Management Platform 461

vRealize Orchestrator 465

The Combined Solution Stack 468

Workflow Examples 468

Summary 472

Chapter 10 • Monitoring and Storage Operations Design 473

Storage Monitoring 473

Monitoring Component Health 474

Monitoring Capacity 474

Monitoring Storage Performance 475

Monitoring Security 476

Storage Component Monitoring 477

Monitoring Storage on Host Servers 477

Monitoring the Storage Fabric 477

Monitoring a Storage Array System 480

Storage Monitoring Challenges 481

Common Storage Management and Monitoring Standards 483

Virtual SAN Monitoring and Operational Tools 486

vRealize Operations Manager 492

Management Pack for Storage Devices 492

Storage Partner Solutions 494

vRealize Log Insight 497

Log Insight Syslog Design 498

End-to-End Monitoring Solution Summary 499

Storage Capacity Management and Planning 499

Management Strategy Design 502

Process and Approach 503

Capacity Management for Virtual SAN 505

Summary 505

Index 509

The New copy of this book will include any supplemental materials advertised. Please check the title of the book to determine if it should include any access cards, study guides, lab manuals, CDs, etc.

The Used, Rental and eBook copies of this book are not guaranteed to include any supplemental materials. Typically, only the book itself is included. This is true even if the title states it includes any access cards, study guides, lab manuals, CDs, etc.