Intelligent Signal Processing

Note: Supplemental materials are not guaranteed with Rental or Used book purchases.

ISBN: 9780780360105 | 0780360109
Cover: Hardcover
Copyright: 1/15/2001

Buy New

Print on Demand: 2-4 Weeks. This item cannot be cancelled or returned.

$270.01

"IEEE Press is proud to present the first selected reprint volume devoted to the new field of intelligent signal processing (ISP). ISP differs fundamentally from the classical approach to statistical signal processing in that the input-output behavior of a complex system is modeled by using intelligent or model-free techniques, rather than relying on the shortcomings of a mathematical model. Information is extracted from incoming signal and noise data, making few assumptions about the statistical structure of signals and their environment. Intelligent Signal Processing explores how ISP tools address the problems of practical neural systems, new signal data, and blind fuzzy approximators. The editors have compiled 20 articles written by prominent researchers covering 15 diverse, practical applications of this nascent topic, exposing the reader to the signal processing power of learning and adaptive systems. This essential reference is intended for researchers, professional engineers, and scientists working in statistical signal processing and its applications in various fields such as humanistic intelligence, stochastic resonance, financial markets, optimization, pattern recognition, signal detection, speech processing, and sensor fusion. Intelligent Signal Processing is also invaluable for graduate students and academics with a background in computer science, computer engineering, or electrical engineering. About the Editors Simon Haykin is the founding director of the Communications Research Laboratory at McMaster University, Hamilton, Ontario, Canada, where he serves as university professor. His research interests include nonlinear dynamics, neural networks and adaptive filters and their applications in radar and communications systems. Dr. Haykin is the editor for a series of books on Adaptive and Learning Systems for Signal Processing, Communications and Control (Publisher) and is both an IEEE Fellow and Fellow of the Royal Society of Canada. Bart Kosko is a past director of the University of Southern Californias (USC) Signal and Image Processing Institute. He has authored several books, including Neural Networks and Fuzzy Systems, Neural Networks for Signal Processing (Publisher, copyright date) and Fuzzy Thinking (Publisher, copyright date), as well as the novel Nanotime (Publisher, copyright date). Dr. Kosko is an elected governor of the International Neural Network Society and has chaired many neural and fuzzy system conferences. Currently, he is associate professor of electrical engineering at USC."

Preface

xvii

List of Contributors

xix

Humanistic Intelligence: ``Wear Comp'' As a New Framework and Application for Intelligent Signal Processing

(39)

Introduction

(2)

Why Humanistic Intelligence?

(1)

Humanistic Intelligence Does Not Necessarily Mean ``user-friendly''

(1)

``Wear Comp'' As Means of Realizing Humanistic Intelligence

(3)

Basic Principals of WearComp

(1)

Operational Modes of WearComp

(2)

The Six Basic Signal Flow Paths of WearComp

(1)

Philosophical Issues

(4)

Fundamental Issues of WearComp

(1)

Historical context

(1)

The shift from guns to cameras and computers

(1)

The shift from draconian punishment to micro management

(1)

Fundamental issues of WearComp

(1)

Aspects of WearComp and Personal Empowerment

(1)

Practical Embodiment of Wearcomp

(6)

Building Signal-Processing Devices Directly into Fabric

(1)

Remaining issue of underwearable signal processing hardware

(1)

Multidimensional Signal Input for Humanistic Intelligence

(1)

Safety first

(1)

More than just a health status monitor

(1)

Personal Imaging Application of Humanistic Intelligence

(18)

Some Simple Illustrative Examples

(1)

Always ready: From point and click to ``look and think''

(1)

Personal safety device for reducing crime

(1)

The retro-autofocus example: Human in the signal processing loop

(1)

Mathematical Framework for Personal Imaging

(1)

Quantigraphic imaging and the Wyckoff Principle

(2)

Video orbits

(4)

Dynamic range and ``dynamic domain''

(6)

Bi-foveated WearCam

(1)

Lightspace modeling for HI

(3)

Beyond Video: Synthetic Synesthesia and Personal Imaging

(2)

Synthetic Synesthesia: Adding New Sensory Capabilities to the Body

(1)

Safety First

(1)

A true extension of the mind and body

(1)

Conclusions

(1)

Acknowledgments

(1)

References

(3)

Adaptive Stochastic Resonance

(68)

Abstract

(1)

Stochastic Resonance and Adaptive Function Approximation

(5)

Sr Dynamical Systems

(5)

Sr Performance Measures

(5)

Additive Fuzzy Systems and Function Approximation

(2)

Sr Learning and Equilibrium

(10)

The Signal-to-Noise Ratio in Nonlinear Systems

(5)

Supervised Gradient Learning and SR Optimality

(5)

Sr Learning: Simulation Results

(18)

SR Test Case: The Quartic Bistable System

(7)

Other SR Test Cases

(4)

Fuzzy Sr Learning: The Quartic Bistable System

(5)

Conclusions

(3)

References

(15)

The Standard Additive Model (SAM) Theorem

104

(2)

Sam Gradient Learning

106

(2)

Learning in the Presence of Noise

108

(19)

Abstract

108

(1)

Introduction

108

(2)

Financial Time Series Prediction

110

(1)

Impact of Noise on Learning

111

(5)

The Learning Problem

111

(2)

Performance of a Learning System

113

(3)

Estimating the Model Limitation

116

(1)

Application to Financial Market Forecasting

116

(1)

Conclusion

117

(1)

Acknowledgments

118

(1)

Appendix

119

(6)

References

125

(2)

Incorporating Prior Information in Machine Learning by Creating Virtual Examples

127

(36)

Abstract

127

(1)

Learning From Examples

128

(1)

Background: Learning as Function Approximation

128

(1)

Prior Information and the Problem of Sample Complexity

129

(4)

Virtual Examples: A Framework for Prior Information

133

(5)

The General Framework

134

(1)

Techniques for Prior Information and Related Research

135

(1)

Prior Knowledge in the Choice of Variables of Features

135

(1)

Prior Knowledge in the Learning Technique

135

(1)

Generating New Examples with Prior Knowledge

136

(1)

Incorporating Prior Knowledge as Hints

137

(1)

Virtual Examples and Regularization

138

(5)

Regularization Theory and RBF

138

(2)

Regularization Theory in Presence of Radial Symmetry

140

(1)

Radial Symmetry and ``Virtual'' Examples

141

(2)

Virtual Examples in Vision and Speech

143

(15)

Virtual Views for Object Recognition

144

(2)

Symmetry as Prior Information

146

(1)

More General Transformations: Linear Object Classes

147

(1)

3D Objects, 2D Projections, and Linear Classes

148

(1)

Implications

149

(1)

Learning the Transformation

150

(2)

Virtual Examples in Speech Recognition

152

(6)

Conclusions

158

(1)

References

158

(5)

Deterministic Annealing for Clustering, Compression, Classification, Regression, and Speech Recognition

163

(67)

Abstract

163

(1)

Introduction

163

(3)

Deterministic Annealing for Unsupervised Learning

166

(22)

Clustering

166

(1)

Principled Derivation of Deterministic Annealing

167

(2)

Statistical Physics Analogy

169

(3)

Mass-Constrained Clustering

172

(5)

Preferred Implementation of the DA Clustering Algorithm

177

(1)

Illustrative Examples

178

(2)

Extensions and Applications

180

(1)

Vector Quantization for Noisy Channels

180

(2)

Entropy-Constrained Vector Quantizer Design

182

(1)

Structurally Constrained Vector Quantizer Design

183

(2)

Graph-Theoretic and Other Optimization Problems

185

(3)

Deterministic Annealing for Supervised Learning

188

(19)

Problem Formulation

188

(2)

Basic Derivation

190

(2)

Generality and Wide Applicability of the DA Solution

192

(1)

Regression, Classification, and Clustering

192

(1)

Structures

193

(4)

Experimental Results

197

(1)

VQ Classifier Design

197

(4)

RBF Classifier Design

201

(1)

MLP Classifier Design

201

(1)

Piecewise Regression

202

(4)

Mixture of Experts

206

(1)

Speech Recognition

207

(10)

Problem Formulation

212

(1)

Design by Deterministic Annealing

213

(2)

Experimental Results

215

(2)

The Rate-Distortion Connection

217

(2)

Recent DA Extensions

219

(1)

Summary

220

(1)

References

221

(9)

Local Dynamic Modeling with Self-Organizing Maps and Applications to Nonlinear System Identification and Control

230

(42)

Abstract

230

(1)

Introduction

230

(2)

Dynamic Modeling

232

(3)

Global and Local Models

235

(5)

Global Dynamic Models

236

(1)

Local Dynamic Models

237

(2)

State Dependent Prediction of Nonlinear AR Processes

239

(1)

Kohonen's Self-Organizing Map (SOM)

240

(5)

SOM Networks and Kohonen Learning

240

(3)

Codebook in Reconstruction Space

243

(2)

SOM-Based Local Linear Models

245

(6)

Dynamic Learning in the SOM

248

(3)

Experimental Results

251

(4)

Design of Switching Controllers Based on SOM

255

(5)

Problem Specification

255

(1)

The Control Strategy

255

(3)

Design of the PMMSC

258

(2)

Experimental Control of the Wind Tunnel

260

(4)

Comparison of PPMSC Control to Existing Automatic Controller and Expert Operator

260

(4)

Conclusions

264

(3)

References

267

(3)

Appendix

270

(2)

A Signal Processing Framework Based on Dynamic Neural Networks with Application to Problems in Adaptation, Filtering and Classification

272

(23)

Abstract

272

(1)

Introduction

272

(1)

Network Architecture and Execution

273

(2)

Gradient Calculation

275

(1)

EKF Multi-Stream Training

275

(5)

The Kalman Recursion

275

(2)

Multi-Stream Training

277

(1)

Some Insight into the Multi-Stream Technique

278

(1)

Advantages and Extensions of Multi-Stream Training

279

(1)

Synthetic Examples

280

(5)

Multiple Series Prediction and Droupout Rejection

280

(1)

Training

280

(1)

Testing

280

(1)

Discussion

281

(1)

Modeling with Stability Enforcement

282

(1)

Training without Constraints

283

(1)

Training with Constraints

284

(1)

Discussion

284

(1)

Automotive Examples

285

(7)

Sensor-Catalyst Modeling

285

(2)

Experimental Data

287

(1)

Training and Testing

287

(2)

Engine Misfire Detection

289

(1)

Applying Recurrent Networks

290

(2)

Summary and Conclusions

292

(1)

Acknowledgments

292

(1)

References

292

(3)

Semiparametric Support Vector Machines for Nonlinear Model Estimation

295

(11)

Abstract

295

(1)

Introduction

295

(1)

Problem Setting

296

(1)

Optimum Line Search for a Generic Direction

297

(1)

Simple Algorithms for Purely Nonparametric Approach

298

(2)

A Gradient-Based Approach

299

(1)

A Coordinate Descent Approach

299

(1)

The Classic SVM Problem

300

(1)

General Semiparametric SVM

301

(1)

The Augmented Lagrangian Approach

301

(1)

Training Semiparametric SVM

301

(1)

A Possible Simplification

302

(1)

Regularized Parametric SVM Approach

302

(1)

Experimental Results

302

(3)

Conclusions

305

(1)

Acknowledgments

305

(1)

References

305

(1)

Gradient-Based Learning Applied to Document Recognition

306

(46)

Introduction

306

(4)

Learning from Data

307

(1)

Gradient-Based Learning

308

(1)

Gradient Back-Propogation

308

(1)

Learning in Real Handwriting Recognition Systems

309

(1)

Globally Trainable Systems

309

(1)

Convolutional Neural Networks for Isolated Character Recognition

310

(4)

Convolutional Networks

311

(1)

LeNet-5

312

(2)

Loss Function

314

(1)

Results and Comparison with Other Methods

314

(7)

Database: the Modified NIST Set

314

(1)

Results

315

(1)

Comparison with Other Classifiers

316

(1)

Linear Classifier, and Pairwise Linear Classifier

316

(1)

Baseline Nearest Neighbor Classifier

317

(1)

Principal Component Analysis (PCA) and Polynomial Classifier

317

(1)

Radial Basis Function Network

318

(1)

One-Hidden Layer Fully Connected Multilayer Neural Network

318

(1)

Two-Hidden Layer Fully Connected, Multilayer Neural Network

318

(1)

A Small Convolutional Network: LeNet-1

318

(1)

LeNet-4

318

(1)

Boosted LeNet-4

318

(1)

Tangent Distance Classifier (TDC)

319

(1)

Support Vector Machine (SVM)

319

(1)

Discussion

319

(1)

Invariance and Noise Resistance

320

(1)

Multi-Module Systems and Graph Transformer Networks

321

(3)

An Object-Oriented Approach

321

(1)

Special Modules

322

(1)

Graph Transformer Networks

323

(1)

Multiple Object Recognition: Heuristic Over-Segmentation

324

(2)

Segmentation Graph

324

(1)

Recognition Transformer and Viterbi Transformer

324

(2)

Global Training for Graph Transformer Networks

326

(6)

Viterbi Training

326

(1)

Discriminative Viterbi Training

327

(2)

Forward Scoring, and Forward Training

329

(1)

Discriminative Forward Training

330

(1)

Remarks on Discriminative Training

331

(1)

Multiple Object Recognition: Space Displacement Neural Network

332

(4)

Interpreting the Output of an SDNN with a GTN

333

(1)

Experiments with SDNN

333

(1)

Global Training of SDNN

334

(1)

Object Detection and Spotting with SDNN

335

(1)

Graph Transformer Networks and Tranducers

336

(3)

Previous Work

336

(1)

Standard Transduction

336

(1)

Generalized Transduction

336

(2)

Notes on the Graph Structures

338

(1)

GTN and Hidden Markov Models

339

(1)

An On-Line Handwriting Recognition System

339

(3)

Preprocessing

339

(1)

Network Architecture

340

(1)

Network Training

341

(1)

Experimental Results

341

(1)

A Check Reading System

342

(3)

A GTN for Check Amount Recognition

342

(2)

Gradient-Based Learning

344

(1)

Rejecting Low Confidence Checks

344

(1)

Results

344

(1)

Conclusions

345

(1)

Appendices

345

(3)

Pre-Conditions for Faster Convergence

345

(1)

Stochastic Gradient vs. Batch Gradient

346

(1)

Stochastic Diagonal Levenberg-Marquardt

346

(2)

Acknowledgments

348

(1)

References

348

(4)

Pattern Recognition Using a Family of Design Algorithms Based Upon Generalized Probabilistic Descent Method

352

(103)

Abstract

352

(1)

Introduction

353

(13)

Speech Pattern Recognition Using Modular System

354

(3)

Classifier Design Based on Maximum-Likelihood Method

357

(1)

Classifier Design Based on Discriminant Function Approach

358

(7)

Motivation of GPD Development and Paper Organization

365

(1)

Discriminative Pattern Classification

366

(20)

Bayes Decision Theory

366

(2)

Minimum Error Rate Classification

368

(2)

Discriminant Function Approach

370

(5)

Probabilistic Descent Method

375

(1)

Formalization

375

(3)

Probabilistic descent Theorem

378

(3)

Problems

381

(5)

Generalized Probabilistic Descent Method

386

(14)

Formulation Concept

386

(1)

GPD Embodiment for Distance Classifier

386

(1)

Discriminant function for dynamic patterns

387

(1)

Formulation

387

(3)

Design Optimality

390

(2)

Minimum Classification Error Learning

392

(4)

Speech Recognition Using MCE/GPD-trained Distance Classifiers

396

(3)

Remarks

399

(1)

Relationships Between MCE/GPD and Others

400

(5)

Relation with LVQ

400

(2)

Relation with Maximization of Mutual Information

402

(1)

Relation with Minimization of Squared Error Loss'

403

(2)

Derivatives of GPD

405

(21)

Overview

405

(1)

Segmental-GPD for Continuous Speech Recognition

406

(4)

Minimum Error Training for Open-Vocabulary Recognition

410

(1)

Open-Vocabulary speech recognition

410

(1)

Minimum spotting error learning

411

(4)

Discriminative utterance verification

415

(2)

Discriminative Feature Extraction

417

(1)

Fundamentals

417

(3)

An Example implementation for spectrum-based speech recognition

420

(1)

Discriminative metric design

421

(1)

Minimum error learning subspace method

422

(2)

Speaker Recognition Using GPD

424

(2)

Concluding Remarks

426

(6)

Recent Progress of DFA-Based Speech Recognition

426

(2)

Advantages of GPD Formalization

428

(2)

Future Issues in GPD-based Pattern Recognition

430

(2)

Acknowledgments

432

(1)

References

433

(14)

Appendix

447

(8)

An Approach to Adaptive Classification

455

(10)

Abstract

455

(1)

Introduction

455

(1)

The Proposed Learning Method

456

(4)

Problem Statement

456

(1)

Model Interence and Updates

457

(1)

Restarting

458

(1)

The Viterbi Algorithm

459

(1)

Synopsis of Our Learning Method

459

(1)

Relation to Other Learning Method

459

(1)

Examples

460

(3)

Fixed Linear and Nonlinear Generating Function

460

(1)

Time-Varying Function

461

(2)

Conclusions and Future Directions

463

(1)

References

463

(2)

Reduced-Rank Intelligent Signal Processing with Application to Radar

465

(19)

Introduction

465

(1)

Background

466

(1)

Karhunen-Loeve Analysis

467

(2)

The Karhunen-Loeve Transformation

467

(1)

The Karhunen-Loeve Expansion

467

(1)

Implementing the KLT

468

(1)

The Multiple Signal Model and Wiener Filtering

469

(1)

The Signal-Dependent KLT for Statistical Signal Processing

470

(2)

The KLT and Principal-Components

470

(1)

The Cross-Spectral Metic: An Intelligent and Signal-Dependent KLT

471

(1)

Intelligent Signal Representation for Statistical Signal Processing

472

(6)

A New Criterion for Signal Representation and Its Implementation

473

(1)

A Generalized Joint-Process KLT: Nonunitary Diagonalization of the Covariance

474

(1)

Analysis of the JKLT

475

(3)

Radar Example

478

(4)

Radar Signal Processing

478

(1)

Estimation of the Statistics and Sample Support

479

(1)

Simulation

480

(2)

Conclusions

482

(1)

Acknowledgments

482

(1)

References

482

(2)

Signal Detection in a Nonstationary Environment Reformulated as an Adaptive Pattern Classification Problem

484

(57)

Abstract

484

(1)

Introduction

485

(3)

An Overview of Nonstationary Behavior and Time-Frequency Analysis

488

(4)

Theoretical Background

492

(7)

Multiple Window Estimates

495

(2)

Spectrum Estimation as an Inverse Problem

497

(2)

High-Resolution Multiple-Window Spectrograms

499

(8)

Non-Stationary Quadratic-Inverse Theory

501

(4)

Multiple Window Estimates of the Loeve Spectrum

505

(2)

Spectrum Analysis of Radar Signals

507

(10)

Modular Learning Machine for Adaptive Signal Detection

517

(6)

How Does the Adaptive Receiver of Fig. 4 Respond to Nonstationary Environment

521

(2)

Case Study: Radar Target Detection of a Small Target in Sea Clutter

523

(9)

Details of the Receiver

525

(4)

Detection Results

529

(3)

Robustness of the Detector

532

(1)

Cost Functions for Supervised Training of the Pattern Classifiers

532

(3)

Summary and Discussion

535

(1)

Acknowledgments

536

(1)

References

536

(5)

Data Representation Using Mixtures of Principal Components

541

(13)

Abstract

541

(1)

Introduction

541

(1)

A Spectrum of Representations

542

(1)

Principal Components

542

(1)

Vector Quantization

542

(1)

A Mixture of Principal Components

542

(1)

Subspace Pattern Recognition

543

(1)

Similarity Measure

543

(1)

Class Prototype

544

(1)

Norm Invariance

544

(1)

Training

544

(2)

Topological Organization

546

(1)

Grey-Scale Feature Extraction

546

(2)

Training Data

547

(1)

Network Basis Vectors

547

(1)

Segmentation Results

547

(1)

Illumination Variations

548

(1)

Color Image Analysis

548

(3)

Color Representation

548

(2)

Vector Angle versus Euclidean Distance

550

(1)

Edge Detection

551

(1)

Segmentation

551

(1)

Conclusion

551

(1)

References

552

(2)

Image Denoising by Sparse Code Shrinkage

554

(15)

Abstract

554

(1)

Introduction

554

(1)

Maximum Likelihood Denoising of Nongaussian Random Variables

555

(3)

Maximum Likelihood Denoising

555

(1)

Modeling Sparse Densities

555

(1)

Laplace Density

555

(1)

Mildly Sparse Densities

556

(1)

Strongly Sparse Densities

557

(1)

Choice of Model

557

(1)

Some Other Models

557

(1)

Finding the Sparse Coding Transformation

558

(1)

Mean-Square Error Approach

558

(2)

Minimum Mean-Square Estimator in Scalar Case

558

(1)

Analysis of Mean-Square Error

559

(1)

Minimum Mean Squares Approach to Basis Estimation

559

(1)

Sparse Code Shrinkage

560

(1)

Comparison with Wavelet and Coring Methods

560

(1)

Extensions of the Basic Theory

561

(1)

Nongaussian Noise

561

(1)

Estimation of Parameters from Noisy Data

561

(1)

Non-orthogonal Bases

561

(1)

Experiments

561

(5)

Generation of Image Data

561

(1)

Remarks on Image Data

562

(1)

Windowing

562

(1)

The Local Mean

562

(1)

Normalizing the Local Variance

562

(1)

Transform Estimation

563

(1)

Methods

563

(1)

Results

563

(1)

Component Statistics

564

(1)

Denoising Results

565

(1)

Conclusion

566

(1)

References

567

(1)

Appendix

567

(2)

Index

569

(4)

About the Editors

573

Please wait while the item is added to your bag...

FREE SHIPPING

Rent More, Save More! Use code: BBRENTAL

5% off 1 book, 7% off 2 books, 10% off 3+ books

Intelligent Signal Processing

Summary

Author Biography

Table of Contents

Supplemental Materials

FREE SHIPPING

Rent More, Save More! Use code: BBRENTAL

5% off 1 book, 7% off 2 books, 10% off 3+ books

Intelligent Signal Processing

Summary

Author Biography

Table of Contents

Supplemental Materials

Digital License