Recurrent Neural Networks for Prediction Learning Algorithms, Architectures and Stability

, by Mandic, Danilo P.; Chambers, Jonathon A.

Note: Supplemental materials are not guaranteed with Rental or Used book purchases.

ISBN: 9780471495178 | 0471495174
Cover: Hardcover
Copyright: 9/5/2001

Buy New

Print on Demand: 2-4 Weeks. This item cannot be cancelled or returned.

$289.87

New technologies in engineering, physics and biomedicine are demanding increasingly complex methods of digital signal processing. By presenting the latest research work the authors demonstrate how real-time recurrent neural networks (RNNs) can be implemented to expand the range of traditional signal processing techniques and to help combat the problem of prediction. Within this text neural networks are considered as massively interconnected nonlinear adaptive filters. ? Analyses the relationships between RNNs and various nonlinear models and filters, and introduces spatio-temporal architectures together with the concepts of modularity and nesting ? Examines stability and relaxation within RNNs ? Presents on-line learning algorithms for nonlinear adaptive filters and introduces new paradigms which exploit the concepts of a priori and a posteriori errors, data-reusing adaptation, and normalisation ? Studies convergence and stability of on-line learning algorithms based upon optimisation techniques such as contraction mapping and fixed point iteration ? Describes strategies for the exploitation of inherent relationships between parameters in RNNs ? Discusses practical issues such as predictability and nonlinearity detecting and includes several practical applications in areas such as air pollutant modelling and prediction, attractor discovery and chaos, ECG signal processing, and speech processing Recurrent Neural Networks for Prediction offers a new insight into the learning algorithms, architectures and stability of recurrent neural networks and, consequently, will have instant appeal. It provides an extensive background for researchers, academics and postgraduates enabling them to apply such networks in new applications. VISIT OUR COMMUNICATIONS TECHNOLOGY WEBSITE! http://www.wiley.co.uk/commstech/ VISIT OUR WEB PAGE! http://www.wiley.co.uk/

Preface

Introduction

(8)

Some Important Dates in the History of Connectionism

(1)

The Structure of Neural Networks

(2)

Perspective

(1)

Neural Networks for Prediction: Perspective

(1)

Structure of the Book

(2)

Readership

(1)

Fundamentals

(22)

Perspective

(1)

Chapter Summary

(1)

Adaptive Systems

(3)

Configurations of Adaptive Systems Used in Signal Processing

(2)

Blind Adaptive Techniques

(1)

Gradient-Based Learning Algorithms

(2)

A General Class of Learning Algorithms

(1)

Quasi-Newton Learning Algorithms

(1)

A Step-by-Step Derivation of the Least Mean Square (LMS) Algorithm

(3)

The Wiener Filter

(1)

Further Perspective on the Least Mean Square (LMS) Algorithm

(1)

On Gradient Descent for Nonlinear Structures

(1)

Extension to a General Neural Network

(1)

On Some Important Notions From Learning Theory

(5)

Relationship Between the Error and the Error Function

(1)

The Objective Function

(1)

Types of Learning with Respect to the Training Set and Objective Function

(1)

Deterministic, Stochastic and Adaptive Learning

(1)

Constructive Learning

(1)

Transformation of Input Data, Learning and Dimensionality

(2)

Learning Strategies

(1)

General Framework for the Training of Recurrent Networks by Gradient-Descent-Based Algorithms

(2)

Adaptive Versus Nonadaptive Training

(1)

Performance Criterion, Cost Function, Training Function

(1)

Recursive Versus Nonrecursive Algorithms

(1)

Iterative Versus Noniterative Algorithms

(1)

Supervised Versus Unsupervised Algorithms

(1)

Pattern Versus Batch Learning

(1)

Modularity Within Neural Networks

(3)

Summary

(2)

Network Architectures for Prediction

(16)

Perspective

(1)

Introduction

(1)

Overview

(1)

Prediction

(2)

Building Blocks

(2)

Linear Filters

(2)

Nonlinear Predictors

(2)

Feedforward Neural Networks: Memory Aspects

(2)

Recurrent Neural Networks: Local and Global Feedback

(1)

State-Space Representation and Canonical Form

(1)

Summary

(2)

Activation Functions Used in Neural Networks

(22)

Perspective

(1)

Introduction

(4)

Overview

(1)

Neural Networks and Universal Approximation

(3)

Other Activation Functions

(3)

Implementation Driven Choice of Activation Functions

(3)

MLP versus RBF Networks

(1)

Complex Activation Functions

(5)

Complex Valued Neural Networks as Modular Groups of Compositions of Mobius Transformations

(3)

Mobius Transformation

(1)

Activation Functions and Mobius Transformations

(2)

Existence and Uniqueness of Fixed Points in a Complex Neural Network via Theory of Modular Groups

(1)

Summary

(1)

Recurrent Neural Networks Architectures

(22)

Perspective

(1)

Introduction

(3)

Overview

(1)

Basic Modes of Modelling

(2)

Parametric versus Nonparametric Modelling

(1)

White, Grey and Black Box Modelling

(1)

NARMAX Models and Embedding Dimension

(1)

How Dynamically Rich are Nonlinear Neural Models?

(2)

Feedforward versus Recurrent Networks for Nonlinear Modelling

(1)

Wiener and Hammerstein Models and Dynamical Neural Networks

(4)

Overview of Block-Stochastic Models

(1)

Connection Between Block-Stochastic Models and Neural Networks

(3)

Recurrent Neural Network Architectures

(3)

Hybrid Neural Network Architectures

(2)

Nonlinear ARMA Models and Recurrent Networks

(3)

Summary

(2)

Neural Networks as Nonlinear Adaptive Filters

(24)

Perspective

(1)

Introduction

(1)

Overview

(1)

Neural Networks and Polynomial Filters

(3)

Neural Networks and Nonlinear Adaptive Filters

(6)

Training Algorithms for Recurrent Neural Networks

101

(1)

Learning Strategies for a Neural Predictor/Identifier

101

(4)

Learning Strategies for a Neural Adaptive Recursive Filter

103

(1)

Equation Error Formulation

104

(1)

Output Error Formulation

104

(1)

Filter Coefficient Adaptation for IIR Filters

105

(2)

Equation Error Coefficient Adaptation

107

(1)

Weight Adaptation for Recurrent Neural Networks

107

(2)

Teacher Forcing Learning for a Recurrent Perceptron

108

(1)

Training Process for a NARMA Neural Predictor

109

(1)

The Problem of Vanishing Gradients in Training of Recurrent Neural Networks

109

(2)

Learning Strategies in Different Engineering Communities

111

(1)

Learning Algorithms and the Bias/Variance Dilemma

111

(2)

Recursive and Iterative Gradient Estimation Techniques

113

(1)

Exploiting Redundancy in Neural Network Design

113

(1)

Summary

114

(1)

Stability Issues in RNN Architectures

115

(20)

Perspective

115

(1)

Introduction

115

(3)

Overview

118

(1)

A Fixed Point Interpretation of Convergence in Networks with a Sigmoid Nonlinearity

118

(6)

Some Properties of the Logistic Function

118

(3)

Logistic Function, Rate of Convergence and Fixed Point Theory

121

(3)

Convergence of Nonlinear Relaxation Equations Realised Through a Recurrent Perceptron

124

(3)

Relaxation in Nonlinear Systems Realised by an RNN

127

(3)

The Iterative Approach and Nesting

130

(3)

Upper Bounds for GAS Relaxation within FCRNNs

133

(1)

Summary

133

(2)

Data-Reusing Adaptive Learning Algorithms

135

(14)

Perspective

135

(1)

Introduction

135

(3)

Towards an A Posteriori Nonlinear Predictor

136

(1)

Note on the Computational Complexity

137

(1)

Chapter Summary

138

(1)

A Class of Simple A Posteriori Algorithms

138

(4)

The Case of a Recurrent Neural Filter

140

(1)

The Case of a General Recurrent Neural Network

141

(1)

Example for the Logistic Activation Function

141

(1)

An Iterated Data-Reusing Learning Algorithm

142

(1)

The Case of a Recurrent Predictor

143

(1)

Convergence of the A Posteriori Approach

143

(1)

A Posteriori Error Gradient Descent Algorithm

144

(2)

A Posteriori Error Gradient Algorithm for Recurrent Neural Networks

146

(1)

Experimental Results

146

(1)

Summary

147

(2)

A Class of Normalised Algorithms for Online Training of Recurrent Neural Networks

149

(12)

Perspective

149

(1)

Introduction

149

(1)

Overview

150

(1)

Derivation of the Normalised Adaptive Learning Rate for a Simple Feedforward Nonlinear Filter

151

(5)

A Normalised Algorithm for Online Adaptation of Recurrent Neural Networks

156

(4)

Summary

160

(1)

Convergence of Online Learning Algorithms in Neural Networks

161

(10)

Perspective

161

(1)

Introduction

161

(3)

Overview

164

(1)

Convergence Analysis of Online Gradient Descent Algorithms for Recurrent Neural Adaptive Filters

164

(3)

Mean-Squared and Steady-State Mean-Squared Error Convergence

167

(2)

Convergence in the Mean Square

168

(1)

Steady-State Mean-Squared Error

169

(1)

Summary

169

(2)

Some Practical Considerations of Predictability and Learning Algorithms for Various Signals

171

(28)

Perspective

171

(1)

Introduction

171

(3)

Detecting Nonlinearity in Signals

173

(1)

Overview

174

(1)

Measuring the Quality of Prediction and Detecting Nonlinearity within a Signal

174

(7)

Deterministic Versus Stochastic Plots

175

(1)

Variance Analysis of Delay Vectors

175

(1)

Dynamical Properties of NO2 Air Pollutant Time Series

176

(5)

Experiments on Heart Rate Variability

181

(14)

Experimental Results

181

(14)

Prediction of the Lorenz Chaotic Series

195

(2)

Bifurcations in Recurrent Neural Networks

197

(1)

Summary

198

(1)

Exploiting Inherent Relationships Between Parameters in Recurrent Neural Networks

199

(22)

Perspective

199

(1)

Introduction

199

(5)

Overview

204

(1)

Static and Dynamic Equivalence of Two Topologically Identical RNNs

205

(4)

Dynamic Equivalence of Two Isomorphic RNNs

205

(3)

Extension to a General RTRL Trained RNN

208

(1)

Extension to Other Commonly Used Activation Functions

209

(1)

Extension to Other Commonly Used Learning Algorithms for Recurrent Neural Networks

209

(1)

Relationships Between β and n for the Backpropagation Through Time Algorithm

210

(3)

Results for the Recurrent Backpropagation Algorithm

211

(1)

Results for Algorithms with a Momentum Term

211

(1)

Simulation Results

212

(1)

Summary of Relationships Between β and n for General Recurrent Neural Networks

213

(1)

Relationship Between n and β for Modular Neural Networks: Perspective

214

(1)

Static Equivalence Between an Arbitrary and a Referent Modular Neural Network

214

(1)

Dynamic Equivalence Between an Arbitrary and a Referent Modular Network

215

(1)

Dynamic Equivalence for a GD Learning Algorithm

216

(3)

Dynamic Equivalence Between Modular Recurrent Neural Networks for the ERLS Learning Algorithm

217

(1)

Equivalence Between an Arbitrary and the Referent PRNN

218

(1)

Note on the β-n-W Relationships and Contractivity

218

(1)

Summary

219

(2)

Appendix A The O Notation and Vector and Matrix Differentiation

221

(2)

A.1 The O Notation

221

(1)

A.2 Vector and Matrix Differentiation

221

(2)

Appendix B Concepts from the Approximation Theory

223

(4)

Appendix C Complex Sigmoid Activation Functions, Holomorphic Mappings and Modular Groups

227

(4)

C.1 Complex Sigmoid Activation Functions

227

(4)

C.1.1 Modular Groups

228

(3)

Appendix D Learning Algorithms for RNNs

231

(8)

D.1 The RTRL Algorithm

231

(3)

D.1.1 Teacher Forcing Modification of the RTRL Algorithm

234

(1)

D.2 Gradient Descent Learning Algorithm for the PRNN

234

(2)

D.3 The ERLS Algorithm

236

(3)

Appendix E Terminology Used in the Field of Neural Networks

239

(2)

Appendix F On the A Posteriori Approach in Science and Engineering

241

(4)

F.1 History of A Posteriori Techniques

241

(1)

F.2 The Usage of A Posteriori

242

(3)

F.2.1 A Posteriori Techniques in the RNN Framework

242

(1)

F.2.2 The Geometric Interpretation of A Posteriori Error Learning

243

(2)

Appendix G Contraction Mapping Theorems

245

(6)

G.1 Fixed Points and Contraction Mapping Theorems

245

(1)

G.1.1 Contraction Mapping Theorem in R

245

(1)

G.1.2 Contraction Mapping Theorem in RN

246

(1)

G.2 Lipschitz Continuity and Contraction Mapping

246

(1)

G.3 Historical Perspective

247

(4)

Appendix H Linear GAS Relaxation

251

(12)

H.1 Relaxation in Linear Systems

251

(2)

H.1.1 Stability Result for

253

(1)

H.2 Examples

253

(10)

Appendix I The Main Notions in Stability Theory

263

(2)

Appendix J Deseasonalising Time Series

265

(2)

References

267

(14)

Index

281

Please wait while the item is added to your bag...

FREE SHIPPING

Amazon no longer offers textbook rentals. We do!

Recurrent Neural Networks for Prediction Learning Algorithms, Architectures and Stability

Summary

Author Biography

Table of Contents

Supplemental Materials