A Field Guide to Dynamical Recurrent Networks

Note: Supplemental materials are not guaranteed with Rental or Used book purchases.

ISBN: 9780780353695 | 0780353692
Cover: Hardcover
Copyright: 1/15/2001

Buy New

Print on Demand: 2-4 Weeks. This item cannot be cancelled or returned.

$261.51

Acquire the tools for understanding new architectures and algorithms of dynamical recurrent networks (DRNs) from this valuable field guide, which documents recent forays into artificial intelligence, control theory, and connectionism. This unbiased introduction to DRNs and their application to time-series problems (such as classification and prediction) provides a comprehensive overview of the recent explosion of leading research in this prolific field.A Field Guide to Dynamical Recurrent Networks emphasizes the issues driving the development of this class of network structures. It provides a solid foundation in DRN systems theory and practice using consistent notation and terminology. Theoretical presentations are supplemented with applications ranging from cognitive modeling to financial forecasting.A Field Guide to Dynamical Recurrent Networks will enable engineers, research scientists, academics,and graduate students to apply DRNs to various real-world problems and learn about different areas of active research. It provides both state-of-the-art information and a road map to the future of cutting-edge dynamical recurrent networks.

Preface

xvii

Acknowledgments

xix

List of Figures

xxi

List of Tables

xxvii

List of Contributors

xxix

PART I INTRODUCTION

(12)

Dynamical Recurrent Networks

(10)

John F. Kolen

Stefan C. Kremer

Introduction

(1)

Dynamical Recurrent Networks

(2)

Overview

(5)

Architectures

(1)

Capabilities

(1)

Algorithms

(1)

Limitations

(1)

Applications

(1)

Conclusion

(2)

PART II ARCHITECTURES

(42)

Networks with Adaptive State Transitions

(12)

David Calvert

Stefan C. Kremer

Introduction

(1)

The Search for Context

(2)

Context as Input History

(1)

Recurrent Approaches to Context

(1)

Representing Context

(1)

Training

(1)

Architectures

(6)

Jordan

(2)

Elman

(2)

Williams and Zipser

(1)

Giles et al.

(1)

Locally Recurrent Connections

(1)

Conclusion

(2)

Delay Networks: Buffers to the Rescue

(12)

Tsung-Nan Lin

C. Lee Giles

Introduction to Delay Networks

(1)

Back-Propagation Through Time Learning Algorithm

(3)

Delay Networks with Feedback: NARX Networks

(2)

Long-Term Dependencies in NARX Networks

(3)

Vanishing Gradients and Long-Term Dependencies

(2)

The Effect of Delay Buffer on Vanishing Gradients

(1)

Experimental Results: The Latching Problem

(2)

Conclusion

(1)

Memory Kernels

(16)

Ah Chung Tsoi

Andrew Back

Jose Principe

Mike Mozer

Introduction

(1)

Different Types of Memory Kernels

(4)

The Parameters in Each Modular Component Are Different

(2)

Identical Parameters for Each Modular Component

(1)

Generic Representation of a Memory Kernel

(1)

State Observable Situation

(1)

Output Observable Situation

(1)

Basis Issues

(2)

Completeness

(1)

Gamma Filter

(1)

Laguerre Filters

(1)

General Conditions

(1)

Universal Approximation Theorem

(1)

Training Algorithms

(3)

Training Algorithm for the Generic Predictor Weights

(1)

State Observable Form

(1)

Output Observable Form

(1)

Illustrative Example

(3)

Conclusion

(1)

PART III CAPABILITIES

(98)

Dynamical Systems and Iterated Function Systems

(26)

John F. Kolen

Introduction

(1)

Dynamical Systems

(15)

Definition

(3)

Trajectories, Transients, and Attractors

(6)

Attractor Dimensionality

(1)

Bifurcations and Phase Transitions

(3)

Summary

(1)

Iterated Function Systems

(6)

Basic Iterated Function Systems Theory

(3)

Random Iteration

(1)

Summary

(2)

Symbolic Dynamics

(2)

The DRN Connection

(1)

Conclusion

(2)

Representation of Discrete States

(20)

C. Lee Giles

Christian Omlin

Introduction

(1)

Finite-State Automata

(2)

Neural Network Representations of DFA

(14)

Preliminaries

(1)

Feedforward Neural Networks

(2)

First- Versus Second-Order Networks

(3)

Locally Recurrent Neural Networks

(2)

Recurrent Cascade Correlation Networks

(2)

Elman Networks

(3)

NARX Recurrent Neural Networks

(2)

Pushdown Automata

(2)

Turing Machines

101

(1)

Conclusion

102

(1)

Simple Stable Encodings of Finite-State Machines in Dynamic Recurrent Networks

103

(26)

Mikel L. Forcada

Raphael C. Carrasco

Introduction

103

(3)

A Survey of Previous Work

103

(3)

Organization of the Chapter

106

(1)

Definitions

106

(3)

Mealy and Moore Machines and Deterministic Finite-State Automaton

106

(1)

Dynamic Recurrent Networks

107

(2)

Encoding

109

(5)

Conditions for a DRN to Behave as a FSM

109

(2)

A General Encoding Scheme

111

(3)

Encoding of Mealy Machines in DRN

114

(9)

Mealy Machines in Second-Order DRN

114

(6)

Mealy Machines in First-Order DRN

120

(3)

Encoding of Moore Machines in DRN

123

(2)

Moore Machines in First-Order DRN

123

(1)

Moore Machines in Second-Order DRN

124

(1)

Encoding of Deterministic Finite-State Automata in DRN

125

(1)

Encoding DFA in Elman Nets

125

(1)

Encoding DFA in Second-Order DRN

125

(1)

Conclusion

126

(1)

Acknowledgments

127

(2)

Representation Beyond Finite States: Alternatives to Pushdown Automata

129

(14)

Janet Wiles

Alan D. Blair

Mikael Boden

Introduction

129

(1)

Finding Structure in Continuous-State Space

129

(1)

Hierarchies of Languages and Machines

130

(4)

The Chomsky Hierarchy

130

(1)

Regular and Context-Free Languages

131

(1)

Context-Sensitive and Recursively Enumerable

132

(1)

Learning and Induction

132

(1)

Alternative Hierarchies

133

(1)

Performance Limitations

133

(1)

DRNs and Nonregular Languages

134

(7)

Augmenting DRNs with an External Stack

134

(1)

DRNs without an External Stack

135

(1)

Representing Counters in Recurrent Hidden Units

136

(2)

Learning, Stability, and Generalization for anbn

138

(1)

Learning the Context-Sensitive Language anbncn

139

(2)

Generalization and Inductive Bias

141

(1)

Conclusion

142

(1)

Universal Computation and Super-Turing Capabilities

143

(10)

Hava T. Siegelmann

Introduction

143

(1)

The Model

144

(1)

Preliminary: Computational Complexity

145

(1)

Summary of Results

146

(3)

Networks That Are Turing Equivalent

147

(1)

Networks That Are Beyond Turing

148

(1)

Pondering Real Weights

149

(1)

Analog Computation

149

(1)

Conclusion

150

(1)

Acknowledgments

151

(2)

PART IV ALGORITHMS

153

(76)

Insertion of Prior Knowledge

155

(24)

Paolo Frasconi

C. Lee Giles

Marco Gori

Christian Omlin

Introduction

155

(1)

Constrained Nondeterministic Insertion in First-Order Networks

156

(4)

Motivations

156

(1)

Time-Warping Automata

156

(1)

Mapping Network Dynamics into Symbolic Dynamics

157

(1)

State Memorization

157

(1)

Neural Logic Operators

158

(1)

The Injection Algorithm

159

(1)

Second-Order Networks

160

(15)

DFA Encoding Algorithm

160

(2)

Empirical Results

162

(2)

Stability of the DFA Representation

164

(2)

Scaling Issues

166

(1)

DFA States with Large Indegree

166

(1)

Extension to Fuzzy Domains

167

(3)

Fuzzy Representation

170

(5)

Other Related Techniques

175

(2)

Recurrent Radial Basis Function Networks

175

(1)

Injecting ``Automata Knowledge'' in R2BF Networks

176

(1)

Conclusion

177

(2)

Gradient Calculations for Dynamic Recurrent Neural Networks

179

(28)

Barak A. Pearlmutter

Introduction

179

(3)

Why Recurrent Networks?

179

(1)

Why Hidden Units?

180

(1)

Continuous Versus Discrete Time

181

(1)

Learning in Networks with Fixed Points

182

(6)

Will a Fixed Point Exist?

182

(1)

Problems with Fixed Points

183

(1)

Recurrent Back Propagation

183

(3)

Deterministic Boltzmann Machines

186

(2)

Computing the Gradient Without Assuming a Fixed Point

188

(8)

Back Propagation Through Time

188

(3)

Real-Time Recurrent Learning

191

(1)

More Efficient Online Techniques

192

(1)

Time Constants

193

(1)

Time Delays

193

(1)

Extended RTRL to Time Constants and Time Delays

194

(1)

Long Short-Term Memory

195

(1)

Some Simulations

196

(2)

A Rotated Figure Eight

197

(1)

Computational Neuroscience: A Simulated Leech

197

(1)

Stability and Perturbation Experiments

198

(1)

Other Non-Fixed Point-Techniques

199

(4)

``Elman Nets''

199

(1)

The Moving Targets Method

199

(1)

Feedforward Networks with State

200

(1)

Teacher Forcing in Continuous Time

200

(1)

Jordan Nets

201

(1)

Teacher Forcing, RTRL, and the Kalman Filter

202

(1)

Learning with Scale Parameters

203

(1)

Conclusion

203

(4)

Complexity Comparison

203

(1)

Speeding the Optimization

204

(1)

Prospects and Future Work

205

(1)

Conclusion

205

(2)

Understanding and Explaining DRN Behavior

207

(22)

Christian Omlin

Introduction

207

(1)

Performance Deterioration

208

(1)

Dynamic Space Exploration

209

(6)

Extraction Algorithm

209

(2)

Example

211

(2)

Model Selection

213

(2)

DFA Extraction: Fool's Gold?

215

(1)

Theoretical Foundations

216

(2)

How Can DFA Outperform Networks?

218

(2)

Definitions

218

(1)

Example

218

(2)

Alternative Extraction Methods

220

(5)

Self-Clustering Recurrent Networks

221

(1)

Explicit Cluster Modeling

221

(1)

Learning Discrete Representations

222

(1)

DFA Extraction as a Learning Problem

223

(2)

Extension to Fuzzy Automata

225

(1)

Application to Financial Forecasting

226

(1)

A Hybrid Neural Network Architecture

226

(1)

Conclusion

227

(2)

PART V LIMITATIONS

229

(26)

Evaluating Benchmark Problems by Random Guessing

231

(6)

Jurgen Schmidhuber

Sepp Hochreiter

Yoshua Bengio

Introduction

231

(1)

Random Guessing (RG)

231

(1)

Experiments

232

(2)

Latch and 2-Sequence Problems

232

(1)

Parity Problem

233

(1)

Tomita Grammars

234

(1)

Final Remarks

234

(1)

Conclusion

235

(1)

Acknowledgments

235

(2)

Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies

237

(8)

Sepp Hochreiter

Yoshua Bengio

Paolo Frasconi

Jurgen Schmidhuber

Introduction

237

(1)

Exponential Error Decay

237

(3)

Dilemma: Avoiding Aradient Decay Prevents Long-Term Latching

240

(1)

Remedies

241

(2)

Conclusion

243

(2)

Limiting the Computational Power of Recurrent Neural Networks: VC Dimension and Noise

245

(10)

Christopher Moore

Introduction

245

(1)

Time-Bounded Networks and VC Dimension

246

(4)

Real-Time Dynamical Recognizers

246

(1)

VC Dimension

247

(1)

A Memory Task That Needs a Large VC Dimension

248

(1)

Sinusoidal Transfer Functions and Others

249

(1)

Robustness to Noise

250

(4)

Robustness with Perfect Reliability

250

(1)

Performance as a Function of Noise

251

(2)

Robustness with Limited Reliability

253

(1)

Conclusion

254

(1)

Acknowledgments

254

(1)

PART VI APPLICATIONS

255

(120)

Dynamical Recurrent Networks in Control

257

(34)

Danil V. Prokhorov

Gintaras V. Puskorius

Lee A. Feldkamp

Introduction

257

(1)

Description and Execution of TLRNN

258

(2)

Elements of Training

260

(6)

Derivative Calculations

260

(2)

Weight Update Methods

262

(4)

Basic Approach to Controller Synthesis

266

(6)

Specifics of Controller Training

266

(2)

Modular Approach

268

(2)

Multi-Stream Training

270

(2)

Example 1

272

(10)

MIMO Control Problem

272

(1)

State Variables Accessible and Plant Equations Known

273

(2)

State Variables Inaccessible and Plant Equations Unknown

275

(2)

Further Testing, Improvements, and Summary of the Results

277

(5)

Example 2

282

(6)

Financial Portfolio Optimization

282

(2)

General Training Considerations

284

(1)

Empirical Simulations

284

(4)

Possible Extensions

288

(1)

Conclusion

288

(3)

Sentence Processing and Linguistic Structure

291

(20)

Whitney Tabor

Introduction

291

(4)

The Role of Recurrent Connectionist Networks

292

(2)

The Role of Sentence Processing Studies

294

(1)

Overview

295

(1)

Case Studies: Dynamical Networks for Sentence Processing

295

(13)

Frequency Sensitivity

297

(4)

Phrase Structure

301

(7)

Conclusion

308

(3)

Neural Network Architectures for the Modeling of Dynamic Systems

311

(40)

Hans-Georg Zimmermann

Ralph Neuneier

Introduction and Overview

311

(1)

Modeling Dynamic Systems by Feedforward Neural Networks

312

(9)

Preprocessing the Data

312

(1)

Suppressing Outliers by Net Internal Preprocessing

313

(1)

Merging MLP and RBF Networks

314

(2)

Modeling Dynamics by an Interaction Layer

316

(1)

Statistical Averaging

317

(1)

An Integrated Feedforward Architecture for Forecasting

318

(3)

Modeling Dynamic Systems by Recurrent Neural Networks

321

(13)

Representing Dynamic Systems by Recurrent Networks

322

(1)

Finite Unfolding in Time

323

(3)

Overshooting

326

(1)

Analysis of Partially Externally Driven Systems

327

(1)

Principle of Uniform Causality

328

(2)

Undershooting

330

(2)

Special Dynamics

332

(2)

Combining State-Space Reconstruction and Forecasting

334

(16)

Finite Unfolding in Space and Time

335

(3)

Smoothness

338

(3)

Separation of Variants and Invariants

341

(1)

Nonlinearity Versus Noise

342

(2)

Intrinsic Time

344

(1)

Measuring the Observed Dynamics

344

(2)

Extended State-Space Transformations and Periodic Orbits

346

(1)

Overshooting in Space and Time

347

(1)

Undershooting in Space and Time

348

(2)

Conclusion

350

(1)

From Sequences to Data Structures: Theory and Applications

351

(24)

Paolo Frasconi

Marco Gori

Andreas Kuchler

Alessandro Sperduti

Introduction

351

(1)

Historical Remarks

352

(2)

Adaptive Processing of Structured Information

354

(12)

Structured Domains

354

(1)

The Computational Model

355

(2)

Graphical Transduction

357

(2)

On the Representational Power of Recursive Models

359

(1)

Learning Algorithms

359

(7)

Applications

366

(8)

Pattern Recognition

367

(1)

Computational Chemistry

368

(2)

Adaptive Search Control for Symbolic Deduction Systems

370

(4)

Conclusion

374

(1)

PART VII CONCLUSION

375

(4)

Dynamical Recurrent Networks: Looking Back and Looking Forward

377

(2)

Stefan C. Kremer

John F. Kolen

Introduction

377

(1)

The Challenges

377

(1)

The Potential

378

(1)

The Approaches

378

(1)

The Successes

378

(1)

Conclusion

378

(1)

Bibliography

379

(30)

Glossary

409

(6)

Index

415

(8)

About the Editors

423

Please wait while the item is added to your bag...

FREE SHIPPING

Amazon no longer offers textbook rentals. We do!

A Field Guide to Dynamical Recurrent Networks

Summary

Author Biography

Table of Contents

Supplemental Materials