Introduction to Stochastic Processes with R

Citation

Dobrow, R. P. (2016). Introduction to Stochastic Processes with R. Wiley.

Chapter Summary

The book runs to about 500 pages: nine chapters, then four appendices (a tutorial on getting started with R, a probability review, a summary of common distributions and a matrix algebra review) and answers to selected odd-numbered exercises. The publisher counts more than 200 examples and 600 end-of-chapter exercises, and a companion website holds the data files and all the R code and scripts used in the book. Sections marked with an asterisk are optional, as is the whole of the final chapter.

Chapter 1: Introduction and Review

The chapter contrasts deterministic and stochastic models, using exponential growth of a bacteria population alongside a random birth model, and defines a stochastic process through examples such as PageRank, the Reed–Frost epidemic model, the board game Monopoly, random walk with gambler's ruin and Brownian motion. It introduces Monte Carlo simulation and then gives a long review of conditional probability and conditional expectation, including random sums of random variables, the tools the rest of the book relies on.

Chapter 2: Markov Chains: First Steps

A first treatment of discrete-time Markov chains: the definition, a "cornucopia" of examples, basic computations with transition matrices, numerical evidence about long-term behaviour, and simulation. An optional section reviews mathematical induction.

Chapter 3: Markov Chains for the Long Term

The longest chapter. It covers limiting and stationary distributions, communication between states, irreducibility, periodicity, ergodic chains, time reversibility and absorbing chains. Optional sections treat regeneration and the strong Markov property and give proofs of the limit theorems.

Chapter 4: Branching Processes

Population models in which each individual has a random number of offspring: mean generation size, probability generating functions and the probability of extinction.

Chapter 5: Markov Chain Monte Carlo

Markov chains as a simulation tool. The chapter presents the Metropolis–Hastings algorithm and the Gibbs sampler, with optional sections on perfect sampling, the link between rate of convergence and eigenvalues, and card shuffling measured by total variation distance.

Chapter 6: Poisson Process

Arrival and interarrival times, infinitesimal probabilities, thinning and superposition, the uniform distribution of arrival times, spatial and nonhomogeneous Poisson processes, and a closing section titled "Parting Paradox".

Chapter 7: Continuous-Time Markov Chains

Chains that move in continuous time, introduced through "alarm clocks" and transition rates, then the infinitesimal generator, long-term behaviour, time reversibility, queueing theory and Poisson subordination.

Chapter 8: Brownian Motion

Brownian motion as a limit of random walk, as a Gaussian process, and through its transformations and properties, followed by variations and applications and an introduction to martingales.

Chapter 9: A Gentle Introduction to Stochastic Calculus*

An optional closing chapter on the Itô integral and stochastic differential equations. The publisher lists Black–Scholes option pricing among the book's topics.

Key Concepts

Chapters 1–3: Foundations and discrete-time Markov chains

• Conditional expectation: the expected value of one random variable given another, treated as a random variable in its own right.
• Monte Carlo simulation: estimating probabilities and expectations by simulating a random process many times.
• Transition matrix: the matrix of one-step probabilities whose powers give multi-step behaviour.
• Stationary and limiting distributions: long-run behaviour of a chain, guaranteed for ergodic chains.
• Absorbing chains: chains that eventually stop in a trapping state, used for questions such as gambler's ruin.

Chapters 4–5: Branching and MCMC

• Probability generating function: the tool used to compute extinction probabilities of a branching process.
• Metropolis–Hastings and Gibbs sampling: algorithms that build a Markov chain whose stationary distribution is a desired target.
• Total variation distance: a measure of how close a chain is to stationarity, applied to card shuffling.

Chapters 6–7: Continuous time

• Poisson process: random arrivals with independent, exponentially distributed interarrival times.
• Thinning and superposition: splitting and merging Poisson processes.
• Infinitesimal generator: the matrix of transition rates that describes a continuous-time chain.
• Queueing models: continuous-time chains for customers arriving and being served.

Chapters 8–9: Brownian motion and stochastic calculus

• Brownian motion: a continuous-time, continuous-state process obtained as a limit of random walks, and a Gaussian process.
• Martingale: a process whose expected future value, given the present, equals its current value.
• Itô integral and stochastic differential equations: integration with respect to Brownian motion and the equations built from it.

Critical Analysis

Strengths

• Simulation built in. R is part of the presentation, not an add-on: Chapter 1 introduces Monte Carlo simulation, Chapter 2 has a section on simulating Markov chains, exercises marked "R" send the reader to script files such as the Reed–Frost model, and the companion website holds all the code used in the book. Appendix A gets readers started with R.
• Examples from many fields. The opening chapter alone draws on web search, epidemics, board games, air monitoring and gambling, and the publisher lists random walk on graphs, card shuffling, genetics, cryptography and option pricing among the topics. The examples are short and concrete.
• Wide coverage for an introduction. Besides the standard Markov chains and Poisson processes, the book includes branching processes, a full chapter on Markov chain Monte Carlo, queueing, Brownian motion, martingales and a first look at Itô calculus, so a course can choose its emphasis.
• Support for self-study. Review appendices on probability and matrix algebra, a long review of conditional probability and expectation in Chapter 1, and answers to selected odd-numbered exercises make the book usable outside a classroom.

Limitations

• Breadth over depth. The optional "Gentle Introduction" to stochastic calculus covers the Itô integral and stochastic differential equations in about 30 pages; readers interested in finance or SDEs will need the dedicated books in this course. Renewal theory does not appear as a topic in the table of contents.
• Proofs partly optional. The proofs of the Markov chain limit theorems sit in a starred section at the end of Chapter 3, so a course that skips starred material proves less than it states.
• Assumes a probability course. Chapter 1 and Appendix B review probability, but at a quick pace; the book is not a first course in probability.
• Tied to one language. The software support is in R only; readers who work in Python or other languages must translate the scripts.

Where It Fits in the Course

This is a natural first textbook for the stochastic processes course, especially for students who learn well by simulating. Sheldon Ross's Introduction to Probability Models covers similar ground (Markov chains, Poisson processes, continuous-time chains) with more weight on applications such as queues, reliability and renewal phenomena and with many worked examples; it can be used alongside Dobrow for additional problems or as an alternative main text. Sidney Resnick's Adventures in Stochastic Processes is a more rigorous treatment of Markov chains, renewal and point processes, Brownian motion and random walks, and is a good next step after Chapters 3, 6 and 8.

For finance and stochastic calculus, Dobrow's Chapter 9 is only a preview. Steven Shreve's Stochastic Calculus for Finance I develops martingales, change of measure and option pricing in the discrete-time binomial model and pairs well with Chapters 3 and 8 here. Bernt Øksendal's Stochastic Differential Equations, Lamberton and Lapeyre's Introduction to Stochastic Calculus Applied to Finance and Tomas Björk's Arbitrage Theory in Continuous Time then take up Itô calculus and continuous-time finance in full.

How to Use This Book

The book suits undergraduates in mathematics, statistics, the sciences and engineering who have taken a calculus-based course in probability and some linear algebra; the publisher also points to beginning graduate students and to applied mathematicians and statisticians wanting a review. Appendix A provides a start in R for those new to it.

A reading plan for a one-semester course:

• Core: Chapter 1 (move quickly through the review if your probability is strong, but work the conditional expectation examples), Chapters 2 and 3, Chapter 6 and Chapter 7.
• Choose one or two: Chapter 4 for population models, Chapter 5 for simulation and Bayesian computation, Chapter 8 for Brownian motion and martingales.
• Optional: the starred sections, which include the proofs of the limit theorems in Chapter 3, and Chapter 9 for students heading to finance.

Run the R scripts from the companion website as you read: simulate a chain, then compare the simulation with the exact stationary distribution or absorption probability computed in the text. Do the exercises marked "R" along with the analytic ones, and check odd-numbered answers in the back of the book.