Computer Vision: Algorithms and Applications

Citation

Szeliski, R. (2022). Computer Vision: Algorithms and Applications (2nd ed.). Springer. Author's page for the free online edition: https://szeliski.org/Book

Chapter Summary

The second edition has fifteen chapters and a set of appendices, about 725 pages of main text and 925 pages in the Springer edition. It is a revision built around deep learning: the author explains in an interview that the first edition (2010) followed the traditional layering of low-level, mid-level and high-level vision, and that deep learning upended that organisation. Chapter lengths below are taken from the Springer page ranges and show where the weight lies. Each chapter ends with exercises, which the publisher says emphasise testing algorithms and include suggestions for small mid-term projects.

Chapter 1: Introduction (about 26 pages)

The opening chapter introduces the field and, according to the publisher, includes tips for using the book in a variety of customised and project-oriented courses.

Chapter 2: Image Formation (about 57 pages)

How an image comes to be formed from a scene. As the first technical chapter, it underlies the later chapters on features, motion and 3D structure.

Chapter 3: Image Processing (about 67 pages)

The basic operations that transform and analyse images, the working vocabulary of low-level vision.

Chapter 4: Model Fitting and Optimization (about 34 pages)

The shortest of the technical foundation chapters; it concerns fitting models to data and the optimisation methods used throughout the later chapters.

Chapter 5: Deep Learning (about 85 pages)

The longest chapter and the main new material in this edition. The author credits a student with writing the material on the more advanced deep-learning topics. The publisher notes that the edition treats recent deep-learning models alongside the classical ones.

Chapter 6: Recognition (about 59 pages)

How computers recognise what an image contains.

Chapters 7–9: Feature Detection and Matching; Image Alignment and Stitching; Motion Estimation (about 67, 41 and 40 pages)

These chapters cover finding and matching image features, aligning and stitching images, and estimating motion between frames.

Chapters 10–14: Computational Photography; Structure from Motion and SLAM; Depth Estimation; 3D Reconstruction; Image-Based Rendering (about 60, 52, 44, 42 and 42 pages)

The second half of the book moves to applications and 3D: computational photography, recovering camera motion and structure from images (including SLAM), estimating depth, reconstructing 3D shape, and rendering new views from images. The publisher names applications such as mobile computational photography, autonomous navigation and augmented reality.

Chapter 15: Conclusion (3 pages)

A very short closing chapter.

Appendices

The publisher lists appendices on linear algebra, numerical techniques, estimation theory, datasets and software.

Key Concepts

Chapters 1–4: Foundations

• Image formation: the link between a 3D scene and a 2D image.
• Image processing: the operations used to transform and analyse images.
• Model fitting and optimisation: estimating parameters from data.

Chapters 5–6: Learning and recognition

• Deep learning: the neural-network methods that, in the author's words, have taken over as the preferred technique.
• Recognition: identifying what an image contains.

Chapters 7–9: Features, alignment and motion

• Feature detection and matching: finding and relating points across images.
• Alignment and stitching; motion estimation: relating multiple views and frames.

Chapters 10–14: Photography and 3D

• Computational photography: image capture and processing pipelines.
• Structure from motion and SLAM; depth estimation; 3D reconstruction; image-based rendering: recovering and using 3D information.

Critical Analysis

Strengths

• A scientific, project-friendly approach. The publisher describes a "scientific approach to the formulation of computer vision problems" analysed with classical and deep learning models, with exercises that stress testing algorithms and suggest small mid-term projects.
• Updated to the deep-learning era. The second edition adds new material on deep learning and on applications such as mobile computational photography, autonomous navigation and augmented reality, plus about 1,500 new citations and 200 new figures. Chapter 5 is the longest in the book.
• Free to read. The author provides an electronic edition for personal use from his website, which makes the book unusually accessible for self-study.
• Reference value. The author describes the book as a reference peppered with citations, a starting point for people who want to learn about a topic and the papers to read next.

Limitations

• Heavy on references for beginners. The author acknowledges that some undergraduates find the many references distracting and says a shorter book with fewer of them would be the ideal undergraduate textbook.
• Very broad. Fifteen chapters spanning geometry, learning, photography and graphics-related topics cannot be covered in a single course; readers have to choose.
• No official slides. The author's page says there are no official slide sets for the book, only slides from related university courses.

Where It Fits in the Course

Within this machine-learning and AI course list, Szeliski is the specialist book on vision. James et al.'s An Introduction to Statistical Learning with Applications in Python and Kevin Murphy's Probabilistic Machine Learning supply general learning foundations (the first with minimal mathematics and Python labs, the second a comprehensive probabilistic treatment), which are a natural background for Chapter 5. Russell and Norvig's Artificial Intelligence: A Modern Approach gives a broad AI context that includes vision, and Melanie Mitchell's Artificial Intelligence: A Guide for Thinking Humans is an accessible, non-technical account of what current systems can and cannot do.

Read Szeliski after a first machine-learning course, when the interest turns to images and video specifically.

How to Use This Book

The publisher suggests the book for an upper-level undergraduate or graduate course in computer science or engineering; the list of appendices indicates that linear algebra and numerical methods are part of the expected toolkit, and programming practice is needed for the exercises.

A practical plan:

• Core: Chapters 1–3 for the basics, Chapter 4 for fitting and optimisation, Chapter 5 for deep learning, and Chapter 6 for recognition.
• Geometry track: Chapters 7–9, then 11–13. The University of Washington graduate course from which the book grew (CSE 576, 2020) assigned chapters roughly in the order 1, 2–3, 4, 5, 6, 7–8, 9, 12, 11, 13, 10.
• Applications: Chapter 10 (computational photography) and Chapter 14 (image-based rendering) can be read independently of the geometry track, according to interest.
• Practice: do the exercises, especially the small projects, and read the cited papers for any topic you intend to implement.