Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models

Citation

Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of 19 August 2026). Online manuscript. Free official edition: https://web.stanford.edu/~jurafsky/slp3/

Chapter Summary

The free third-edition draft has 26 chapters in three parts—Large Language Models; Advanced LLM Topics and Tools; Annotating Linguistic Structure—plus web-only appendices. The summaries below follow the chapter titles in the draft's table of contents.

Chapter 1: Introduction

The new opening chapter introduces large language models (LLMs) for newcomers: what an LLM is, what models learn from predicting the next token, the underpinnings in neural networks and embeddings, and a brief history of speech and language processing from Shannon and early distributional semantics onwards. It previews how models are trained (pretraining, instruction, preference and verifiable-reward data), inference, agents and evaluation, and closes with safety and alignment and a discussion of anthropomorphic terminology.

Chapter 2: Words and Tokens

Starting from ELIZA's pattern matching, the chapter defines words and morphemes, explains Unicode code points and UTF-8, and presents byte-pair encoding as the standard data-driven tokeniser for LLMs. It also covers corpora, regular expressions, simple Unix tools and rule-based tokenisation, and the minimum edit distance algorithm.

Chapter 3: N-gram Language Models

N-gram models are used to introduce the core ideas of language modelling: estimating probabilities from counts, training and test sets, perplexity, sampling sentences, overfitting, and smoothing, interpolation and backoff, with an advanced section relating perplexity to entropy.

Chapter 4: Logistic Regression

This chapter (headed "Logistic Regression and Text Classification") introduces supervised classification with the sigmoid, cross-entropy loss and gradient descent, extends it to multinomial (softmax) regression, and covers evaluation with precision, recall and F-measure, cross-validation, statistical significance testing, avoiding harms in classification, model interpretation and regularisation.

Chapter 5: Embeddings

Vector semantics is introduced through lexical semantics, count-based embeddings and cosine similarity, then word2vec (skip-gram) and GloVe. The chapter discusses visualisation, the semantic properties of embeddings (such as analogies), bias in embeddings, and evaluation.

Chapter 6: Neural Networks

Feedforward networks are built up from single units, motivated by the XOR problem, and applied to classification with embeddings as inputs; training with backpropagation on computation graphs is explained, along with neural language models.

Chapter 7: Transformers and Pretraining

The central architecture chapter explains attention and multi-head attention, transformer blocks and the residual stream, positional embeddings, the language-modelling head, decoding and sampling, pretraining of transformer LLMs, and current refinements such as rotary embeddings and the KV cache.

Chapter 8: Post-training

Post-training addresses limits of pretrained models: instruction tuning (supervised fine-tuning), parameter-efficient fine-tuning such as LoRA, and learning from preferences with methods like PPO and DPO for alignment, with reinforcement learning from verifiable rewards named as a further stage.

Chapter 9: Masked Language Models

Part II begins with bidirectional transformer encoders of the BERT family, trained by masked language modelling, and shows how to fine-tune them for classification and for sequence labelling such as named entity recognition, contrasting encoder, decoder and encoder–decoder architectures.

Chapter 10: Interpretability

A partial ("stub") chapter on mechanistic interpretability: contextual embeddings, probing, the logit lens, sparse auto-encoders, causal interpretability techniques, and in-context learning and induction heads.

Chapter 11: Retrieval-based Models

Headed "Information Retrieval and Retrieval-Augmented Generation", this chapter covers sparse retrieval with tf-idf, BM25 and inverted indexes, evaluation of retrieval systems, dense retrieval with embeddings, retrieval-augmented generation (RAG), question-answering datasets and their evaluation.

Chapter 12: Agents

In this draft the chapter is a placeholder with no content yet.

Chapter 13: Machine Translation

The chapter discusses language divergences and typology, the encoder–decoder transformer with cross-attention, beam search, low-resource translation (including backtranslation), evaluation of adequacy and fluency with human judgement and metrics such as chrF, and bias and ethical issues.

Chapter 14: RNNs and LSTMs

Recurrent networks are presented as an alternative way of modelling time: simple RNNs as language models and for other tasks, stacked and bidirectional variants, LSTMs that address vanishing gradients, and RNN encoder–decoders with attention, the historical route to the transformer.

Chapter 15: Phonetics and Speech Feature Extraction

The speech chapters begin with phones, the IPA and ARPAbet, articulatory phonetics, prosody, and acoustic phonetics (waveforms, spectra, spectrograms, formants), ending with feature extraction for recognition via the log mel spectrum and MFCCs.

Chapter 16: Automatic Speech Recognition

This chapter defines the ASR task and explains convolutional networks, the encoder–decoder architecture with attention, self-supervised models such as HuBERT, CTC for streaming recognition, and evaluation by word error rate.

Chapter 17: Text-to-Speech

Modern TTS is presented as conditional generation with a language model over discrete audio tokens produced by a neural codec (vector and residual vector quantisation), with VALL-E as the example; the chapter also covers TTS evaluation, other speech tasks and spoken language models.

Chapter 18: Sequence Labeling for Parts of Speech and Named Entities

Part III opens with word classes, part-of-speech tagging and named entity tagging, then the generative hidden Markov model tagger with Viterbi decoding and the discriminative conditional random field, with evaluation of NER.

Chapter 19: Context-Free Grammars and Constituency Parsing

Constituency is modelled with context-free grammars and treebanks; the chapter covers Chomsky normal form, structural ambiguity, CKY dynamic-programming parsing, span-based neural constituency parsing, parser evaluation and head-finding.

Chapter 20: Dependency Parsing

Dependency relations and treebanks are introduced, followed by transition-based (stack-based, greedy) parsing and graph-based parsing with maximum spanning trees, and evaluation with labelled and unlabelled attachment accuracy.

Chapter 21: Information Extraction: Relations, Events, and Time

The chapter covers relation extraction (patterns, supervised, bootstrapping, distant supervision and open IE), event extraction, representing time and aspect, the TimeBank corpus, automatic temporal analysis, and template filling.

Chapter 22: Semantic Role Labeling

Semantic roles are introduced from Pāṇini's kārakas to thematic roles, diathesis alternations, PropBank and FrameNet; the chapter then covers semantic role labelling algorithms, selectional restrictions and primitive decomposition of predicates.

Chapter 23: Lexicons for Sentiment, Affect, and Connotation

Building on sentiment analysis, the chapter defines emotion and other affective states, surveys sentiment and affect lexicons, and explains how lexicons are created by human labelling, semi-supervised induction and supervised learning, and used for sentiment, affect, entity-centric affect and connotation frames.

Chapter 24: Coreference Resolution and Entity Linking

The chapter explains coreference phenomena, tasks and datasets, mention detection, mention-pair, mention-rank and entity-based architectures (with feature-based and neural classifiers), entity linking, evaluation, Winograd schema problems and gender bias in coreference.

Chapter 25: Discourse Coherence

Local and global coherence are covered through coherence relations (Rhetorical Structure Theory and the Penn Discourse Treebank), discourse parsing, Centering Theory and the entity grid, neural models of local coherence, and global structure such as argument mining.

Chapter 26: Conversation and its Structure

A short initial stub on the properties of human conversation and dialogue acts, intended also as a tool for analysing human–LLM conversation.

Key Concepts

Chapters 1–3: LLMs, tokens and n-grams

• Large language model: a neural network trained to predict the next token, which can then generate text by sampling.
• Byte-pair encoding (BPE): a data-driven method that builds a vocabulary of subword tokens.
• Perplexity: the inverse probability of a test set, normalised by length, used to evaluate language models.

Chapters 4–6: Classification, embeddings and neural networks

• Cross-entropy loss and gradient descent: the objective and optimisation method used to learn classifier weights.
• Static word embedding: a dense vector for each word, learned for example by word2vec's skip-gram.
• Backpropagation: computing gradients through a computation graph to train neural networks.

Chapters 7–8: Transformers and post-training

• Self-attention and the residual stream: the mechanism by which each token's representation draws on other tokens, passed up through stacked blocks.
• Pretraining and decoding: learning from vast text by next-token prediction, then generating with sampling methods.
• Instruction tuning and preference alignment: post-training steps that make models follow instructions and align with human preferences.

Chapters 9–12: Advanced LLM topics

• Masked language modelling: training a bidirectional encoder to predict hidden tokens from context on both sides.
• Probing and sparse auto-encoders: interpretability tools for discovering what models represent.
• Retrieval-augmented generation: conditioning an LLM's output on retrieved passages to improve factuality.

Chapters 13–14: Translation and recurrent networks

• Encoder–decoder with cross-attention: mapping an input sequence to an output sequence of different length.
• LSTM: a gated recurrent network that decides what to remember and forget.

Chapters 15–17: Speech

• Log mel spectrum and MFCCs: acoustic features used as input to speech recognisers.
• CTC and word error rate: a loss for streaming recognition and the standard ASR metric.
• Neural audio codec: a model turning speech into discrete tokens that a language model can generate.

Chapters 18–20: Tagging and parsing

• HMM and CRF taggers: generative and discriminative sequence models decoded with the Viterbi algorithm.
• CKY parsing: dynamic-programming constituency parsing over a grammar in Chomsky normal form.
• Transition-based and graph-based dependency parsing: greedy stack-based parsing versus maximum spanning tree methods.

Chapters 21–23: Extracting meaning

• Distant supervision: training relation extractors from an existing database of relations.
• Semantic roles (PropBank, FrameNet): who did what to whom in an event.
• Affect lexicon: a list of words annotated for sentiment, emotion or connotation.

Chapters 24–26: Discourse and conversation

• Coreference chain: the set of mentions in a text that refer to the same entity.
• Coherence relations and Centering Theory: models of how sentences and entities make a discourse hang together.
• Dialogue acts: categories of what an utterance does in conversation.

Critical Analysis

Strengths

• Current and comprehensive. The August 2026 draft reorganises the book around LLMs—tokenisation, transformers, pretraining, post-training, masked models, interpretability and retrieval—while keeping chapters on speech recognition and synthesis and on classic linguistic annotation tasks, so one text spans both modern methods and the field's foundations.
• Builds from simple models. N-grams, logistic regression and feedforward networks are introduced before transformers, so concepts such as perplexity, cross-entropy loss, embeddings and gradient descent are in place when LLMs are explained.
• Linguistic and historical grounding. Chapters connect methods to linguistics (morphology, typology, phonetics, semantic roles, coreference, discourse) and most end with summaries and "Historical Notes"; Chapter 1 and several others address safety, bias and harms.
• Open access with teaching materials. The complete draft and individual chapters are freely available, with slides for many chapters and a set of web-only appendices (for example on HMMs, naive Bayes and WordNet).

Limitations

• An unfinished draft. In this release Chapter 10 is described as a stub, Chapter 12 (Agents) is a placeholder with no content, and Chapter 26 is an initial stub; several chapters (for example 6, 8, 9 and 14) have no exercise sets, and chapter titles in the text sometimes differ from those in the table of contents.
• Not the printed edition. The free text is the third-edition draft; the last printed edition, the second (Pearson, 2008), predates the neural and LLM methods around which this draft is organised. Readers using the 2008 print edition should expect different chapters and content.
• Mathematically demanding. The book assumes comfort with probability, linear algebra and calculus (gradients, softmax, matrix formulations of attention), which can be a steep climb for linguistics students without that background.
• Moving target. The material changes between releases (the site lists earlier January 2026 and August 2025 drafts), so page and chapter references may shift, and some sections on current models will age quickly.

Where It Fits in the Course

In the Computational and Corpus Linguistics list, Jurafsky and Martin is the comprehensive, technical NLP text. Language and Computers gives a non-programming conceptual introduction to many of the same applications, and Bird, Klein and Loper's Natural Language Processing with Python builds hands-on skills with tagging, parsing and classification; both are good preparation, and this draft then supplies the algorithms and the neural and LLM-based methods they do not cover.

The corpus-linguistics titles—Doing Corpus Linguistics, McEnery and Hardie's Corpus Linguistics and Gries's Quantitative Corpus Linguistics with R—serve a different aim: analysing language use with corpora and statistics rather than building language-processing systems. Students in a corpus-focused course may use only selected chapters here (for example 2, 3, 5 and 18) as background on tokenisation, n-grams, embeddings and tagging.

How to Use This Book

The book suits advanced undergraduates and graduate students in computational linguistics, computer science or data science who have some programming experience and basic probability, linear algebra and calculus. A core sequence is Chapters 1–8 (tokens, n-grams, classification, embeddings, neural networks, transformers, post-training). After that, choose by interest: Chapters 9–11 and 13 for advanced LLM topics and translation; Chapters 15–17 for speech; Chapters 18–25 for linguistic structure and meaning. Chapter 14 (RNNs and LSTMs) is useful background but no longer a prerequisite.

Use the chapter summaries and historical notes for review and the exercises where provided; the authors' site offers slides and the web-only appendices on topics such as HMMs and naive Bayes. Because the draft is updated regularly, record the release date you are using and check the site for later versions.

Edition and Sources

This guide is based on the third-edition draft of Speech and Language Processing dated 19 August 2026, read in the complete PDF from the authors' official site, https://web.stanford.edu/~jurafsky/slp3/. It is not the second edition (Pearson, 2008), which is not freely available. The guide draws on the site's release notes and appendix list, the summary of contents and full table of contents, and for all 26 chapters the opening pages and, where present, the end-of-chapter summaries, together with selected sections of Chapters 1, 7 and 23 and a count of exercise and historical-notes sections.