# Social research methods: choose a design and audit a measurement — Study worksheet [Study guides](https://unireadinglist.com/guides/social-research-methods/) [Course topic: Research Methods](https://unireadinglist.com/subjects/sociology/#course-research-methods) Sources checked: 2026-11-04 Study route and exercise: University Reading List editorial project. A first structured methods route for readers able to follow a social-science argument and elementary proportions. Bhattacherjee is a free graduate primer, Creswell/Creswell a sixth-edition proposal guide, and Salganik a first-edition digital-research companion. They do different jobs. The invented coding cases teach a defined nominal-agreement statistic; no fixed threshold is offered as a universal quality standard. ## Choose your starting point - [ ] I need a free methods foundation Use Bhattacherjee for the chain from constructs and theory to design, measurement, sampling and analysis. Supplement technique-specific detail when a project requires it. - [ ] I need to turn a question into a proposal Use Research Design when comparing qualitative, quantitative and mixed approaches and writing a coherent plan are the immediate tasks. - [ ] My question involves digital traces or platforms Use Bit by Bit after the basic design question is clear. Its historical examples are not instructions for current API access or permission to collect people’s data. ## Compare the three books ### Social Science Research: Principles, Methods, and Practices Anol Bhattacherjee · 2 · 2012 Free compact methods spine Choose it when: You need a connected map of concepts, measurement, design and inference. Use another route when: You need a full software/statistics course or specialist qualitative analysis from one short primer. Selected reading: Second edition: the theorizing, design, measurement, reliability/validity and sampling sections; locate the actual headings in the official text. Limits and trade-offs: USF identifies a2012 graduate-oriented text and a noncommercial share-alike licence. “Beginner” here means beginning research methods, not no academic reading or statistics preparation. #### Try before choosing Choose an invented construct such as “response offers a reason”; write a coding rule and two borderline examples. Check your work: The observable category is distinct from the construct. Explain where coder judgement enters and what a repeated classification could fail to establish. Your notes: ________________________________ ISBN: 9781475146127 [Official free reading](https://digitalcommons.usf.edu/oa_textbooks/3/) [Book record and access options](https://unireadinglist.com/subjects/sociology/social-science-research-bhattacherjee/) ### Research Design: Qualitative, Quantitative, and Mixed Methods Approaches John W. Creswell; J. David Creswell · 6 · 2023 Proposal and approach comparison Choose it when: You need a research question, purpose statement and method to fit together. Use another route when: You expect brief methods chapters to replace full technique training or assume mixed methods means merely having two datasets. Selected reading: Sixth edition: 1,4 and 6–10 for approaches, ethics, purpose, questions and the three method families. Limits and trade-offs: The publisher verifies the ten-chapter structure. It is a design/proposal book; a polished proposal alone does not show a measure is reliable or valid. #### Try before choosing Draft a purpose statement and then one design for a coding question; state which conclusion that design can and cannot support. Check your work: The sampling unit, classification rule and inference match the question. A causal claim needs more than a two-coder agreement table. Your notes: ________________________________ ISBN: 9781071817940 [Book record and access options](https://unireadinglist.com/subjects/sociology/research-design-qualitative-quantitative-and-mixed-methods-approaches/) ### Bit by Bit: Social Research in the Digital Age Matthew J. Salganik · 1 · 2017 Free digital-research comparison Choose it when: You want to distinguish observing, asking, experimenting and collaborating in digital settings. Use another route when: You need current platform instructions or a first glossary before choosing a research approach. Selected reading: First edition: 1.5 for the map,2–5 for contrasting designs, and 6 for ethics. Use the first-edition route rather than the older Open Review version. Limits and trade-offs: The author emphasizes enduring research abstractions. Historical platform examples do not establish present access, consent or reuse rights; some activities need additional coding or mathematics. #### Try before choosing Use the author’s four-design map to propose two ways to investigate one invented question, and an ethical issue each creates. Check your work: The designs change the researcher-participant relationship and the type of inference. Public availability alone does not settle whether collecting or combining records is appropriate. Your notes: ________________________________ [Official free reading](https://www.bitbybitbook.com/en/1st-ed/preface/) [Book record and access options](https://unireadinglist.com/subjects/sociology/bit-by-bit-salganik/) ## Check your preparation ### What counts as a unit? Your notes: ________________________________ ### Does agreement prove validity? Your notes: ________________________________ ### What mathematics is used below? Your notes: ________________________________ ## Six steps with a concrete output - [ ] Define the question and construct Selected reading: Bhattacherjee’s theory/measurement sections. Make this: Write a construct, one operational rule and a deliberately ambiguous invented paragraph. Check your work: A reader can say how the rule assigns a category and where it leaves judgement. Your notes: ________________________________ - [ ] Choose an approach Selected reading: Research Design 1 and 6–10; Bit by Bit 1.5. Make this: Compare two possible designs for the same question with their units, outputs and limits. Check your work: The design follows the inference sought, not merely a preferred software package. Your notes: ________________________________ - [ ] Make a small codebook Selected reading: Measurement and reliability discussions. Make this: Write mutually exclusive, exhaustive A/B rules for the supplied fictional items; keep a disagreement log. Check your work: Rules can be applied independently before discussion; reconciliation is not presented as independent initial agreement. Your notes: ________________________________ - [ ] Audit the first table Selected reading: The original worked example below. Make this: Compute margins, observed agreement, baseline agreement and κ. Check your work: Every proportion has a stated denominator; no unprovided true label is invented. Your notes: ________________________________ - [ ] Test the apparent improvement Selected reading: The follow-up and the distinction between descriptive statistics and evidence. Make this: Compare a new table’s raw agreement and κ with the first one. Check your work: Changing item composition/margins prevents a simple claim of coder improvement. Your notes: ________________________________ - [ ] Write a limited methods memo Selected reading: Bit by Bit 6 and your proposal. Make this: State sampling, coding, agreement and validity limits; list the next evidence needed before real research. Check your work: No fictional example is reported as a participant observation, and no actual recruitment or scraping is performed. Your notes: ________________________________ ## Worked exercise: Two coders and 80 invented paragraphs Both coders independently classify the same 80 invented paragraphs as A or B. Counts are A/A 36, A/B 4, B/A 12, B/B 28, where the first label is coder 1. Compute raw agreement and the stipulated unweighted Cohen κ, then state what is not established. Assumptions: There are no missing classifications, and A/B are nominal, exclusive and exhaustive. The table is constructed, not collected. Define po as diagonal proportion and pe as the sum of products of the two coders’ category proportions. No true category or random sample is supplied. Your notes: ________________________________ ## Change an assumption: Higher raw agreement can accompany lower κ Use a separate invented batch of 80 paragraphs with counts A/A 0, A/B 4, B/A 4, B/B 72. Retain independent classification, the same nominal categories and the same κ definition. No claim of a before/after intervention is supplied. Your notes: ________________________________ ## Before moving on - I can connect a research question, construct, unit and proposed observation. - I can reproduce both coding summaries and explain why they are not validity tests. - I can choose a main text by the task I need next and state one methodological training gap. ## Answer and interpretation ### What counts as a unit? Specify whether the unit is a person, post, paragraph, event or organization; changing the unit changes the denominator and often the claim. ### Does agreement prove validity? No. Two coders can agree on the same poor rule. Reliability concerns consistency; validity concerns whether the measure supports its intended interpretation. ### What mathematics is used below? A two-by-two count table, proportions and κ=(po−pe)/(1−pe). Here pe is the product-marginal baseline, not a demonstrated psychological mechanism of random guessing. ### Two coders and 80 invented paragraphs 1. Observed agreement is(36+28)/80=0.80. Coder 1 assigns A to 40/80 and B to 40/80; coder 2 assigns A to 48/80 and B to 32/80. 2. pe=(40/80)(48/80)+(40/80)(32/80)=0.50. Thus κ=(0.80−0.50)/(1−0.50)=0.60. 3. There are 16 disagreements: 4 in one direction and 12 in the other. Without a reference truth or evidence about the codebook, this does not identify which coder is correct or whether the intended construct is measured. Raw agreement is 80% and κ=0.60 under the specified baseline. These are descriptive agreement summaries of this invented set, not accuracy, construct validity or a universal pass/fail rating. Test the result: Recover both margins and 80 from the four cells. Changing to a reconciled consensus table would change the question; it must not be called the initial independent agreement. ### Change an assumption: Higher raw agreement can accompany lower κ 1. Raw agreement is 72/80=0.90, larger than the first table’s 0.80. Each coder assigns A to 4/80=0.05 and B to 76/80=0.95. 2. pe=0.05²+0.95²=0.905. Therefore κ=(0.900−0.905)/(1−0.905)=−1/19≈−0.05263. 3. Most items are jointly labelled B, but there is no joint A. The margins and item composition changed, so raw agreement and chance-adjusted agreement need not rank the tables identically. The second table has 90% raw agreement and negative κ under this definition. Neither value alone demonstrates a better coder or valid measure; a label-prevalence shift can change the comparison. Test the result: Check 0+4+4+72=80 and 0.905\>0.900. Do not use a generic κ adjective as the conclusion; state the table, baseline and inference. No health, personality or social-group classification is performed. ## Common mistakes ### Treating a construct as its indicator The label is an operational choice that requires justification, not the concept itself. ### Making a percentage into truth Agreement, accuracy and validity answer different questions; name the evidence each would need. ### Treating digital access as permission A technically accessible source does not settle research ethics, consent or contextual expectations. ## Sources and edition checks - [USF: Social Science Research record and licence](https://digitalcommons.usf.edu/oa_textbooks/3/) USF verifies the 2012 free text’s graduate audience and CC BY-NC-SA3.0. The second-edition identity is retained; the route uses named sections, not a newly invented chapter map. - [SAGE: Research Design sixth-edition contents](https://www.sagepub.com/shop/buy-a-book/research-design-6-270550) SAGE’s sixth-edition contents verify chapters 1–10 and the proposal-oriented comparison of approaches. No full-book review or method effectiveness is claimed. - [Salganik: first-edition outline](https://www.bitbybitbook.com/en/1st-ed/introduction/outline/) The author’s first-edition outline and ethics chapter distinguish the four research approaches and their responsibilities. The 2017 publication/2018 copyright distinction is retained. - [Penn State: nominal agreement and kappa definitions](https://online.stat.psu.edu/stat509/Lesson07) Penn State’s university teaching material gives nominal-agreement definitions. The original Cohen 1960 paper is identified in the research note; the two tables and all calculations are original and no clinical example is reused. These are proposed study activities, not official course assignments or a claim that every book has been read cover to cover. [How we curate](https://unireadinglist.com/editorial/)