Research Methods Articles

Crano & Lak (2012) The evolution of research methodologies in social psychology

  • This article highlights key developments shaping the field: rise of the experimental method, Fechner’s quantification of mental events, correlation, introspectionism, randomization, critiques of randomization, crises in psychology, and the move from the laboratory to the field

Rosenthal (1994) Science and ethics in conducting, analyzing, and reporting psychological research

  • Poor research design is unethical. It’s a waste of time and money and leads to poor results.
  • Avoid data dropping, outlier rejection, subject selection, misrepresenting findings (intentional, unintentional, questionable generalizability), hyper-claiming (saying research will achieve more than it can) and causism (implying a causal relationship where none has been established).
  • IRB should consider scientific competence of researchers and adopt cost-utility analysis
  • Failure to report is an ethical issue. Data snooping is good so you get money’s worth from the research. Meta-analysis is an ethical imperative

Rosnow (1997) Hedgehogs, foxes, and the evolving social contract in psychological science

  • Researchers can use the restrictions of an IRB to exploit boundaries and find more creative ways of doing research; they should look at IRB as an opportunity
  • Researchers have a responsibility to do research that matters
  • Pointless research is a waste of participants’ time
  • Social contract = duty to avoid harm while producing useful results
  • IRBs don’t focus on cost of not doing research
  • Hedgehogs = social scientists of the past who could be single minded in pursuits and interests
  • Foxes = social scientists of today who have to be clever about working around obstacles

Lipsey (1991) Social responsibility, mentoring, and research training

  • Social responsibility is more than simply training students in scientific integrity and not to lie, cheat, or steal. It means acknowledging that behavioral science is a normative endeavor with sociocultural biases and social impact, differing from physical sciences.
  • Those in behavioral sciences should train and mentor students entering the field
  • Graduate training should include: theoretical discourse (implicit assumptions of descriptive statements), practical/normative discourse, meta-theoretical discourse

Martinson, et. al (2005) Scientist behaving badly

  • Misbehaviors represent greater threats to the scientific enterprise than those caused by high profile misconduct cases

Gholson & Barker (1985) Kuhn, Lakatos, and Lauden

  • Kuhn: talked about scientific revolution – calm then chaos, there are big shifts. But things don’t change. Old scientists just die and new scientists with new theories take over.
  • Lakatos: Theories are linked by a common hard core (shared commitments) with a protective belt (dispensable hypotheses). New theories must address old + new evidence
  • Lauden: An evolutionary approach – science progresses and we don’t need to unify the theories. They can co-exist (ie: behavioral vs. cognitive psychology)

Gergen (1973) Social psychology as history

  • Doesn’t believe social psychology is science; theories are unstable and are simply reflections of contemporary history. Knowledge of social interaction can’t be scientifically accumulated.
  • Social psychology alters the behavior it seeks to study in 3 ways: (1) Prescriptive bias (subjects are influenced by the researcher); (2) Enlightenment effect (science and society = feedback loop); (3) Knowledge of social research changes behavior (people resent psychological theories and react recalcitrantly)

Schlenker (1974) Social psychology and science (response to Gergen)

  • Says that theoretical principles are not culturally and temporally bound because (1) they’re abstract; (2) just because we can’t find regularities doesn’t mean they don’t exist (speaks more about our ability); (3) many similarities exist between all known societies but the very question of whether hypotheses are transcultural is empirical question; (4) ability to predict the future is rarely realized in any open system
  • Argues against the enlightenment effect: (1) information is withheld during experiments to control variables; (2) unforeseen behaviors can usually be explained by the same/another set of hypotheses (suicidal predictions and self-fulfilling prophecies); (3) Gergen’s suggestion that you cannot have a universal proposition is in fact a universal proposition; (4) Whether people change behavior because they gain more information is an empirical question

Wallach & Wallach (1993) Gergen versus the mainstream: Are hypotheses in social psych empirical?

  • Social psychology theories can be empirically tested because of: (1) converging operations (multiple sources of evidence increases confidence); (2) additional background knowledge (people will not deceive unless incentivized); (3) predictive/explanatory power
  • Near tautologies (cannot be disconfirmed): Yes, it is true that some social psychology propositions are derived from near-tautologies. Any experiment testing near-tautologies has limited value and should be avoided. However, testing near-tautologies has benefit when there’s minimal manipulation, unexpected effects, or produces effects under circumstances that are practically significant.

Mook (1983) In defense of external invalidity

  • The goal of science is to understand. External validity not necessarily important.
  • We don’t care if we can generalize or not because (1) we may be asking whether something can happen, not if it does; (2) we may be predicting about something that happens in the lab; (3) we may demonstrate the power of a phenomenon by showing it could happen, even when unnatural conditions should preclude it; (4) lab may produce conditions that have no counterpart in real life anyway.

Locke (1986) Generalizing from the laboratory to the field

  • Says Mook missed the point. The real argument: lab can’t be generalized because it’s too different to the real world and there’s no definition for ways the environment must be similar
  • Only the essential features need be replicated for generalizability; not known in advance.
  • Lab should (1) identify essential features; (2) assess whether these functions support generalizability

Orne & Scheibe (1978) …factors in the production of sensory deprivation effects

  • Students in a sensory deprivation room with a panic button and a first aid kit had a worse experience than those who didn’t have these cues. Bias is not conscious.
  • Demand characteristics (social cues) interfere with results.

Parsons (1978) What caused the Hawthorne Effect?

  • The basis for the Hawthorne Effect came from the Relay Assembly Test Room experiments. Original results claimed factory workers increased productivity no matter what the researcher did (turning lights high or dimming). However, revisiting this experiment we see confounds in experimental procedures. No control, no baseline established, frequent feedback given, pay incentives, production rates increased with worker practice, workforce changed, etc.

Langer & Newman (1979) The role of mindlessness in a typical social psych experiment

  • Speaker was introduced to class and a posttest was given to on speech’s content. Those who were mindless (performed poorly on the test) were more responsive to the social cues given in the introduction than those who were mindful.

Campbell & Fiske (1959) Convergent and discriminant validation by the MTMMM

  • Measures convergent validation (degree constructs correlate with each other) and discriminant validation (degree they do not correlate) – best to use at least two methods and two traits
  • Diagonal lines measure validity; values within the triangles measure reliability
  • Four aspects of MTMMM need to exist for a test to be valid: (1) MTHM (diagonal line) must correlate; (2) MTHM should have higher correlations than HTHM; (3) MTHM differences should exceed MTMM differences; (4) Same patterns should be shown in all heterotrait triangles of monomethod and heteromethod

Crano (2000) MTMMM and Campbell’s views on the proper conduct of social inquiry

  • Reviews MTMMM and adds these unstated assumptions: all measurements methods are equally reliable, trait and method factors are uncorrelated, all traits are identically affected by any given method of measurement, no association exits among the various methods factors
  • After 40 years of exploration, even Campbell and Fiske are pessimistic about whether a really good matrix is even possible, but this model sets a good vision for triangulation, multiple operationalism of measures whose relationship is known, recognizing validation is a process.

Crano & Messe (1985) Comprehension artifact

  • Messe (former student of Crano’s) urged customers to reduce electricity usage through dense reading material, however, it didn’t work. Crano and Messe decided that the comprehension artifact generated a Type II error
  • Treatment knowledge check assesses (1) whether participants understood the treatment; (2) if they were aware of appropriate response. Note: this is different than a manipulation check (which seeks to establish that the treatment has been received)

Baron & Kenny (1986) The moderator-mediator variable distinction

  • A moderator influences the strength of a relationship; mediator explains the relationship between two variables. Example: annual health checkups and social class. Age may be a moderator – relationship is stronger for older people. Level of education may be a mediator – explaining relationship between annual health checkups and social class.
  • Lots of letters in the article: S (stressor), P (perceived control), O (outcome), blah blah blah

Greenwald, et. al (1986) Under what condition does theory obstruct research progress?

  • Disconfirmation dilemma arises when one’s theory isn’t confirmed. Researcher can (1) discard the theory or (2) persevere in testing it (risk of theory testing morphing into theory confirming)
  • Theory obstructs research progress when testing theory is the goal of research. In this instance, the researcher believes in the theory more than the means and methods used to test it. Can be prevented with results-centered approach: (1) condition-seeking (reduces the generalizability of a finding to produce a qualified conclusion) and (2) design approach (attempts to find conditions in which a “presently unobtainable result” can be produced). Yes, theory still plays a role in these approaches, but that they will produce results much faster.

Wicker (1989) Substantive theorizing

  • The substantive theorizing approach emphasizes consideration of concepts and substantive domain before methodology.
  • Social, spatial, and temporal context need to be given more attention
  • Use multiple methods

Ableson (1995) Statistics as a principled argument

  • Don’t judge in isolation – we need p-value and confidence interval.
  • Effect size: small effect on a large does is unimpressive; small effects in small manipulations is impressive

Cohen (1994) The earth is round (p<.05)

  • In null hypothesis significance testing (NHST), we want to know the probability the null is true given data we have obtained. However, what we’re really doing is learning the probability of obtaining our data given the null is true. Bayes theorem is the only way to determine the inverse probability of a conditional probability. To use this theorem to determine the probability of the data set given the null hypothesis, we need to know the probability of the null occurring. We never know this.
  • Problems with NHST: common misperception of p-value, misuse of conditional probability, false belief that if you reject the null then alternative is true, with a large enough sample the null will be false, no demonstration of the magnitude of the direction, coefficients vary from one population to another
  • Ways to correct problems: (1) theories should be tested by attempts to falsify them (Popperian ideal); (2) should include effect size and confidence interval; (3) graph data to understand it

Cortina & Dunalap (1997) On the logic and purpose of significance testing (in response to Cohen)

  • Good experiments are (1) objective (set cut-off values before collecting data) (2) exclude alternative hypotheses (3) exclude alterative explanations (ie: sampling error, done by setting arbitrary p-value)
  • CI and NHST require the same types of info, so replacing one with the other doesn’t make sense. Not the measurement that’s the problem; it’s the interpretation.

Bakeman (in Reis & Judd, 1st Ed.), Behavioral Observation and Coding

  • When coding behaviors, coders need to: (1) detect episodes and (2) code them
  • The agreement between two behavioral coders is marked by Cohen’s (1960) kappa. A ratio of consistency of finding vs. consistency we expect to find. It varies from zero (no agreement) to one (perfect agreement). Kappas of .40 to .60 are fair; good is .60-.75; and over .75 is excellent.

Crano, et. al (2008) The at-risk adolescent marijuana nonuser

  • This experiment studied marijuana in youth in terms of resolute non-users vs. vulnerable non-users. However, for purposes of this class, the point of the article was to discuss using data from a secondary source and how to establish a classification system
  • There are big resources of data out there – using it for secondary data analysis can save money

Hajebi, et. al (2012) Telephone versus face-to-face administration

  • Phone surveys have been shown to have limitations and to be more subject to social desirability bias than face-to-face interviews, but this experiment of psychiatric outpatient services in Iran showed that phone interviews could serve as a suitable replacement for in-person interviews
  • However, this was a follow-up phone call, not an initial phone call. And the person doing the phone call was the same person doing the in-person interview, so very well trained

Yeager & Krosnick (2010) The validity of self-reported nicotine product use

  • Upshot: people don’t lie on self-report interviews about nicotine use
  • There’s a theory that social desirability bias may interfere with self-reports in interviews. Though studies have shown this is not true in regard to nicotine use, this research had been conducted prior to 1994 and the anti-tobacco push. This experiment had people self-report nicotine use then they took blood samples.
  • This research showed that 4.5% underreported use. After correcting for previous delay in collections that percentage dropped to 1.17%. After correcting for secondhand smoke exposure, the percentage dropped to only .89%.

Rosenthal, (1979) The “file drawer problem” and tolerance for null results

  • By definition, 5% of articles have Type I Errors, but these are over-represented in the set of published articles
  • Sobering lesson: if an accepted finding is marginal, then a small numbers of file drawer studies could sway the conclusion
  • Cheering lesson: as the number of studies supporting a finding grows, the potential for file drawer reversals declines

Hall, et. al (1994) Hypotheses and problems in research synthesis

  • In research synthesis (meta-analysis), we learn from combining or comparing studies
  • Boundaries to consider (1) cause and effect (make sure experiments had high internal validity, (2) generalization, (3) theory development (synthesize to inform theory)
  • Moderators can be: low inference (setting, etc. of original study) or high interference (added by meta-analysis)