skip to content

Department of Psychology

Postgraduate Research Methods Class

Prof Mark Haggard

Thursdays 4-5pm from 19 January to 23 February 2023. 

The last dozen years have under the banner of ‘Replication Crisis’ seen a resurgence of concern for appropriate scientific method in Psychology and similar sciences, following similar soul-searching in Medicine (EBM) 20 years before. The emphasis in both movements has been avoidance of biased Type 1 errors. How exactly do we now do better? This is a conceptual rather than nuts-and-bolts prescriptive ‘how-to’ course, although it uses practical illustrations of typical analysis problems faced. For some highly specialised techniques eg multi-level modelling (MLM). It conveys awareness of the reasons for particular potential of, and constraints met by, the particular technique, but does not for time reasons, go into all its details. If you get the concepts right, including pervasively important concepts like the df-ratio, you can develop the orientation and motivation to acquire such techniques using software support, textbooks and articles.

You’ll have noticed that the word ‘statistics’ appears here only in the title of the recommended book below. The statistical underpinning for methods is of course crucial but the mis-labelling of these topics as ‘statistics’ has had unfortunate effects. Some of what is needed is close to philosophy of science. We will redress this balance.

The ‘Methods Course’ of the title is the focus, but the totality includes four further related ingredients, which can occur throughout the year, but the first may best occur immediately.

  • Remedial preparation. If your Undergraduate course has not been very strong in methods, you need to assess your basic knowledge and to plan for remedial reading in the 1st term. I suggest that you look through the elementary but useful Statistics Explained 2nd Edition by Steve McKillup, Cambridge University Press. 2012.   You need not be able yourself to teach all the techniques covered in this book or explain the formulae, but they issues and techniques ought all to feel elementary and familiar to you. If not, act now. I’m available (mph38@cam.ac.uk) to discuss what remediation reading should help and will be in the Department most Wednesdays and Thursdays and available to discuss this.

 

  • Detailed discussion of analysis strategies and protocols before data are acquired. An experiment or observational study is a formal structure with principles, even rules, for analysis and consequent interpretation. There are also principles for design to secure the ‘best’ data to answer the research question, which are not usually called ‘rules’. Of course these principles have several aspects and are subject to trade-offs.  Invoking ‘experience and intuition’ is how we void having to specify things that are hard to formulate precisely, and judging what will be ‘best’ is one such matter. The most pathological avoidance is failure to reflect in advance on how data should be analysed, but this is improving with adoption of a priori protocols.

 

  • Review of and advice on analyses in progress. A provisional answer to a question usually generates more questions. What do the interim results mean? What variables should be included or eliminated from the analyses to be reported? What pre-assumptions need to be checked?   Few general rules round such questions are widely familiar, but some of those ‘known’ can be at best mis-targeted, at worst wrong. For example:  for parametric analyses, normality of raw variables is in fact not crucially important, and normality of residuals, whilst generally desirable is mis-sold as crucial to literal interpretation of the p-value (which should not be the focus anyway), and it is not as important as the (under-played) equal-interval properties of measurement. Wrong-turnings can be taken in course of analyses when these principles re mis-applied. The course includes explanations of why what is wrong is wrong in such examples.  I also have a set of prepared notes on special methods issues such as Transforming, and will circulate a list of such notes shortly. I tend to circulate particular notes around the time a topic is raised in class, but these can be asked for at any time

 

  • Quality monitoring of Internet resources. Assuming that you typically develop good strategies for generating productive sets of search terms, rapid searchability makes the internet a marvellous resource for answering ’What’s this’ or ‘What’s next’ types of question about analysis methods. Both types of search are relevant here: journal article keywords/titles and general topic all-net searches including ’grey’ literature’. The lack of curation is both a virtue and a vice. Pieces of information or advice can conflict, and mostly they do not come documented as to their credibility or adoption in conventional circles. I am happy to comment on what you may find, suggest alternatives etc.
Class 1: Thursday 19 January 2023 at 4pm in Psychology Classroom, Downing Site

Reasons for the Replication Crisis: bad assumptions and motivations. Primacy of Effect size not ‘significance’ for interpretation. Balancing Type 1 errors with Type 2. Multiple senses of ‘power’ and ‘effect size’. Honesty about where weaknesses lie. Introduction to multiple testing and adjustments for it. Beware circularity and magical thinking.

Class 2: Thursday 26 January 2023 at 4pm in Psychology Classroom, Downing Site

Principles of practice in acquiring and handling data. Pre-listing confounders. Checking and cross-checking. ‘Preliminary analyses’: their use and reporting. When must the operations yielding a measure be rigorously respected? Transformation versus bootstrapping. How much detail: text, table footnotes, appendix.

Class 3: Thursday 2 February 2023 at 4pm in Psychology Classroom, Downing Site

ANOVA as the basis of most statistical analysis. Maximising the between- to the within- variance ratio.  Classical experimental design as special case of confounder minimisation within multiple regression. What is the relevant error term to report, when there could be several? Interactions ad non-linearities: when can you ‘afford’ to address these and when not?

Class 4: Thursday 9 February 2023 at 4pm in Psychology Classroom, Downing Site
The Modelling Approach, eg multiple regression as the expression of what analyses should be aiming to conclude. Multiple senses of the word ‘model’; trade-offs in model choice‘. Succinct presentation and comparison of alternative models as exemplifying openness.
Class 5: Thursday 16 February 2023 at 4pm in Psychology Classroom, Downing Site

Non-parametric methods: moving on from Chi-sq and Wilcoxon or Mann-Whitney Categorical dependent variables: logistic and polychotomous/multinomial regression.

Class 6: Thursday 23 February 2023 at 4pm in Psychology Classroom, Downing Site

 Structural equation modelling (SEM) to strengthen causal inference via graphical flow models. More than one type of goodness of fit. Elegant and demanding, but can it sometimes conceal banality?