Worksheet 03: Critical Analysis - Level up
Methods and results
Critical Proposal preparation
Please read the Critical Proposal Coursework Briefing document. This is important information and the accompanying Rubric will give you all the information you could possibly need.
It is compulsory to include the coversheet information. You can just copy and paste the content onto the front of whatever document you submit, but the most important feature is a FULL APA format reference (including direct link) to your Target paper.
It is ALSO COMPULSORY to include:
An effect size for at least one of your IVs
This must be in the Design Schematic image - editable in www.draw.io using the template file in the Coursework Information section of the VLE
DesignSchematic
The only other thing to bear in mind is that the selection of a ‘good paper’ is part of the assessment! But you are strongly advised to show it to your Lab Tutor for immediate feedback on screen. NOT VIA EMAIL.
Get the methods section of your Target Paper on screen!
Up to this point, you’ve likely read a few methods sections.
Maybe it hasn’t fully occurred to you yet, but you will be relying on this information HEAVILY!
I believe this is a dramatic change in perspective. And one that forces a re-appraisal of quality.
It’s ok if you don’t fully understand any highly technical aspects, such as fMRI procedures or any descriptions of delivery or stimulus presentation, such as computer programmes or survey software (that’s for week 6!)
Think more carefully about each of the following aspects and jot down some ideas for your Critical Proposal:
1. Design of the study
2. Participants and recruitment
3. Materials
4. Procedure
Design
How easily can you identify the design?
It may not be a fully experimental design. It may be quasi-experimental or even correlational.
What is the design? What are the IVs?
Are these between- or within-subject? Or a mix?
What is/are the DV(s)?
Participants
Who took part in the study?
Numbers, background, age range and demographics?
Where did they come from?
How heterogeneous a sample is it?
Do you see any problems with the sample that might impact the results?
Does this sound like a ‘high-quality’ sample?
Does this sample appear to generalize to the entire population of interest? Were there any exclusion criteria? Or inclusion criteria? Was there any deception or information about the study retained in recruitment? This may only come to light in the procedure section. Was this necessary?
If you had to replicate this exact sample, how easy would that be to do? If you could make the sample ‘better’ in any way, what would you do to achieve this? Does the paper mention how participants were allocated to condition or ‘categorised’.
Did the researchers perform a Power Calculation to estimate a suitable sample size? We will be talking about Power Calculations next week.
Materials
You will need ‘materials’ to run your study. Many of you will build upon the tools used in the papers you critique.
Now is an excellent time to start looking at these in detail.
Does the research use any questionnaires or psychometric measures? Can you identify these?
It’s always important to carefully consider the heritage and development of any questionnaires you use, either as a means of categorizing participants into groups (e.g. based on Anxiety).
In future lectures we go in to more detail on Psychometrics and Scale design, so thinking about this is useful.
Are there any tasks employed? How are they designed and delivered?
Is any stimulus material described?
If music or faces are used, where are these sourced, and how are they selected?
Would you be able to replicate all the necessary stimulus materials? Can you be confident that you have similar stimulus sets after this replication process?
Are you able to, for example, correctly replicate a stimulus set of Happy and Sad music? Or Angry and Fearful faces?
Can you find enough to build the entire experiment?
Are any of the materials piloted to make sure they work? Are there any manipulation or attention checks? What about awareness of research hypotheses? Could a participant identify these?
Is there any specialist equipment involved? Is there anything particular about the physical space employed if testing f2f?
This may be relevant for next year. It is a good idea to think about the challenges of running a study online. Can you capture the appropriate data? If your chosen study includes anything like an intervention, or presentation of music or visual stimulus, how confident are you this is done well?
When your mark is awarded for the Final Year Dissertation, all the decisions you make to ensure quality and accuracy of data will count towards that grade.
Procedure
Was there room for the experimenter to exert any bias? Was there consistency between experimenters or confederates? How was this achieved?
What is the order of the various parts of the study? Is this the same for all participants? Are any elements counterbalanced? If so, how?
If you had to estimate how long the procedure lasts, how long would it be? Do you get a sense that participants were looked after properly? Do you get a sense that optimal performance was obtained from the participants? Thinking about the duration and complexity of the task, to what extent is fatigue an issue?
Is there a chance that performance may change in other ways over time? If a control condition was used, how closely did this match the experimental condition? If data are collected in a way dependent on the participant (e.g diary study, “over the course of a week” between sessions), how reliable is the resultant data?
Closing considerations
You have taken part in a few experiments yourself.
The chances are, you were an excellent participant. How do you think the majority of participants may respond? How closely do you think they follow instructions?
Running studies online can be tricky, but can offer new opportunities. Can you think of any obstacles? What about potential benefits of online research?
You won’t be able to pay or incentivise participants to take part in your research. How are you going to get the best performance out of them? How would you like to be treated as a participant in your first experiment?