20 Design Feedback
20.1 Review and provide feedback on your research design.
ChatGPT’s value doesn’t end at generating experiment stimuli or performing power simulations. It can also provide insightful review and feedback on your research designs, further enhancing the quality of your studies. By offering an alternative viewpoint, this AI tool can help uncover potential pitfalls, suggest improvements, and ensure your design aligns with your research objectives.
Let’s consider an experiment inspired by Eryn Newman’s ‘truthiness’ research. Newman presented various claims to participants, such as ‘Giraffes are the only mammals that can’t jump’, and asked them to discern the veracity of these statements. Interestingly, she found that participants were more likely to believe a claim if it was accompanied by a photograph.
In this experiment, we aim to extend this effect by using videos, photos, or quotes of individuals making these claims instead of relying on generic stock footage or images. We are going to test three different conditions:
Video: Participants will watch a video of an alleged expert making the claim.
Photo: Participants will see a still image of an alleged expert making the claim along with a quote.
Quote: Participants will read a quote from an alleged expert making the claim, but without any accompanying media.
Each participant will be presented with 30 different claims (half true, half false). A third of these claims will be presented via a mock news report (video), a third will be presented as a quote next to a still frame from this report (photo), and the final third will be presented as a standalone quote. In this repeated-measures design, each participant is exposed to all three levels of the ‘Presentation’ variable: video, photo, and quote.
Now, how can ChatGPT assist in refining this research design? Simply give it all of this information and provide the following prompt.
20.1.1 Prompt
We're about to run an experiment that builds on Eryn Newman's 'truthiness' work. She presented a bunch of claims, like Giraffes are the only mammals that can't jump, and asked people to decide whether the claim is true or not. Again and again, she found that if the claim was accompanied by a photograph, people were more likely to believe it. We're going to extend this effect by using video, photos, or quotes of people making these various claims rather than using stock footage or photos. We'll also test the generalisability of the truthiness effect with two domains: trivia and COVID-19 claims. We'll compare the following three conditions:
Video: Video footage of an expert making the claim.Photo: A still image of an expert making the claim alongside a quote.Quote: A quote from an expert making the claim.
We'll present each participant with 60 different claims (30 per domain, half are true and half are false), where one-third of the claims will be presented by an alleged expert during a mock news report (video), one-third will be presented as a quote next to a still frame from this report (photo), and one-third will be presented as a quote without any media (quote). In this repeated measures design, the same participant sees all three levels of this 'Presentation' variable: video, photo, and quote in a different random order across both trivia and COVID-19 domains. The domain order will be counterbalanced to eliminate ordering effects.
To avoid potential confounding effects where the claim itself might inherently seem more or less plausible, we're randomising not only the order of presentation within each domain but also the association between claims (both true and false) and the type of presentation. This means that for different participants, the same claim might be presented as a quote, photo, or video.
The insights and suggestions offered by ChatGPT can greatly enhance the overall robustness and reliability of your research design. Leveraging its capabilities to analyse and critique your experimental setup can help in identifying potential oversights, refining methodologies, and achieving a more comprehensive alignment with your study’s objectives. This AI-powered assistance ensures an added layer of scrutiny that aids in improving the quality and validity of your research.
