Assessment 2
PS923 - Methods and Analysis in Behavioural Science
Autumn Term 2024 (updated: 2024-12-06)
• This assessment counts for 36% of your overall grade.
• Submission Instructions: Submit your solution as one html or pdf document containing R code, R output, figures, and written out text (i.e., full sentences) to Tabula (Assessment 2) by 12:00 noon (midday) on Wednesday, 15th January 2025.
• Please use RMarkdown to create the document.
• Important: Your document should be called YOUR-STUDENT-ID_a2 (followed by the correct file ex- tension). Please also add your student ID to the top of the document. To help ensure anonymous marking, please refrain from using your name in either the document, script, or the file name.
• Your text does not need to contain references (i.e., references to scientific papers).
General Guidelines
Please complete the following tasks. Your answers should have two separate sections for each task, one immediately after the other.
In the first section, write out your answers using complete sentences, as you might for the results section of a paper. Include descriptive statistics in the text, or in tables or figures as appropriate. Tables and figures should be of publication quality (i.e., fully labelled, etc.). Integrate inferential statistics into your description of the results. Your answers might be short. Given the correctness/appropriateness of the statistical analysis, the first section will play the main role for your mark.
The second section acts as an appendix; this should include the complete R code that you used and its output. Add comments (after a #) to explain what the code does. The code should show all of the commands that you used; enough for others to replicate exactly what you did (I will be copying and pasting code to run it, so make sure that works). The second section will be used to help identify the source of any mistakes. You can include figures here that you used to explore the data that you do not wish to include in the first section. For practical reports and papers you would only submit the first section in the main manuscript.
For an example of such a solution, see the Assignment 1 sheet.
Finally, please note that submitting AI-generated text for this assessment will be considered as plagiarism; i.e., suspected cases will be referred to the academic integrity panel.
Task 1 Personalised references
When applying for a job, it’s obviously highly desirable to have a relevant CV and good references. This task relates to whether, and the extent to which, the quality or style of references may influence decisions.
The file candidate .csv provides simulated (i.e., fictitious) data for how potential employers (identified by uID number) perceived various job candidates (rated on a scale of 0 to 100; 0 denoting definitely wouldn’t interview, 100 being definitely would interview). Each rating was given after reading the candidate’s CV and a reference letter for the candidate.
The experiment used two types of candidates; each was either applying for a managerial role (where the employer would be likely to interact regularly with the candidate in-person if they subsequently got the job) or a technical role (where they would be less likely to meet on a day-to-day basis). The reference of each candidate had a bias: each was manipulated to either be slightly positive or slightly negative. Half the references were personalised (these included a small photo of the reviewer’s face in a corner of the page, and a ‘flashy’ signature in coloured ink) whilst the other half were not (no photo, and just the printed name of the referee). Note that each employer only saw one reference (positive or negative) for any particular candidate, and each employer saw an equal number of positively and negatively biased reviews across the different items.
Given that references supply information, the general expectation is that the positively biased references will tend to produce higher ratings than the negatively biased references, but is this effect similar for both personalised and non-personalised references (and for different role types)? Some might expect that the impact of the reference bias would be greater for personalised references than the nonpersonal.
The focal hypothesis is that the effect of bias (on ratings) is stronger for personalised references than for nonpersonal references. The type of role (management or technical) mainly serves as a control variable but should also be considered. Please analyse the data with a repeated-measures (within-subjects) ANOVA and report the results as you would in a journal paper.
If you were to run a similar study in future (i.e., with the same general aims, but with the potential for small changes in the design), is there anything particular that you would change, or specifically aim to control for? Please comment on this at the end of your report.
Task 2 Short-cuts and time penalties
Time is a precious commodity, so it is not surprising that many choices in life depend on perceptions of risk in relation to how much time something might take or save.
This task provides simulated data (in file speed_greed.csv) on how individuals make choices when they are trying to achieve a goal in a minimal amount of time. The participants encounter choices in an online (single-player) game; they make several such choices before completing each level of the game. The options are sometimes useful shortcuts and sometimes impose time penalties. The decisions govern how much of a risk they take when choosing between shortcuts, or when avoiding time penalties. Prior to each decision, the participant learns whether they will face a shortcut option or a time-penalty (i.e., they know that they will gain or lose time on that decision, but not how much time).
Having learned that there is an available shortcut, for instance, they will be given a choice between a high- variance choice (e.g., saving either 100 or 500 seconds, with equal probability), or a low-variance choice with the same average (e.g., saving either 200 or 400 seconds, again with equal probability). Thus, either option saves the same amount of time on average (300 seconds in this example) – but one option is known to have a higher variance (i.e., it is more ‘risky’). If, instead of a shortcut, they had been informed that they would receive a time penalty, then after making their choice for whether to go for the more or less risky option, they would then lose that amount of time (according to the same general scheme).
Having familiarised themselves with the game (moving up, down, left, right, and how to select options when faced with choices), each participant was tasked with traversing 3 levels (L1, L2, L3). To motivate the participants, payment for their involvement was linked to the speed with which they completed the 3 levels (the faster, the better). In each level, 18 key decisions were recorded (of whether they took the high or low risk option); choices for a high variance outcome are denoted by 1; choices for the low variance outcome are denoted by 0.
The file provides data for 80 participants; the data is in a wide format (one line per participant), with the intent being to have recorded a 1 or 0 for each decision; however, the recording system was not perfect; very occasionally it would not record a value for one of the choices.
Each participant was assigned to one Experience group: shortcut, penalty, or mixed, which governed experiences during level 2 of the game. In levels 1 and 3, some gamble options were for shortcuts, and some were for time penalties. The options presented to a participant in level 1 was repeated in level 3 (though in a different order, to help prevent participants noticing). During level 2, the participant’s group determined whether they repeatedly faced shortcuts, time penalties or a mix of the two. The order of trials in each group was randomized separately for each participant, so each individual saw the trials in an order that was uniquely created for them.
Your task is to analyse the data with an ANOVA and address the research question of whether a series of positive or negative experiences (i.e., shortcuts or penalties) has an effect on the probability of making a risky (i.e., high variance) choice. In other words, are risk preferences stable or affected by recent experiences? Please report the results as you would in a journal paper.