Paper: Fast et al 2009 experiment 4

Section A. Popper (25 marks) 

1) Concisely state the theory that the authors present as being put up to test?

People assigned a role of high rather than low social power will perceive themselves to have more control (even when they don’t); and specifically that this effect is mediated by feeling happier when given high power (the authors would put the last claim differently, but this is what I believe was actually tested).

2) What pattern of results, if any, would falsify the theory?

The previous experiments had found the principal effect of assigned social power influencing perceived control. In experiment 4, the theory under test predicts that assigned high social power would increase happiness. If happiness does not increase more than a minimal amount the theory would be falsified. Thus a minimal amount should be specified to achieve this falsifiability; this point is taken up in question 6.

3) What background knowledge inspired this theory but is not being directly tested?

The relationship of perceived control to health (e.g. low perceived control is associated with depression) and socioeconomic status.

4) What background knowledge must be assumed in order for the test to be a test of the theory in (1)?

a) The measure of happiness actually measured happiness.
b) Subjects reported how they actually felt; they did not guide their answers to the questions just based on what they thought the experimenter wanted
c) Telling subjects that they might be employer or applicant, rather than clearly one or the other, does not put them in a state of uncertainty and hence increased arousal.

5) How safe is the background knowledge in (4)?

a) Not very safe. The happiness measure averaged together several questions: ‘happy’, ‘excited’, ‘enthusiastic’ and ‘proud’. But excitement and pride are compatible with feeling unhappy (e.g. one can be proud and angry). For example, the arousal caused by uncertainty may give high excitement ratings, which does not necessarily mean that one is happy. Why not just use the scale for happy?
b) It would have been good to ask subjects at the end what they thought the experiment was testing, otherwise demand characteristics remains a plausible counter-explanation
c) Not safe; it has not been tested. Reporting separate ratings for happiness and excitement could have tested it.

 Section B. Neyman Pearson (25 marks)

6) Have the authors determined what difference (or range of differences) would be expected if the theory were true?

No.

7) If not, do you know any results or other papers that could allow you to state an expected size of difference? Provide an expected difference and state your reasons.

In previous experiments in the paper and in the current one) perceived control (/optimism) changed by about 1 unit of a seven point scale due to the manipulation of social power.  Thus, if happiness mediated the effect of perceived control it should change by at least this much on a seven point scale. (NB: the standard deviation for the happiness ratings was about the same as for the perceived control/optimism ratings.)

8) Have the authors established their sensitivity to pick up such a difference, through power or confidence intervals? If not, provide a calculation yourself.

No they did not.

In experiment 3 the Cohen’s d for the effect of power on perceived control was about 0.6; in experiment 2 it was closer to 2. So for experiment 4 one might expect a d of roughly 1 for the effect on perceived control. Indeed, this was the effect size for optimism, which measures something very similar if not the same. Thus we need to be able to pick up an effect on happiness of about d = 1.
With 43 subjects, the power of the experiment is 0.89 (using Gpower). By convention psychologists often regard a power of greater than 0.8 acceptable so by this convention the authors had sufficient power. However I personally regard 0.8 as unacceptable for important tests. Assuming one wants a power of 95%, the experiment was not designed to be sensitive.

In terms of a confidence interval, with 41 degrees of freedom, the critical value of t is 2.02 (obtained from: http://www.psychstat.missouristate.edu/introbook/tdist.htm  ). The obtained mean difference was -0.54 (minus, because in the wrong direction), and the obtained t was 1.40. So the standard error of the difference was 0.54/1.4 = 0.39. Thus the confidence interval is [-0.54 – 2.02*0.39, -0.54 + 2.02*0.39] = [-1.32, 0.24].  The confidence interval allows us to reject any increase in measured happiness above about a quarter of a scale point, ruling out an increase of 1 or more units. Thus, the confidence interval shows the study actually was sensitive.

9) Was the test of the theory  in (1) severe (in Popper’s sense)?

By a reasonably common convention the study had enough power (i.e. greater than 80%), and in this statistical sense the test was severe (though not if one wanted a power of more than 90%). However, in a more general sense, the test was not severe as other plausible theories exist that would predict the results even if the theory in 1) above was false (namely the theories mentioned in 5 above).

Section C Lakatos (25 marks)

10) State the hard core of the research programme the authors are working in

The work falls in the area of social cognition which has core beliefs that one can understand social processes by reference to cognitive processes using the methods of cognitive psychology. The first sentence of the introduction may define a more specific hard core: ‘Having a sense of control of the future is a fundamental motive for humans’. That paragraph then reviews papers that could be construed as a historical commitment to this idea.

11) Does the paper contribute to the research programme in a progressive or degenerating way? State your reasons.

The claim that social power makes one feel in control in situations unrelated to one’s power role and indeed in cases where one has no control (like their experiment 1) is novel, and confirmed. Claiming the effect is not due to happy mood is falsifiable, and survives testing, even if the test was not very severe. All we need is one novel prediction and its corroboration for the contribution to be progressive and the paper certainly provides that.

Section D  Bayes (25 marks)

12) What was the mean difference obtained in the study?

-0.54

13) What was the standard error of this difference?

0.39

14) Extending your answer in (7), specify a probability distribution for the difference expected by the theory and justify it

The population mean difference in happiness needed is at least tat found for perceived control. Based on past studies the population effect size for perceived control can be estinated at about 1 unit (confirmed in this study), but there will be some uncertainty in this estimate. A population mean difference less than 1 is thus increasingly unlikely. A population mean difference greater than 2 or 3 is also unlikely for a 7-point scale. I represent these constraints with a normal with a mean of 1.5 and a standard deviation of 0.75. Thus, the population mean difference is constrained to be in the region from roughly 0 to 3.

15) What is the Bayes factor in favour of the theory over the null hypothesis?

0.07

16) What does this Bayes factor tell you that the t-test does not?

Assuming the experiment is well designed, we can reduce our odds in favour of the happiness mediation hypothesis over the null hypothesis to 1/14th of our prior values (as 0.07 = 1/14). The evidence strongly supports the null hypothesis over the happiness mediation hypothesis. The t-test only allowed us to accept the null hypothesis and reject the mediation hypothesis by a procedure with acceptable error rates, without telling us by how much our confidence in the hypotheses should change, and indeed without telling us how strong the evidence is.

1000 words