A while back I said I would share my thoughts on using Prolific Academic - in particular for people to try and figure out whether this could be a viable alternative to Amazon Mechanical Turk. I have been using MTurk on and off for about 3 years now. Notwithstanding the occasional grumpy reviewer who hates MTurk data (see below for a comment I was on the receiving end of), I still find this to be a pretty nifty method for getting good quality data. Ouch. Nevertheless, over the last few years, a few stories have come out that have started to worry me (and others) about the long-term viability of MTurk as a behaviour research platform. First are the rumours that new workers from non-US countries are routinely being turned away. Linked to that problem is the estimation that, despite boasting over 500,000 workers from 190+ countries, in reality the vast majority of HITs are completed by a much smaller sample of workers. One paper estimated that all researchers using MTurk are sharing a workforce of around 7,000 workers (i.e. only a few times larger than the participant pool at a typical university). In addition, many of these workers will have taken part in the standard studies (e.g. Dictator Games, Prisoner's Dilemmas) several times before. This non-naivete, coupled with the fact that not all research labs refrain from deceiving subjects (thereby effectively pissing in the shared pool) has led people to question to what extent the data collected from this online workforce is really representative of anything other than itself. Enter Prolific Academic. At the outset, this looks like a great proposition - a dedicated online platform that has been established explicitly with research in mind. So, unlike MTurk, which was co-opted for this task, Prolific Academic has been designed with researchers' wants and needs put first. And I have to say - there is a lot to like. First major advantage: Prolific collects and stores some pretty detailed demographic information from all participants upfront - so no more having to add the standard demographic questionnaire to your studies. This is handy for a few reasons - first, it means that you can post shorter and therefore cheaper experiments; second, there is less chance of participants getting bored by filling out the same information again and again on different tasks; and finally - most usefully - you can pre-screen participants on the basis of age, gender, education level and pretty much any other demographic variable of interest (e.g. phone operating system!) for your study. Pretty cool. The second thing I like about Prolific is the way they nudge (well, shove) researchers to pay participants a fair wage. When you post a task you have to estimate how long it will take - based on this estimation, you then also have to enter a baseline level of compensation that reflects the minimum hourly wage. I like this feature. One thing that could make it better, however, would be if you were able to include the minimum bonus that you will pay subjects (depending on how they perform in the task) as part of this estimation. This would ensure that subjects were getting a fair wage, while also allowing researchers to maximise the potential of their research budget. I also like the fact that Prolific suggests you send partial payments to workers who participate but somehow fail to complete the task. Some of the things I have experienced that I hope are under development - or that would put me off using it again are the following: Speed - sometimes the site is sooooo sloooooow. Not too sure why. But I don't think I'm the only one to have experienced this. Bulk reject: this is basically a bit of a faff. There is no way to screen through your submissions and bulk reject all those who started but didn't complete (which, in my limited experience, seems to be quite a few). So you have to reject each incomplete or substandard submission individually which (given the speed issue above) can be seriously time consuming. Also there is a limit on the number of submissions you can reject as part of any one study. I totally get the rationale for this - and so far the team at Prolific have been super helpful at bulk rejecting incomplete submissions on my behalf - but this is something that has to be given over to the researchers to manage for themselves if the site is to be viable. One option would be to have a partial payment button which allows researchers to reject incomplete or substandard work but sends a partial payment that could be determined either by the researcher or adjusted to the time that the worker spent on the study. Worker demographics: I've just tried to run a multi-country study on Prolific and - as with MTurk - there are some countries that are very well represented (e.g. USA, UK, India, Brazil) while others just don't have the workforce to make it happen. Hopefully this will be something that picks up as the site gains traction. So overall, I would say that I am cautiously optimistic about using Prolific as an alternative to MTurk. For the time being, I would say that the viable samples are going to be similar to those that can be collected on MTurk - and until it is possible for researchers to manage their own submissions (specifically, rejections) then I will probably stick with MTurk. But the minute this feature becomes available I would definitely consider using it much more.
4 Comments
When does doing good make you look bad? A standard evolutionary explanation for why individuals help others is that it allows them to improve their reputation by doing so. Gaining a good reputation is advantageous because it signals to others the kind of person you are: kind, cooperative, trustworthy - in short, the kind of person that would be good to have as a friend or partner. But is it possible that doing good sometimes makes you look bad? It seems it might be. Humans are meta-cognitive. That means we think about how others think - we infer intentions based on observed actions. Very few, if any, other animals on earth have such sophisticated cognitive machinery. This ability comes with a cost, however. Rather than simply accepting good deeds at face value, we typically go one step further and attempt to second guess the 'real' reason for the action. The slightest hint of self-interest behind an ostensibly charitable deed can often be more damaging to an individual's reputation than if they had simply done nothing at all. This 'tainted altruism' effect can help us to explain why we balk at for-profit, for-good companies, but are (more or less) fine with for-profit, for-bad. A recent experiment showed that when raising money for a charity, people were willing to forego over $100,000 in earnings for the charity by choosing a not-for-profit fundraising company over a more effective but for-profit alternative. The same avoidance of the for-profit fundraiser was not manifest when considering the gains for a corporate entity. Why do people feel this way? Why do we hold the for-good companies to higher moral standards than those who do bad? In my view, the explanation lies in our evolved psychology. Historically and in the modern day world, mutually beneficial interactions rely to a large extent on being able to trust your partner. Interacting with an untrustworthy type leaves you open to being suckered. There is an obvious need to discriminate between people who simply play the part of the Good Samaritan when others are looking (but who might screw you over given half the chance) - and those who seem to do this without concern for their own appearance. Indeed, we instinctively downgrade our moral evaluation of people - and companies - who brag about their good deeds to others. This can even result in the phenomenon of do-gooder derogation - targeting morally superior others. Poor old vegans seem to be a popular target. From an evolutionary perspective, we might expect people to be aware that publicising good deeds can sometimes backfire. And they do seem to be. Publicising donations - something that should result in reputation benefits - can cause donors to feel less happy. Most, if not all, fundraising websites nowadays offer donors the option to give anonymously, thereby eschewing reputation consequences of giving. As expected if excessive generosity can be interpreted as showing off, anonymous donors tend to be those whose gift is very large when compared with others that have been made to that cause. This results in something of a quandry: JustGiving, an online fundraising company, noted that sharing a donation to Facebook raises an extra £4.50 (on average) for the cause in question - but donors were reticent to publicise their good deeds to their social network in this way. By tapping into our evolved psychology, and reframing the message from one of self-presentation ("You're an amazing person! Share your donation.") to one of helping a friend raise more money ("Help your friend raise even more money by sharing their page."), JustGiving were able to make sharing more palatable to donors, increasing compliance by over 20 %. According to Oscar Wilde, "The nicest feeling in the world is to do a good deed anonymously - and have somebody find out". I think that we can go one step further and argue that it doesn't just feel nice. Doing good deeds anonymously is perhaps the most reliable signal we can send to others about our moral integrity. The reputation benefits of being found out occasionally might more than offset the costs of the other truly anonymous actions, allowing a general proclivity for anonymous and unconditional generosity to come under positive selection. The implications for brands and corporations with respect to reputation management are profound: money spent on advertising CSR may well undermine the brand's reputation rather than improving it. Somewhat counter-intuitively, trying to ensure that CSR activities are not promoted or advertised to others might often yield net positive effects, even when some of these activities go undetected. Personification, humanisation, consumer relationship marketing. Brands are increasingly exploring new ways to connect with consumers by developing relationships that persist beyond the immediate transaction. The function of any relationship is to ensure repeated interactions. In the world of brands, this means ensuring consumers come back for more.
For example, in response to criticism that the brand ‘lacked emotional attachment, personality and flair’, Eurostar recently rolled out its flagship ‘Avantage’ customer service training programme. As the head of the programme put it: 'We need to have loyalty in a competitive world, so people come back to us and feel a strong bond with Eurostar.' How successful is this approach? Despite its best intentions, Eurostar (and many others) are still not getting it quite right. In part, this is because they tend to view relationships through a business, rather than an evolutionary, lens. Thousands of years of evolution have shaped our behaviour, nowhere more strikingly than in the context of interpersonal relationships.We expect reciprocity and fairness when we interact with others and we have specialised, formidable powers to identify partners who will adhere to these norms. Social ‘cheats’ are avoided at all costs. A shortcut to detecting cheats is to assess how invested a partner is in the relationship: individuals are typically less concerned with one another’s wellbeing in a short-term fling than in a long-run partnership. For example, we intuitively expect to receive better service at our local restaurant than at a tourist trap because in the former, the owner knows that we will come back again and again if he treats us well, whereas in the latter the prospect of repeat custom is slight. Evolution has taught us to be wary of partners who do not appear to value the relationship because they will be less likely to invest in it. If need be, our instincts tell us to walk away. When brands disappoint Humans haven’t evolved to deal with brands, of course, but the mechanisms we use to facilitate real social relationships are also used when dealing with these more abstract entities. As an example, consider this recent experience with Eurostar. We had planned a brief honeymoon near Montpellier with our one-year-old son in tow. A flight would have been cheaper and much quicker but we chose Eurostar because we liked the experience and the brand. Through a series of minor mishaps we ended up missing our train from Montpellier to Paris by a few minutes and, as a consequence, missed our Eurostar connection back to London. Nevertheless, we felt confident that Eurostar would help us out. It was an innocent, one-off mistake and we were repeat – and therefore (we thought) valued – customers. Eurostar wouldn’t leave us out in the cold. Or would it? The Eurostar representative listened to our (admittedly rather feeble) excuse before announcing that two seats on the next train would cost us almost e500 on top of the e200 or so we had already paid for the missed train. 'Surely there has to be something you can do?' we asked. 'It’s our honeymoon. Can’t you make an exception?' It was no use – the computer had spoken and the answer was no. It was not the showcase of emotional flair and attachment you would expect the brand to strive for. Eurostar’s response to our missed train violated pretty much every hardwired expectation about fairness and reciprocity. And our reaction – to resolve never to travel with Eurostar again – was perfectly predictable given this insight. We knew it was our fault we missed the train. We knew we booked non-exchangeable tickets. However, we still felt that we had been treated unfairly. We had invested in the relationship with Eurostar, choosing it over the quicker and cheaper plane option on several occasions.The punishment for the mistake seemed disproportionate and unfair, especially when we boarded the later train to find it half empty. Even though we had previously had plenty of perfectly pleasant journeys with Eurostar, it took just one bad interaction to destabilise the entire relationship. This so-called ‘judgement bias’ is evolutionarily sensible: people do place disproportionate emphasis on negative interactions, in part due to our extreme aversion to experiencing losses. The computerised approach to consumer relationships is deeply flawed. It completely ignores these nuances and evolved biases that lubricate social interactions and, in doing so, creates apparently irrational, yet entirely predictable, consumer responses. To build meaningful relationships, brands should learn to behave more like real humans in their interactions with consumers. Crucially, human relationships do not operate on strict book-keeping approaches. Insisting inflexibly on tit-for-tat in social exchanges fundamentally undermines the relationship because it implies that the partner does not see the relationship as a long-term venture. Consider how you might feel if a friend pointed out that it was your turn to buy the beers since he bought them last week. Or if she insisted on splitting the bill according to precisely what each person ate and drank. This sets subconscious alarm bells ringing. Keeping tabs implies that the partner is unwilling to invest in the relationship and, therefore, does not value you as an interaction partner. Many brands miss this point. Even if we are keeping track of who paid for what, and when, it pays to be slightly forgiving when building relationships and to give individuals the benefit of the doubt occasionally. Computerised models have shown that a more easy-going approach to give-and-take is more successful when building mutually productive relationships. Perhaps the easiest way to humanise brands is to allow real people to decide how to respond in specific consumer interactions, by empowering frontline staff to make decisions flexibly according to the situation. Many industries that interact directly with consumers are now adopting this ‘service recovery’ method. For example, Hampton Inn reception staff are authorised to solve customer problems by offering dissatisfied customers a free stay. For every $1 refunded, the hotel chain bags around $7 in repeat custom. The John Lewis model of empowering individual staff to choose how the brand interacts with consumers has been extremely successful – the store is consistently voted among the UK’s favourite retailers precisely because of the customer experience. Recent research has also shown that stores with longer returns policies actually experience fewer product returns than those with more stringent policies. Good relations pay off Crucially, a flexible approach to consumer relationships is not fundamentally at odds with profitability. On the contrary, John Lewis has gained market share every year over the past four and reports soaring profits in a market where most high street stores are struggling to stay afloat. How could Eurostar have dealt with our situation in a way that would have been consistent with the concept of a brand relationship? How would a human being have responded? In the best-case scenario, they might have made a special exception and allowed us to use our non-flexible tickets on the next train, provided there were seats free. Another solution might have been to charge us the difference in fare between our purchased tickets and the new ones, or to charge a reasonable administration fee for Eurostar’s trouble. But gouging us for e500 to sit on a halfempty train felt exploitative. It sent an unmistakable signal: that Eurostar did not value the relationship. In a situation where it could have helped us out, it instead turned the knife, compounding our sunk costs with a e500 kicker. In doing so, it violated every principle of fairness and reciprocity that we innately expect when we construct relationships with others. And the reaction is evolutionarily hard-wired too: walk away. Caveat emptor – let the brands beware. - This post was originally published in Market Leader and is available online here. In the previous post, I made a few predictions about how I thought people might feel about different fundraising tactics; and why this might be problematic for charities. I thought it might be nice to supplement those earlier predictions with some data. To this end, I posted a survey online, using Prolific Academic (a surprisingly user-friendly and efficient data collection tool - and maybe a viable alternative to MTurk; more on this in a later post). I recruited 205 UK-based adults to answer a short survey about their own charitable behaviour, and how they feel about different fundraising tactics. Admittedly, this is a small sample size and based on self-reported judgements and intentions, so we have to be careful about the inferences we draw. Nevertheless, the responses from these participants revealed some interesting patterns. Let's start with the good news. At least of the people I surveyed, we seem to be a fairly charitable bunch. Of the 205 people asked, only two said they never give to charity. Everyone else gives at least once per year, with the most common category being the monthly donors (31 % of the respondents). As the figure below shows, people give in many and varied ways, with sponsoring a friend and giving to fundraisers being the two most common donation methods. But what I really wanted to know was: how do people actually feel about different fundraising approaches? First off, consider one of the most controversial methods currently used by charities: street fundraisers. These are the people you might see standing outside on the high street or outside the train station, wielding a clipboard and trying to engage passers-by in conversation. Some people also call them chuggers ('charity-muggers') but, perhaps understandably, the fundraisers aren't keen on this label. A whopping 93.1 % of respondents in my survey reported having encountered a street fundraiser at some point or other and, perhaps unsurprisingly, most people don't like it. Over 3 / 4 of the respondents stated that they either strongly dislike or dislike being approached by street fundraisers (with a mere 5 % of the respondents saying that they like or strongly like it). Nevertheless, the tactic is at least somewhat effective. Almost a third of those who had encountered a street fundraiser said they subsequently ended up becoming a regular donor to the charity. Good news? Perhaps not. More important than how people feel about street fundraisers is how people feel about giving to street fundraisers. For this to be a sustainable fundraising strategy, people should - by and large - enjoy giving when asked by a street fundraiser. Here is where the results should start to worry us. Of those who donated, fewer than half were satisfied with their decision, while a worrying 25 % were manifestly dissatisfied. Common complaints from the disgruntled included feeling pressured into giving, or feeling like they had no choice. How respondents feel about street fundraising and donor upgrading tactics Let's park those results for a moment and briefly consider a different fundraising tactic - donor upgrading. This happens when charities contact an individual who made a one-off donation (or did an event for them) to ask them to make another donation or to become a regular donor. Of the 122 people in my survey who said they had made a one-off donation to (or done an event for) a charity, around half reported that they were subsequently contacted again by the charity, usually to ask them to make an additional or a regular donation. As with street fundraisers, people do not like being contacted by charities asking for more money: 37 % of respondents reported being unhappy (or extremely unhappy) that the charity contacted them and a quarter report they would not give to that charity again in the future. Worryingly, this antipathy towards donating seems to spillover into attitudes towards other charities: almost one in five people who charities try to 'upgrade' in this way report that they are unlikely to give to any other charity again. It's not the 4 in 5 I predicted in the last post - but potentially losing 20 % of the donor base is concerning nevertheless. There is a potential silver lining for charities looking to maintain contact with donors without unintentionally aggravating them. One of the consistent themes coming through in the donor comments was that they like being contacted by charities, when the purpose of the contact is to inform how their efforts have made a difference to the appeal. Positive feedback is, as we all know, rewarding. Perhaps more surprisingly is the finding that people might often be happy to be asked for money once in a while, so long as the ask is not too aggressive. Being asked face to face or on the phone seems to make people feel backed into a corner, or pressured. Of the people who said they didn't mind being asked to give, almost all said they preferred if this was done indirectly - via email - rather than direct face to face or phone methods. Although I didn't explore why this is the case in the current survey, it may be that responding to an email ask feels more like a voluntary donation - and therefore is more likely to elicit warm glow - that more confrontational methods, where the donor feels they have no choice but to give. The implications of the data collected here seem to generally support the idea that some fundraising tactics might have hidden costs, in terms of losing donors from the shared pool. This further strengthens the claim that charities are - to some extent - in a social dilemma with other charities, competing for their share of a limited supply of donors. Some of the current fundraising tactics seem to be obviously unsustainable, in that they remove donors from the pool altogether. To improve the sustainability of the entire sector, cooperation is the key. Since the suicide of 92-year old Olive Cooke in September 2015, charities have found themselves the focus of unwanted attention with regard to their fundraising activities. In particular, the use of so-called aggressive tactics has provoked irritation and distrust from the public. Charities have to be able to ask people to give; they can't rely on spontaneous acts of kindness as these simply aren't frequent or sizeable enough to cut the mustard. Nevertheless, fundraisers cannot allow themselves to fall into the trap of believing that worthy ends justify questionable means. Strong-arm tactics might yield short-term gains - but at what cost?
Evolutionary insights can tell us why people give to charity. Cleverly designed experiments have shown that making donations, or helping someone out, activates reward centres in our brain - it feels good to be good. This has been described as the 'warm glow' of giving. However, - and critically for charities - this fuzzy feeling is less pronounced when we feel pressured into giving. So more invasive fundraising techniques, such as door-to-door fundraising or cold-calling an existing donor to ask for more money, can be a big turn off. One such strategy is 'donor upgrading', where one-off donors (e.g. people who have completed an event for the charity, or made a text donation) then receive a phone call to ask for a regular monthly contribution. It is estimated that around one in five people agree to this request - undoubtedly a boon to the charity. But what about the other four? There is a very real danger that the people who say no (i.e. the vast majority) dislike being pressured in this way and are effectively removed from the donor pool altogether. If I am right, this is clearly a problem for the entire sector. Charities therefore find themselves in a social dilemma. They are incentivised to use short-term profitable strategies to extract money from donors, but the potential costs of doing so are borne by the whole sector. Over-exploitation of a shared resource in such settings results in system collapse - a tragedy of the commons. The only way to avoid this is to cooperate. For charities this might mean putting aside aggressive tactics that focus on maximising income from donors, and investing more into methods that maximise donor satisfaction and enjoyment from giving. In the short term this may well be costly - but it is essential for the long-term sustainability of the sector. Just like fishermen sharing the same pond, charities need to harvest their shared resource of donors responsibly and sustainability - a failure to do so can be catastrophic. Insights from laboratory studies giving people similar incentives has shown that cooperation is most likely to flourish in the presence of explicit incentives: carrots and sticks. Nevertheless, in order to be effective, the incentives have to be stringent enough and to apply to everyone. The recent Etherington Report (aimed at increasing trust and confidence in fundraising regulation) notes that current sanctions against charities are ineffective for reasons that seem clear in light of the science: the Institute of Fundraising's (IoF) Code of Practice is too permissive in what it allows and only applies to member organisations. It comes as no surprise, therefore, that short-term income revenue maximisation has been allowed to dominate long-term sustainability in the charitable sector. It is somewhat ironic that an industry aimed at eliciting pro-social behaviour from others has not yet found a way to work cooperatively among themselves. Nevertheless, the problem is not insurmountable. Charities now have a valuable opportunity to rethink the way they do business, and to find ways to work together to improve the way they use the donor pool. This will mean finding way to more strongly align individual organisations' interests with those of the sector as a whole. There is a wealth of scientific data that can be leveraged to make this happen - and plenty of experts who would love to help. Charities need to cooperate. But will they? |
Nichola Raihani
is a Royal Society University Research Fellow based in the Department of Experimental Psychology at University College London. Archives
April 2016
Categories |