We preregistered this study with the Center for Open Science’s Open Science Framework.
OSF preregistration protocol »
We are participating in COS’s Preregistration Challenge, and will submit our final article to one of the following eligible journals:
| Condition | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| Crackdown | ||||||||
| Yes | • | • | • | • | ||||
| No | • | • | • | • | ||||
| Expected donations | + | + | + | + | ∅ | ∅ | ∅ | ∅ |
| Issue area | ||||||||
| Humanitarian service | • | • | • | • | ||||
| Human rights | • | • | • | • | ||||
| Expected donations | + | + | − | − | + | + | − | − |
| Type of funding | ||||||||
| Government | • | • | • | • | ||||
| Private | • | • | • | • | ||||
| Expected donations | − | + | − | + | − | + | − | + |
The ultimate goal of our research project is to use Bayesian analysis is to test if the region of practical equivalence around the null value (i.e. the proposition that the average treatment effect of NGO crackdown is zero) excludes the posterior 95% HDI.
In this initial pilot study, we use standard frequentist methods to perform power analysis and estimate the desired sample size. We will use the effect size estimates from this study to inform the prior distribution of parameter values in future Bayesian-based experiments (Kruschke 2015, 362).
library(pwr)
u_mine <- 1
# f2_mine <- cohen.ES(test = "f2", size = "small")$effect.size
f2_mine <- 0.02
sig_level_mine <- 0.05
power_mine = 0.90
survey_power <- pwr.f2.test(u = u_mine, v = NULL, f2 = f2_mine,
sig.level = sig_level_mine, power = power_mine)
# n = u + v + 1
n_total <- survey_power$u + survey_power$v + 1
n_group <- n_total / (2 * 2 * 2)Cohen’s rules of thumb for \(f^2\) power are 0.02, 0.15, and 0.35 for small, medium, and large effect sizes.
We are using a 2 × 2 × 2 design, manipulating the experimental condition, the issue area for the NGO, and the source of funding for the NGO. The formula for calculating the \(f^2\)-based sample size with pwr.f2.test requires the following values:
u: degrees of freedom for the numerator. This is a 2 × 2 × 2 design, so the degrees of freedom = \((2 - 1) \times (2 - 1) \times (2 - 1) = 1\)f2: effect size. Here we use a small size, or 0.02.sig.level: significance level, or Type I error probability. Here we use 0.05.power: power of test, or 1 − Type II error probability. Here we use 0.9.Given these parameters, pwr.f2.test yields v, or the degrees of freedom for the denominator. The final sample size n is calculated as u + v + 1.
Thus, the total sample size is 527.3, with 65.91 spread across 8 groups.
This project has received IRB approval from the human subjects research committees at both BYU and Christopher Newport. We have included a generic version of our protocol below.
BYU IRB approval Christopher Newport IRB approval
Charity During Crackdown: Analyzing the Impact of State Repression of NGOs on Philanthropy
State repression of civil society organizations, in particular NGOs, has become ubiquitous over the last few decades. Besides undertaking violence towards costly organizations, states have also increasingly used legislation to crack down on NGOs. By creating barriers to their entry, funding, and operations, these laws restrict the space available for NGOs to engage in advocacy and provide services. How do such legal restrictions on NGOs impact private donors? With states increasingly providing funding for NGOs working in non-contentious issue areas, and a lack of philanthropic culture in many developing countries, numerous NGO rely on funds from private donors in Western countries. We use a survey experiment to examine the impact of state repression of NGOs on the preferences of private donors, as well as their actual behavior donating to besieged NGOs. In particular, we explore how state crackdown against NGOs changes private donors’ preferences and behavior based on the issue area of the organization, and the main source of its funds. Our results have important implications for understanding donor behavior, as well as for the sustainability of civil society organizations in developing countries.
Participants will fill out a survey experiment that (1) describes a prominent international NGO—the International Rescue Committee (IRC)—and (2) requests participants how much money they would hypothetically donate to the organization. We manipulate the vignette describing IRC’s work in three different ways to isolate the effect of civil society legal crackdowns on private philanthropy:
The International Rescue Committee (IRC) focuses on {ISSUE AREA: humanitarian assistance for refugees | human rights for refugees} {MAIN MANIPULATION: and works in countries that have recently passed laws that harshly restrict nonprofit organizations}.
A substantial proportion of IRC’s funding comes from {FUNDING SOURCE: government | private donors}.
Finally, the survey asks participants a series of general questions related to their views of philanthropy, political knowledge, and general demographic features.
Participants of the survey experiment will be recruited via Amazon Mechanical Turk (MTurk), an online crowdsourcing platform that has been extensively used in social science research in recent years. MTurk allows researchers to recruit participants to perform tasks such as filling out surveys and opinion polls, participating in experiments, or coding the content of documents. Researchers advertise their studies as a human intelligence task (HIT) on MTurk, and participants choose only those HITs that interest them, given the promised price and estimated duration of the task. Participants are paid for completing HITs, and payment is transferred directly to participants’ credit cards immediately after the completion of a study.
Our study will be listed as “Survey on international nonprofit organizations” and will take approximately 5 minutes to complete. Participants will be paid $0.75 for completing the experiment, commensurate with a $9/hour wage. After selecting our HIT, participants will be taken directly to the survey experiment, which will be hosted on Qualtrics. MTurk rules state that participants can terminate the study by returning the HIT at any time, without penalty.
We will not collect personally identifiable data. MTurk workers are assigned a semi-random alphanumeric worker ID, which is the only information exposed to us as researchers. We collect MTurk worker IDs to distribute compensation after completion of the experiment and to exclude workers from future iterations of the study. Worker IDs will not be shared with anyone outside of the research team, will be removed from the data set, and will not be associated with any survey responses. Both Amazon and Qualtrics use SSL encryption (HTTPS) when collecting data from workers and transferring data to servers, so responses will remain confidential during the administration of the survey.
After collecting data and paying participants, we plan on creating a clean, anonymous, non-identifiable dataset to use for our analysis and for public distribution following the publication of the results of the study. This data will not contain any worker-identifiable information and will be safe to store and use without extra security measures.
Suparna Chaudhry and Andrew Heiss are co-investigators and will both have access to the clean, anonymized. Both have independently submitted IRB applications to their respective institutions.
Participants will be paid $0.75 for successfully completing the experiment, commensurate with a $9/hour wage.
Previous studies have found that including simple questions to screen for attention can improve the results of MTurk experiments and filter out workers who try to get through the HIT as quickly as possible without reading the questions (Berinsky, Margolis, and Sances 2014). To ensure that participants only receive compensation for completing the full experiment, we include two attention check questions. The first (Q1.3) involves reading several sentences and following instructions to select specific responses. The second (Q3.8) is simpler and asks the respondent to select a single color from a list.
Each attention check question serves a different purpose. The longer question (Q1.3) is designed to filter out respondents who are not paying attention at all and are trying to get a quick payout. Respondents who fail this question will exit the experiment and will not receive compensation. In an effort to not waste worker time, this is the first question after the consent form, so little participant time will have been wasted.
The shorter question (Q3.8) is designed to quickly check if the respondent is still paying attention near the end. Participants who fail this question will still be compensated, but we will account for their lack of attention in our statistical analysis and may omit their response.
The study poses minimal risks to participants. The experiment does not include false information—the manipulations we make in the vignettes (shown between curly braces ({})) only serve to make each fact about the International Rescue Committee (IRC) more salient:
Berinsky, Adam J., Michele F. Margolis, and Michael W. Sances. 2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys.” American Journal of Political Science 58 (3): 739–53. https://doi.org/10.1111/ajps.12081.
Kruschke, John K. 2015. Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan. 2nd ed. London: Academic Press / Elsevier. https://doi.org/10.1016/c2012-0-00477-2.