library(tidyverse)
library(targets)
library(broom)
library(janitor)
library(brms)
library(lme4)
library(lmtest) # For log ratio tests
library(performance) # For ICCs
library(scales)
library(modelsummary)
library(here)
CHAINS <- 4
ITER <- 2000
WARMUP <- 1000
BAYES_SEED <- 1234
options(mc.cores = parallel::detectCores(),
brms.backend = "cmdstanr")
# Generated via random.org
set.seed(9936)
# Load data
# Need to use this withr thing because tar_read() and tar_load() need to see the
# _targets folder in the current directory, but this .Rmd file is in a subfolder
withr::with_dir(here::here(), {
source(tar_read(plot_funs))
canary <- tar_read(panel_lagged)
})
# canary <- canary %>%
# mutate(gwcode = as.factor(gwcode), year = as.factor(year)) %>% # Make a factor
# mutate(PTS_factor_lead1 = ordered(PTS_factor_lead1)) %>% # Make this ordered
# # To do fancy Bell and Jones adjustment (https://doi.org/10.1017/psrm.2014.7)
# # (aka Mundlak devices), explanatory variables should be split into a meaned
# # version (\bar{x}) and a de-meaned version (x - \bar{x}) so that they can
# # explain the within-group and between-group variation. We de-mean three variables here:
# group_by(gwcode) %>%
# mutate_at(vars(v2csreprss, v2x_polyarchy, gdpcap_log),
# list(between = ~mean(., na.rm = TRUE), # Between, for countries
# within = ~. - mean(., na.rm = TRUE))) %>% # Within, for countries
# ungroup()
Interpretation reminders
I always forget how to interpret these dang within/between coefficients in REWB models, so here’s a reminder:
- “Within” coefficients show what happens as you move up and down from the average level.
- Template = increasing 1 unit of
x
above the average in a typical country is associated with a β change in y
- “Between” coefficients show what happens as you move the whole average up. They’re trickier (and less important) to interpret. This post explains more, along with Bell, Fairbrother, and Jones (2018).
Check if multilevel models are warranted
First we can start with a simple null model to see what the effect of country and year is on our outcome (i.e. how much do country and year alone explain the variation in repression?)
m0_pts <- lmer(PTS_lead1 ~ (1 | year) + (1 | gwcode),
REML = FALSE, data = canary)
m0_clphy <- lmer(v2x_clphy_lead1 ~ (1 | year) + (1 | gwcode),
REML = FALSE, data = canary)
m0_clpriv <- lmer(v2x_clpriv_lead1 ~ (1 | year) + (1 | gwcode),
REML = FALSE, data = canary)
We then calculate the intraclass correlation coefficients (ICC) for year and country:
# This shows just the ICC. modelsummary() helpfully includes it.
# icc(m0_pts)
modelsummary(list(m0_pts, m0_clphy, m0_clpriv),
estimate = "{estimate}", statistic = NULL,
gof_map = list(list("raw" = "icc", "clean" = "ICC", "fmt" = 4)))
|
Model 1
|
Model 2
|
Model 3
|
(Intercept)
|
2.610
|
0.639
|
0.687
|
ICC
|
0.7272
|
0.8884
|
0.9315
|
These values represent the variation in the outcome variable that can be explained by the country/year structure, which is a ton! For the PTS model, for instance, the ICC is 0.7272, meaning that the random effects structure explains 73% of the variation in PTS.
We can also look at how each of the levels are related to the outcome
icc(m0_pts, by_group = TRUE)
Group
|
ICC
|
gwcode
|
0.7247164
|
year
|
0.0024886
|
icc(m0_clphy, by_group = TRUE)
Group
|
ICC
|
gwcode
|
0.8767427
|
year
|
0.0116359
|
icc(m0_clpriv, by_group = TRUE)
Group
|
ICC
|
gwcode
|
0.9207071
|
year
|
0.0107510
|
It looks like pretty much all the variability in PTS is explained at a country level and not at a year level, which makes sense, since there’s probably a ton of autocorrelation in human rights practices within a country, while global human rights levels probably don’t move simultaneously across years.
Just because the bulk of the relative variation is at at the country level doesn’t mean that there isn’t significant variability within countries over time. However, the lagged DV seems to soak up pretty much all year-based variability, so we should really use a multilevel model because the country structure matters a lot when explaining human rights abuses. There aren’t really too many gains in efficiency or predictive power when including (1 | year)
though.
Gradually build up the model
Understanding all the different country/year dynamics can get complicated, so John Poe suggests slowly building up the model and inspecting it to see what’s going on.
# First we add the main variable of interest (civil society repression here)
m1 <- lmer(PTS_lead1 ~ v2csreprss + (1 | year) + (1 | gwcode),
REML = FALSE, data = canary)
modelsummary(list("Null model" = m1),
coef_omit = "sd__")
|
Null model
|
(Intercept)
|
2.834
|
|
(0.070)
|
v2csreprss
|
-0.265
|
|
(0.017)
|
Num.Obs.
|
3779
|
R2 Marg.
|
0.120
|
R2 Cond.
|
0.707
|
AIC
|
7401.4
|
BIC
|
7432.6
|
ICC
|
0.7
|
On average, increasing civil society repression by a point leads to a decrease of 0.265 points of human rights repression in the following year.
Finally, we can see if adding v2csreprss
is an improvement over the null m0_pts
model:
lrtest(m0_pts, m1) %>% as_tibble()
#Df
|
LogLik
|
Df
|
Chisq
|
Pr(>Chisq)
|
4
|
-3806.965
|
|
|
|
5
|
-3695.701
|
1
|
222.5279
|
0
|
It is! m1
is definitely an improvement over m0_pts
(p < 0.001).
Next, we can split civil society repression into its within and between versions and see how that performs and how it changes the results
m2 <- lmer(PTS_lead1 ~ v2csreprss_within + v2csreprss_between +
(1 | year) + (1 | gwcode),
REML = FALSE, data = canary)
modelsummary(list("Null model" = m1, "Within/between model" = m2),
coef_omit = "sd__")
|
Null model
|
Within/between model
|
(Intercept)
|
2.834
|
2.978
|
|
(0.070)
|
(0.077)
|
v2csreprss
|
-0.265
|
|
|
(0.017)
|
|
v2csreprss_within
|
|
-0.242
|
|
|
(0.018)
|
v2csreprss_between
|
|
-0.435
|
|
|
(0.048)
|
AIC
|
7401.4
|
7389.8
|
BIC
|
7432.6
|
7427.2
|
Log.Lik.
|
-3695.701
|
-3688.881
|
On average, a one-unit increase beyond a country’s average level of civil society repression (i.e. the within-country effect) is associated with a 0.242 point decrease in overall repression the following year
Also, the v2csreprss_between
coefficient can be interpreted somehow, but I’m not 100% sure how. (I think something like “as a country’s average civil society repression increases, general repression worsens by 0.435” or something).
And we can check if m2
is an improvement over m1
(it is!):
lrtest(m1, m2) %>% as_tibble()
#Df
|
LogLik
|
Df
|
Chisq
|
Pr(>Chisq)
|
5
|
-3695.701
|
|
|
|
6
|
-3688.881
|
1
|
13.63967
|
0.0002215
|
Thus…
So, using a Mundlak within/between split of time-varying covariates is warranted, as is using a country/year random effects structure. Woot.
LS0tCnRpdGxlOiAiTW9kZWxpbmcgY2hvaWNlcyIKYXV0aG9yOiAiU3VwYXJuYSBDaGF1ZGhyeSBhbmQgQW5kcmV3IEhlaXNzIgpkYXRlOiAiTGFzdCBydW46IGByIGZvcm1hdChTeXMudGltZSgpLCAnJUYnKWAiCm91dHB1dDogCiAgaHRtbF9kb2N1bWVudDoKICAgIGNvZGVfZm9sZGluZzogc2hvdwplZGl0b3Jfb3B0aW9uczogCiAgY2h1bmtfb3V0cHV0X3R5cGU6IGNvbnNvbGUKLS0tCgpgYGB7ciBzZXR1cCwgaW5jbHVkZT1GQUxTRX0KbGlicmFyeShrbml0cikKbGlicmFyeShrYWJsZUV4dHJhKQprbml0X3ByaW50LmRhdGEuZnJhbWUgPC0gZnVuY3Rpb24oeCwgLi4uKSB7CiAgcmVzIDwtIHBhc3RlKGMoJycsICcnLCBrYWJsZV9zdHlsaW5nKGthYmxlKHgsIGJvb2t0YWJzID0gVFJVRSkpKSwgY29sbGFwc2UgPSAnXG4nKQogIGFzaXNfb3V0cHV0KHJlcykKfQoKcmVnaXN0ZXJTM21ldGhvZCgia25pdF9wcmludCIsICJkYXRhLmZyYW1lIiwga25pdF9wcmludC5kYXRhLmZyYW1lKQpyZWdpc3RlclMzbWV0aG9kKCJrbml0X3ByaW50IiwgImdyb3VwZWRfZGYiLCBrbml0X3ByaW50LmRhdGEuZnJhbWUpCgprbml0cjo6b3B0c19jaHVuayRzZXQoZmlnLnJldGluYSA9IDMsCiAgICAgICAgICAgICAgICAgICAgICB0aWR5Lm9wdHMgPSBsaXN0KHdpZHRoLmN1dG9mZiA9IDEyMCksICAjIEZvciBjb2RlCiAgICAgICAgICAgICAgICAgICAgICBvcHRpb25zKHdpZHRoID0gOTApLCAgIyBGb3Igb3V0cHV0CiAgICAgICAgICAgICAgICAgICAgICBmaWcuYXNwID0gMC42MTgsIGZpZy53aWR0aCA9IDcsIAogICAgICAgICAgICAgICAgICAgICAgZmlnLmFsaWduID0gImNlbnRlciIsIG91dC53aWR0aCA9ICI4NSUiKQoKb3B0aW9ucyhkcGx5ci5zdW1tYXJpc2UuaW5mb3JtID0gRkFMU0UsCiAgICAgICAga25pdHIua2FibGUuTkEgPSAiIikKYGBgCgpgYGB7ciBsb2FkLWxpYnJhcmllcy1kYXRhLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKbGlicmFyeSh0YXJnZXRzKQpsaWJyYXJ5KGJyb29tKQpsaWJyYXJ5KGphbml0b3IpCmxpYnJhcnkoYnJtcykKbGlicmFyeShsbWU0KQpsaWJyYXJ5KGxtdGVzdCkgICMgRm9yIGxvZyByYXRpbyB0ZXN0cwpsaWJyYXJ5KHBlcmZvcm1hbmNlKSAgIyBGb3IgSUNDcwpsaWJyYXJ5KHNjYWxlcykKbGlicmFyeShtb2RlbHN1bW1hcnkpCmxpYnJhcnkoaGVyZSkKCkNIQUlOUyA8LSA0CklURVIgPC0gMjAwMApXQVJNVVAgPC0gMTAwMApCQVlFU19TRUVEIDwtIDEyMzQKCm9wdGlvbnMobWMuY29yZXMgPSBwYXJhbGxlbDo6ZGV0ZWN0Q29yZXMoKSwKICAgICAgICBicm1zLmJhY2tlbmQgPSAiY21kc3RhbnIiKQoKIyBHZW5lcmF0ZWQgdmlhIHJhbmRvbS5vcmcKc2V0LnNlZWQoOTkzNikKCiMgTG9hZCBkYXRhCiMgTmVlZCB0byB1c2UgdGhpcyB3aXRociB0aGluZyBiZWNhdXNlIHRhcl9yZWFkKCkgYW5kIHRhcl9sb2FkKCkgbmVlZCB0byBzZWUgdGhlCiMgX3RhcmdldHMgZm9sZGVyIGluIHRoZSBjdXJyZW50IGRpcmVjdG9yeSwgYnV0IHRoaXMgLlJtZCBmaWxlIGlzIGluIGEgc3ViZm9sZGVyCndpdGhyOjp3aXRoX2RpcihoZXJlOjpoZXJlKCksIHsKICBzb3VyY2UodGFyX3JlYWQocGxvdF9mdW5zKSkKICAKICBjYW5hcnkgPC0gdGFyX3JlYWQocGFuZWxfbGFnZ2VkKQp9KQoKIyBjYW5hcnkgPC0gY2FuYXJ5ICU+JQojICAgbXV0YXRlKGd3Y29kZSA9IGFzLmZhY3Rvcihnd2NvZGUpLCB5ZWFyID0gYXMuZmFjdG9yKHllYXIpKSAlPiUgICMgTWFrZSBhIGZhY3RvcgojICAgbXV0YXRlKFBUU19mYWN0b3JfbGVhZDEgPSBvcmRlcmVkKFBUU19mYWN0b3JfbGVhZDEpKSAlPiUgICMgTWFrZSB0aGlzIG9yZGVyZWQKIyAgICMgVG8gZG8gZmFuY3kgQmVsbCBhbmQgSm9uZXMgYWRqdXN0bWVudCAoaHR0cHM6Ly9kb2kub3JnLzEwLjEwMTcvcHNybS4yMDE0LjcpCiMgICAjIChha2EgTXVuZGxhayBkZXZpY2VzKSwgZXhwbGFuYXRvcnkgdmFyaWFibGVzIHNob3VsZCBiZSBzcGxpdCBpbnRvIGEgbWVhbmVkCiMgICAjIHZlcnNpb24gKFxiYXJ7eH0pIGFuZCBhIGRlLW1lYW5lZCB2ZXJzaW9uICh4IC0gXGJhcnt4fSkgc28gdGhhdCB0aGV5IGNhbgojICAgIyBleHBsYWluIHRoZSB3aXRoaW4tZ3JvdXAgYW5kIGJldHdlZW4tZ3JvdXAgdmFyaWF0aW9uLiBXZSBkZS1tZWFuIHRocmVlIHZhcmlhYmxlcyBoZXJlOgojICAgZ3JvdXBfYnkoZ3djb2RlKSAlPiUKIyAgIG11dGF0ZV9hdCh2YXJzKHYyY3NyZXByc3MsIHYyeF9wb2x5YXJjaHksIGdkcGNhcF9sb2cpLAojICAgICAgICAgICAgIGxpc3QoYmV0d2VlbiA9IH5tZWFuKC4sIG5hLnJtID0gVFJVRSksICAjIEJldHdlZW4sIGZvciBjb3VudHJpZXMKIyAgICAgICAgICAgICAgICAgIHdpdGhpbiA9IH4uIC0gbWVhbiguLCBuYS5ybSA9IFRSVUUpKSkgJT4lICAgIyBXaXRoaW4sIGZvciBjb3VudHJpZXMKIyAgIHVuZ3JvdXAoKQpgYGAKCiMgSW50ZXJwcmV0YXRpb24gcmVtaW5kZXJzCgpJIGFsd2F5cyBmb3JnZXQgaG93IHRvIGludGVycHJldCB0aGVzZSBkYW5nIHdpdGhpbi9iZXR3ZWVuIGNvZWZmaWNpZW50cyBpbiBSRVdCIG1vZGVscywgc28gaGVyZSdzIGEgcmVtaW5kZXI6CgotICJXaXRoaW4iIGNvZWZmaWNpZW50cyBzaG93IHdoYXQgaGFwcGVucyBhcyB5b3UgbW92ZSB1cCBhbmQgZG93biBmcm9tIHRoZSBhdmVyYWdlIGxldmVsLgogIC0gVGVtcGxhdGUgPSBpbmNyZWFzaW5nIDEgdW5pdCBvZiBgeGAgYWJvdmUgdGhlIGF2ZXJhZ2UgaW4gYSB0eXBpY2FsIGNvdW50cnkgaXMgYXNzb2NpYXRlZCB3aXRoIGEgzrIgY2hhbmdlIGluIGB5YAotICJCZXR3ZWVuIiBjb2VmZmljaWVudHMgc2hvdyB3aGF0IGhhcHBlbnMgYXMgeW91IG1vdmUgdGhlIHdob2xlIGF2ZXJhZ2UgdXAuIFRoZXkncmUgdHJpY2tpZXIgKGFuZCBsZXNzIGltcG9ydGFudCkgdG8gaW50ZXJwcmV0LiBbVGhpcyBwb3N0XShodHRwczovL3N0YXRpc3RpY2FsaG9yaXpvbnMuY29tL2JldHdlZW4td2l0aGluLWNvbnRleHR1YWwtZWZmZWN0cykgZXhwbGFpbnMgbW9yZSwgYWxvbmcgd2l0aCBbQmVsbCwgRmFpcmJyb3RoZXIsIGFuZCBKb25lcyAoMjAxOCldKGh0dHBzOi8vZG9pLm9yZy8xMC4xMDA3L3MxMTEzNS0wMTgtMDgwMi14KS4KCgojIENoZWNrIGlmIG11bHRpbGV2ZWwgbW9kZWxzIGFyZSB3YXJyYW50ZWQKCkZpcnN0IHdlIGNhbiBzdGFydCB3aXRoIGEgc2ltcGxlIG51bGwgbW9kZWwgdG8gc2VlIHdoYXQgdGhlIGVmZmVjdCBvZiBjb3VudHJ5IGFuZCB5ZWFyIGlzIG9uIG91ciBvdXRjb21lIChpLmUuIGhvdyBtdWNoIGRvIGNvdW50cnkgYW5kIHllYXIgYWxvbmUgZXhwbGFpbiB0aGUgdmFyaWF0aW9uIGluIHJlcHJlc3Npb24/KQoKYGBge3IgbnVsbC1tb2RlbHN9Cm0wX3B0cyA8LSBsbWVyKFBUU19sZWFkMSB+ICgxIHwgeWVhcikgKyAoMSB8IGd3Y29kZSksCiAgICAgICAgICAgICAgIFJFTUwgPSBGQUxTRSwgZGF0YSA9IGNhbmFyeSkKbTBfY2xwaHkgPC0gbG1lcih2MnhfY2xwaHlfbGVhZDEgfiAoMSB8IHllYXIpICsgKDEgfCBnd2NvZGUpLAogICAgICAgICAgICAgICBSRU1MID0gRkFMU0UsIGRhdGEgPSBjYW5hcnkpCm0wX2NscHJpdiA8LSBsbWVyKHYyeF9jbHByaXZfbGVhZDEgfiAoMSB8IHllYXIpICsgKDEgfCBnd2NvZGUpLAogICAgICAgICAgICAgICBSRU1MID0gRkFMU0UsIGRhdGEgPSBjYW5hcnkpCmBgYAoKV2UgdGhlbiBjYWxjdWxhdGUgdGhlIGludHJhY2xhc3MgY29ycmVsYXRpb24gY29lZmZpY2llbnRzIChJQ0MpIGZvciB5ZWFyIGFuZCBjb3VudHJ5OgoKYGBge3IgaWNjLW1vZGVsc30KIyBUaGlzIHNob3dzIGp1c3QgdGhlIElDQy4gbW9kZWxzdW1tYXJ5KCkgaGVscGZ1bGx5IGluY2x1ZGVzIGl0LgojIGljYyhtMF9wdHMpCgptb2RlbHN1bW1hcnkobGlzdChtMF9wdHMsIG0wX2NscGh5LCBtMF9jbHByaXYpLAogICAgICAgICAgICAgZXN0aW1hdGUgPSAie2VzdGltYXRlfSIsIHN0YXRpc3RpYyA9IE5VTEwsCiAgICAgICAgICAgICBnb2ZfbWFwID0gbGlzdChsaXN0KCJyYXciID0gImljYyIsICJjbGVhbiIgPSAiSUNDIiwgImZtdCIgPSA0KSkpCmBgYAoKVGhlc2UgdmFsdWVzIHJlcHJlc2VudCB0aGUgdmFyaWF0aW9uIGluIHRoZSBvdXRjb21lIHZhcmlhYmxlIHRoYXQgY2FuIGJlIGV4cGxhaW5lZCBieSB0aGUgY291bnRyeS95ZWFyIHN0cnVjdHVyZSwgd2hpY2ggaXMgYSAqdG9uKiEgRm9yIHRoZSBQVFMgbW9kZWwsIGZvciBpbnN0YW5jZSwgdGhlIElDQyBpcyBgciByb3VuZChpY2MobTBfcHRzKSRJQ0NfYWRqdXN0ZWQsIDQpYCwgbWVhbmluZyB0aGF0IHRoZSByYW5kb20gZWZmZWN0cyBzdHJ1Y3R1cmUgZXhwbGFpbnMgYHIgcGVyY2VudChpY2MobTBfcHRzKSRJQ0NfYWRqdXN0ZWQpYCBvZiB0aGUgdmFyaWF0aW9uIGluIFBUUy4KCldlIGNhbiBhbHNvIGxvb2sgYXQgaG93IGVhY2ggb2YgdGhlIGxldmVscyBhcmUgcmVsYXRlZCB0byB0aGUgb3V0Y29tZQoKYGBge3IgaWNjLW1vZGVscy1ieS1ncm91cH0KaWNjKG0wX3B0cywgYnlfZ3JvdXAgPSBUUlVFKQppY2MobTBfY2xwaHksIGJ5X2dyb3VwID0gVFJVRSkKaWNjKG0wX2NscHJpdiwgYnlfZ3JvdXAgPSBUUlVFKQpgYGAKCkl0IGxvb2tzIGxpa2UgcHJldHR5IG11Y2ggYWxsIHRoZSB2YXJpYWJpbGl0eSBpbiBQVFMgaXMgZXhwbGFpbmVkIGF0IGEgY291bnRyeSBsZXZlbCBhbmQgbm90IGF0IGEgeWVhciBsZXZlbCwgd2hpY2ggbWFrZXMgc2Vuc2UsIHNpbmNlIHRoZXJlJ3MgcHJvYmFibHkgYSB0b24gb2YgYXV0b2NvcnJlbGF0aW9uIGluIGh1bWFuIHJpZ2h0cyBwcmFjdGljZXMgd2l0aGluIGEgY291bnRyeSwgd2hpbGUgZ2xvYmFsIGh1bWFuIHJpZ2h0cyBsZXZlbHMgcHJvYmFibHkgZG9uJ3QgbW92ZSBzaW11bHRhbmVvdXNseSBhY3Jvc3MgeWVhcnMuCgpKdXN0IGJlY2F1c2UgdGhlIGJ1bGsgb2YgdGhlIHJlbGF0aXZlIHZhcmlhdGlvbiBpcyBhdCBhdCB0aGUgY291bnRyeSBsZXZlbCBkb2Vzbid0IG1lYW4gdGhhdCB0aGVyZSBpc24ndCBzaWduaWZpY2FudCB2YXJpYWJpbGl0eSB3aXRoaW4gY291bnRyaWVzIG92ZXIgdGltZS4gSG93ZXZlciwgdGhlIGxhZ2dlZCBEViBzZWVtcyB0byBzb2FrIHVwIHByZXR0eSBtdWNoIGFsbCB5ZWFyLWJhc2VkIHZhcmlhYmlsaXR5LCBzbyB3ZSBzaG91bGQgcmVhbGx5IHVzZSBhIG11bHRpbGV2ZWwgbW9kZWwgYmVjYXVzZSB0aGUgY291bnRyeSBzdHJ1Y3R1cmUgbWF0dGVycyAqYSBsb3QqIHdoZW4gZXhwbGFpbmluZyBodW1hbiByaWdodHMgYWJ1c2VzLiBUaGVyZSBhcmVuJ3QgcmVhbGx5IHRvbyBtYW55IGdhaW5zIGluIGVmZmljaWVuY3kgb3IgcHJlZGljdGl2ZSBwb3dlciB3aGVuIGluY2x1ZGluZyBgKDEgfCB5ZWFyKWAgdGhvdWdoLgoKCiMgR3JhZHVhbGx5IGJ1aWxkIHVwIHRoZSBtb2RlbAoKVW5kZXJzdGFuZGluZyBhbGwgdGhlIGRpZmZlcmVudCBjb3VudHJ5L3llYXIgZHluYW1pY3MgY2FuIGdldCBjb21wbGljYXRlZCwgc28gW0pvaG4gUG9lXShodHRwOi8vd3d3LmpvaG5kYXZpZHBvZS5jb20vKSBzdWdnZXN0cyBzbG93bHkgYnVpbGRpbmcgdXAgdGhlIG1vZGVsIGFuZCBpbnNwZWN0aW5nIGl0IHRvIHNlZSB3aGF0J3MgZ29pbmcgb24uCgpgYGB7ciBiYXNpYy1tb2RlbH0KIyBGaXJzdCB3ZSBhZGQgdGhlIG1haW4gdmFyaWFibGUgb2YgaW50ZXJlc3QgKGNpdmlsIHNvY2lldHkgcmVwcmVzc2lvbiBoZXJlKQptMSA8LSBsbWVyKFBUU19sZWFkMSB+IHYyY3NyZXByc3MgKyAoMSB8IHllYXIpICsgKDEgfCBnd2NvZGUpLAogICAgICAgICAgIFJFTUwgPSBGQUxTRSwgZGF0YSA9IGNhbmFyeSkKbW9kZWxzdW1tYXJ5KGxpc3QoIk51bGwgbW9kZWwiID0gbTEpLAogICAgICAgICAgICAgY29lZl9vbWl0ID0gInNkX18iKQpgYGAKCgpgYGB7ciBtb2RlbC1kZXRhaWxzLCBpbmNsdWRlPUZBTFNFfQptMV9vdXQgPC0gYnJvb20ubWl4ZWQ6OnRpZHkobTEpICU+JSAKICBmaWx0ZXIoZWZmZWN0ID09ICJmaXhlZCIpICU+JSAKICBtdXRhdGUodGVybSA9IG1ha2VfY2xlYW5fbmFtZXModGVybSkpICU+JSAKICBzcGxpdCguJHRlcm0pCgojIEZvciBmdW4sIHdlIGNhbiBzZWUgdGhlIGVmZmVjdHMgb2Ygc3BlY2lmaWMgY291bnRyaWVzL3llYXJzIHdpdGggcmFuZWYoKQojIGxtZTQ6OnJhbmVmKG0xKSRnd2NvZGUgJT4lIHJvd25hbWVzX3RvX2NvbHVtbigpICU+JSBhc190aWJibGUoKQojIGxtZTQ6OnJhbmVmKG0xKSR5ZWFyICU+JSByb3duYW1lc190b19jb2x1bW4oKSAlPiUgYXNfdGliYmxlKCkKYGBgCgpPbiBhdmVyYWdlLCBpbmNyZWFzaW5nIGNpdmlsIHNvY2lldHkgcmVwcmVzc2lvbiBieSBhIHBvaW50IGxlYWRzIHRvIGEgZGVjcmVhc2Ugb2YgYHIgYWJzKHJvdW5kKG0xX291dCR2MmNzcmVwcnNzJGVzdGltYXRlLCAzKSlgIHBvaW50cyBvZiBodW1hbiByaWdodHMgcmVwcmVzc2lvbiBpbiB0aGUgZm9sbG93aW5nIHllYXIuCgpGaW5hbGx5LCB3ZSBjYW4gc2VlIGlmIGFkZGluZyBgdjJjc3JlcHJzc2AgaXMgYW4gaW1wcm92ZW1lbnQgb3ZlciB0aGUgbnVsbCBgbTBfcHRzYCBtb2RlbDoKCmBgYHtyIGNoZWNrLWFnYWluc3QtbTB9CmxydGVzdChtMF9wdHMsIG0xKSAlPiUgYXNfdGliYmxlKCkKYGBgCgpJdCBpcyEgYG0xYCBpcyBkZWZpbml0ZWx5IGFuIGltcHJvdmVtZW50IG92ZXIgYG0wX3B0c2AgKHAgPCAwLjAwMSkuCgpOZXh0LCB3ZSBjYW4gc3BsaXQgY2l2aWwgc29jaWV0eSByZXByZXNzaW9uIGludG8gaXRzIHdpdGhpbiBhbmQgYmV0d2VlbiB2ZXJzaW9ucyBhbmQgc2VlIGhvdyB0aGF0IHBlcmZvcm1zIGFuZCBob3cgaXQgY2hhbmdlcyB0aGUgcmVzdWx0cwoKYGBge3IgbXVuZGxhay1tb2RlbH0KbTIgPC0gbG1lcihQVFNfbGVhZDEgfiB2MmNzcmVwcnNzX3dpdGhpbiArIHYyY3NyZXByc3NfYmV0d2VlbiArCiAgICAgICAgICAgICAoMSB8IHllYXIpICsgKDEgfCBnd2NvZGUpLAogICAgICAgICAgIFJFTUwgPSBGQUxTRSwgZGF0YSA9IGNhbmFyeSkKCm1vZGVsc3VtbWFyeShsaXN0KCJOdWxsIG1vZGVsIiA9IG0xLCAiV2l0aGluL2JldHdlZW4gbW9kZWwiID0gbTIpLAogICAgICAgICAgICAgY29lZl9vbWl0ID0gInNkX18iKQpgYGAKCmBgYHtyIG0yLWRldGFpbHMsIGluY2x1ZGU9RkFMU0V9Cm0yX291dCA8LSBicm9vbS5taXhlZDo6dGlkeShtMikgJT4lIAogIGZpbHRlcihlZmZlY3QgPT0gImZpeGVkIikgJT4lIAogIG11dGF0ZSh0ZXJtID0gbWFrZV9jbGVhbl9uYW1lcyh0ZXJtKSkgJT4lIAogIHNwbGl0KC4kdGVybSkKYGBgCgpPbiBhdmVyYWdlLCBhIG9uZS11bml0IGluY3JlYXNlIGJleW9uZCBhIGNvdW50cnkncyBhdmVyYWdlIGxldmVsIG9mIGNpdmlsIHNvY2lldHkgcmVwcmVzc2lvbiAoaS5lLiB0aGUgd2l0aGluLWNvdW50cnkgZWZmZWN0KSBpcyBhc3NvY2lhdGVkIHdpdGggYSBgciBhYnMocm91bmQobTJfb3V0JHYyY3NyZXByc3Nfd2l0aGluJGVzdGltYXRlLCAzKSlgIHBvaW50IGRlY3JlYXNlIGluIG92ZXJhbGwgcmVwcmVzc2lvbiB0aGUgZm9sbG93aW5nIHllYXIKCkFsc28sIHRoZSBgdjJjc3JlcHJzc19iZXR3ZWVuYCBjb2VmZmljaWVudCBjYW4gYmUgaW50ZXJwcmV0ZWQgc29tZWhvdywgYnV0IEknbSBub3QgMTAwJSBzdXJlIGhvdy4gKEkgdGhpbmsgc29tZXRoaW5nIGxpa2UgImFzIGEgY291bnRyeSdzIGF2ZXJhZ2UgY2l2aWwgc29jaWV0eSByZXByZXNzaW9uIGluY3JlYXNlcywgZ2VuZXJhbCByZXByZXNzaW9uIHdvcnNlbnMgYnkgYHIgYWJzKHJvdW5kKG0yX291dCR2MmNzcmVwcnNzX2JldHdlZW4kZXN0aW1hdGUsIDMpKWAiIG9yIHNvbWV0aGluZykuCgpBbmQgd2UgY2FuIGNoZWNrIGlmIGBtMmAgaXMgYW4gaW1wcm92ZW1lbnQgb3ZlciBgbTFgIChpdCBpcyEpOgoKYGBge3IgY2hlY2stYWdhaW5zdC1tMX0KbHJ0ZXN0KG0xLCBtMikgJT4lIGFzX3RpYmJsZSgpCmBgYAoKIyBUaHVz4oCmCgpTbywgdXNpbmcgYSBNdW5kbGFrIHdpdGhpbi9iZXR3ZWVuIHNwbGl0IG9mIHRpbWUtdmFyeWluZyBjb3ZhcmlhdGVzIGlzIHdhcnJhbnRlZCwgYXMgaXMgdXNpbmcgYSBjb3VudHJ5L3llYXIgcmFuZG9tIGVmZmVjdHMgc3RydWN0dXJlLiBXb290Lgo=