| Title: | Tests of Non-Nested Models |
|---|---|
| Description: | Testing non-nested models via theory supplied by Vuong (1989) <DOI:10.2307/1912557>. Includes tests of model distinguishability and of model fit that can be applied to both nested and non-nested models. Also includes functionality to obtain confidence intervals associated with AIC and BIC. This material is partially based on work supported by the National Science Foundation under Grant Number SES-1061334. |
| Authors: | Edgar Merkle [aut, cre], Dongjun You [aut], Lennart Schneider [ctb], Mauricio Garnier-Villarreal [ctb], Seongho Bae [ctb], Phil Chalmers [ctb] |
| Maintainer: | Edgar Merkle <[email protected]> |
| License: | GPL-2 | GPL-3 |
| Version: | 0.5-8 |
| Built: | 2025-01-26 03:13:04 UTC |
| Source: | https://github.com/qpsy/nonnest2 |
Calculate confidence intervals of AIC and BIC for non-nested models.
icci(object1, object2, conf.level = 0.95, ll1 = llcont, ll2 = llcont)icci(object1, object2, conf.level = 0.95, ll1 = llcont, ll2 = llcont)
object1 |
a model object |
object2 |
a model object |
conf.level |
confidence level of the interval |
ll1 |
an optional function for computing log-likelihood contributions of object1 |
ll2 |
an optional function for computing log-likelihood contributions of object2 |
Functionality is currently available for models of classes
lm, glm, glm.nb, clm, hurdle, zeroinfl, mlogit, nls, polr, rlm, and lavaan.
Users should take care to ensure that the two models have the same dependent variable (or, for lavaan objects, identical modeled variables), with observations ordered identically within each model object. Assuming the same data matrix is used to fit each model, observation ordering should generally be identical. There are currently no checks for this, however.
Note: if models are nested or if the "variance test" from
vuongtest() indicates models are indistinguishable, then the
intervals returned from icci() will be incorrect.
an object of class icci containing test results.
Ed Merkle and Dongjun You
Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57, 307-333. <DOI:10.2307/1912557>
Merkle, E. C., You, D., & Preacher, K. (2016). Testing non-nested structural equation models. Psychological Methods, 21, 151-163. <DOI:10.1037/met0000038>
## Not run: ## Count regression comparisons require(MASS) house1 <- glm(Freq ~ Infl + Type + Cont, family=poisson, data=housing) house2 <- glm(Freq ~ Infl + Sat, family=poisson, data=housing) ## CI for BIC icci(house2, house1) ## Further comparisons to hurdle, zero-inflated models require(pscl) bio1 <- glm(art ~ fem + mar + phd + ment, family=poisson, data=bioChemists) bio2 <- hurdle(art ~ fem + mar + phd + ment, data=bioChemists) bio3 <- zeroinfl(art ~ fem + mar + phd + ment, data=bioChemists) icci(bio2, bio1) icci(bio3, bio1) icci(bio3, bio2) ## Latent variable model comparisons require(lavaan) HS.model <- 'visual =~ x1 + x2 + x3 textual =~ x4 + x5 + x6 speed =~ x7 + x8 + x9 ' fit1 <- cfa(HS.model, data=HolzingerSwineford1939, meanstructure=TRUE) fit2 <- cfa(HS.model, data=HolzingerSwineford1939, group="school") icci(fit1, fit2) ## End(Not run)## Not run: ## Count regression comparisons require(MASS) house1 <- glm(Freq ~ Infl + Type + Cont, family=poisson, data=housing) house2 <- glm(Freq ~ Infl + Sat, family=poisson, data=housing) ## CI for BIC icci(house2, house1) ## Further comparisons to hurdle, zero-inflated models require(pscl) bio1 <- glm(art ~ fem + mar + phd + ment, family=poisson, data=bioChemists) bio2 <- hurdle(art ~ fem + mar + phd + ment, data=bioChemists) bio3 <- zeroinfl(art ~ fem + mar + phd + ment, data=bioChemists) icci(bio2, bio1) icci(bio3, bio1) icci(bio3, bio2) ## Latent variable model comparisons require(lavaan) HS.model <- 'visual =~ x1 + x2 + x3 textual =~ x4 + x5 + x6 speed =~ x7 + x8 + x9 ' fit1 <- cfa(HS.model, data=HolzingerSwineford1939, meanstructure=TRUE) fit2 <- cfa(HS.model, data=HolzingerSwineford1939, group="school") icci(fit1, fit2) ## End(Not run)
Obtain log-likelihood values associated with individual observations, evaluated at the ML estimates.
llcont(x, ...)llcont(x, ...)
x |
a model object |
... |
arguments passed to specific methods |
This is a S3 generic function.
Currently, the method is defined for lm, glm, glm.nb,
clm, hurdle, zeroinfl, mlogit, nls,
polr, rlm, lavaan, vglm, mirt, and OpenMx objects.
An object of class numeric containing individuals' contributions to the log-likelihood. The sum of these contributions equals the model log-likelihood.
Ed Merkle, Dongjun You, Lennart Schneider, Mauricio Garnier-Villarreal, and Phil Chalmers
## Fit gamma glm, check that sum of llcont() equals ## the model loglikelihood: clotting <- data.frame(u = c(5,10,15,20,30,40,60,80,100), lot1 = c(118,58,42,35,27,25,21,19,18), lot2 = c(69,35,26,21,18,16,13,12,12)) gam1 <- glm(lot1 ~ log(u), data = clotting, family = Gamma) sum(llcont(gam1)) logLik(gam1)## Fit gamma glm, check that sum of llcont() equals ## the model loglikelihood: clotting <- data.frame(u = c(5,10,15,20,30,40,60,80,100), lot1 = c(118,58,42,35,27,25,21,19,18), lot2 = c(69,35,26,21,18,16,13,12,12)) gam1 <- glm(lot1 ~ log(u), data = clotting, family = Gamma) sum(llcont(gam1)) logLik(gam1)
Test pairs of models using Vuong's (1989) <DOI:10.2307/1912557> theory. This includes a test of model distinguishability and a test of model fit.
vuongtest( object1, object2, nested = FALSE, adj = "none", ll1 = llcont, ll2 = llcont, score1 = NULL, score2 = NULL, vc1 = vcov, vc2 = vcov )vuongtest( object1, object2, nested = FALSE, adj = "none", ll1 = llcont, ll2 = llcont, score1 = NULL, score2 = NULL, vc1 = vcov, vc2 = vcov )
object1 |
a model object |
object2 |
a model object |
nested |
if |
adj |
Should an adjusted test statistic be calculated? Defaults to “none”, with possible adjustments being “aic” and “bic” |
ll1 |
an optional function for computing log-likelihood contributions of object1 |
ll2 |
an optional function for computing log-likelihood contributions of object2 |
score1 |
an optional function for computing scores of object 1 |
score2 |
an optional function for computing scores of object 2 |
vc1 |
an optional function for computing the asymptotic covariance matrix of the object1 parameters |
vc2 |
an optional function for computing the asymptotic covariance matrix of the object2 parameters |
For non-nested models, the test of distinguishability indicates whether or not the models can possibly be distinguished on the basis of the observed data. The LRT then indicates whether or not one model fits better than another.
For nested models (nested=TRUE), both tests serve as robust
alternatives to the classical likelihood ratio tests. In this case,
the adj argument is ignored.
Users should take care to ensure that the two models have the same dependent variable (or, for lavaan objects, identical modeled variables), with observations ordered identically within each model object. Assuming the same data matrix is used to fit each model, observation ordering should generally be identical. There are currently no checks for this, however.
an object of class vuongtest containing test results.
Ed Merkle and Dongjun You
Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57, 307-333. <DOI:10.2307/1912557>
Merkle, E. C., You, D., & Preacher, K. (2016). Testing non-nested structural equation models. Psychological Methods, 21, 151-163. <DOI:10.1037/met0000038>
## Not run: ## Count regression comparisons require(MASS) house1 <- glm(Freq ~ Infl + Type + Cont, family=poisson, data=housing) house2 <- glm(Freq ~ Infl + Sat, family=poisson, data=housing) house3 <- glm(Freq ~ Infl, family=poisson, data=housing) ## house3 is nested within house1 and house2 anova(house3, house1, test="Chisq") anova(house3, house2, test="Chisq") ## house 2 is not nested in house1, so this test is invalid anova(house2, house1, test="Chisq") ## Use vuongtest() instead vuongtest(house2, house1) ## Application to models with different distributional assumptions require(pscl) bio1 <- glm(art ~ fem + mar + phd + ment, family=poisson, data=bioChemists) bio2 <- hurdle(art ~ fem + mar + phd + ment, data=bioChemists) bio3 <- zeroinfl(art ~ fem + mar + phd + ment, data=bioChemists) vuongtest(bio2, bio1) vuongtest(bio3, bio1) vuongtest(bio1, bio2) vuongtest(bio1, bio3) vuongtest(bio3, bio2) ## Application to latent variable models require(lavaan) HS.model <- 'visual =~ x1 + x2 + x3 textual =~ x4 + x5 + x6 speed =~ x7 + x8 + x9 ' fit1 <- cfa(HS.model, data=HolzingerSwineford1939) fit2 <- cfa(HS.model, data=HolzingerSwineford1939, group="school") vuongtest(fit1, fit2) ## Supplying custom vcov function require(lme4) require(merDeriv) fm1 <- lmer(Reaction ~ Days + (Days | Subject), sleepstudy, REML=FALSE) fm2 <- lmer(Reaction ~ Days + (Days || Subject), sleepstudy, REML=FALSE) vcl <- function(obj) vcov(obj, full=TRUE) vuongtest(fm1, fm2, vc1=vcl, vc2=vcl, nested=TRUE) ## End(Not run)## Not run: ## Count regression comparisons require(MASS) house1 <- glm(Freq ~ Infl + Type + Cont, family=poisson, data=housing) house2 <- glm(Freq ~ Infl + Sat, family=poisson, data=housing) house3 <- glm(Freq ~ Infl, family=poisson, data=housing) ## house3 is nested within house1 and house2 anova(house3, house1, test="Chisq") anova(house3, house2, test="Chisq") ## house 2 is not nested in house1, so this test is invalid anova(house2, house1, test="Chisq") ## Use vuongtest() instead vuongtest(house2, house1) ## Application to models with different distributional assumptions require(pscl) bio1 <- glm(art ~ fem + mar + phd + ment, family=poisson, data=bioChemists) bio2 <- hurdle(art ~ fem + mar + phd + ment, data=bioChemists) bio3 <- zeroinfl(art ~ fem + mar + phd + ment, data=bioChemists) vuongtest(bio2, bio1) vuongtest(bio3, bio1) vuongtest(bio1, bio2) vuongtest(bio1, bio3) vuongtest(bio3, bio2) ## Application to latent variable models require(lavaan) HS.model <- 'visual =~ x1 + x2 + x3 textual =~ x4 + x5 + x6 speed =~ x7 + x8 + x9 ' fit1 <- cfa(HS.model, data=HolzingerSwineford1939) fit2 <- cfa(HS.model, data=HolzingerSwineford1939, group="school") vuongtest(fit1, fit2) ## Supplying custom vcov function require(lme4) require(merDeriv) fm1 <- lmer(Reaction ~ Days + (Days | Subject), sleepstudy, REML=FALSE) fm2 <- lmer(Reaction ~ Days + (Days || Subject), sleepstudy, REML=FALSE) vcl <- function(obj) vcov(obj, full=TRUE) vuongtest(fm1, fm2, vc1=vcl, vc2=vcl, nested=TRUE) ## End(Not run)