# Separation (statistics)

> Mediated Wiki article. Canonical URL: https://mediated.wiki/source/Separation_(statistics)
> Markdown URL: https://mediated.wiki/source/Separation_(statistics).md
> Source: https://en.wikipedia.org/wiki/Separation_(statistics)
> Source revision: 1265306374
> License: Creative Commons Attribution-ShareAlike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/)

In [statistics](/source/Statistics), **separation** is a phenomenon associated with models for [dichotomous](/source/Dichotomy) or categorical outcomes, including [logistic](/source/Logistic_regression) and [probit regression](/source/Probit_regression). Separation occurs if the predictor (or a [linear combination](/source/Linear_combination) of some subset of the predictors) is associated with only one outcome value when the predictor range is split at a certain value.

## The phenomenon

For example, if the predictor *X* is continuous, and the outcome *y* = 1 for all observed *x* > 2. If the outcome values are (seemingly) perfectly determined by the predictor (e.g., *y* = 0 when *x* ≤ 2) then the condition "complete separation" is said to occur. If instead there is some overlap (e.g., *y* = 0 when *x* < 2, but *y* has observed values of 0 and 1 when *x* = 2) then "quasi-complete separation" occurs. A [2 × 2 table](/source/Contingency_table) with an empty (zero) cell is an example of quasi-complete separation.

## The problem

This observed form of the data is important because it sometimes causes problems with the estimation of regression coefficients. For example, [maximum likelihood (ML) estimation](/source/Maximum_likelihood_estimation) relies on maximization of the likelihood function, where e.g. in case of a [logistic regression](/source/Logistic_regression) with completely separated data the maximum appears at the parameter space's margin, leading to "infinite" estimates, and, along with that, to problems with providing sensible [standard errors](/source/Standard_error).[1][2] Statistical software will often output an arbitrarily large parameter estimate with a very large standard error.[3]

## Possible remedies

An approach to "fix" problems with [ML estimation](/source/Maximum_likelihood_estimation) is the use of [regularization](/source/Regularization_(mathematics)) (or "[continuity corrections](/source/Continuity_correction)").[4][5] In particular, in case of a logistic regression problem, the use of *exact logistic regression* or *Firth logistic regression*, a bias-reduction method based on a penalized likelihood, may be an option.[6]

Alternatively, one may avoid the problems associated with likelihood maximization by switching to a [Bayesian](/source/Bayesian_inference) approach to inference. Within a Bayesian framework, the pathologies arising from likelihood maximization are avoided by the use of [integration](/source/Integral) rather than [maximization](/source/Mathematical_optimization), as well as by the use of sensible [prior probability distributions](/source/Prior_probability).[7]

## References

1. Zeng, Guoping & Zeng, Emily (2019). "On the Relationship between Multicollinearity and Separation in Logistic Regression". *[Communications in Statistics](/source/Communications_in_Statistics)*. **50** (7): 1989–1997. Simulation and Computation. [doi:10.1080/03610918.2019.1589511](https://doi.org/10.1080/03610918.2019.1589511). [S2CID 132047558](https://api.semanticscholar.org/CorpusID:132047558)

1. Albert, A. & Anderson, J. A. (1984). "On the Existence of Maximum Likelihood Estimates in Logistic Regression Models". *Biometrika*. **71** (1–10): 1–10. [doi:10.1093/biomet/71.1.1](https://doi.org/10.1093/biomet/71.1.1)

1. McCullough, B. D. & Vinod, H. D. (2003). "Verifying the Solution from a Nonlinear Solver: A Case Study". *[American Economic Review](/source/American_Economic_Review)*. **93** (3): 873–892. [doi:10.1257/000282803322157133](https://doi.org/10.1257/000282803322157133). [JSTOR 3132121](https://www.jstor.org/stable/3132121)

1. Cole, S.R.; Chu, H.; Greenland, S. (2014), "Maximum likelihood, profile likelihood, and penalized likelihood: A primer", *American Journal of Epidemiology*. **179** (2): 252–260, [doi:10.1093/aje/kwt245](https://doi.org/10.1093/aje/kwt245). [PMC 3873110](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3873110). [PMID 24173548](https://pubmed.ncbi.nlm.nih.gov/24173548)

1. Sweeting, M.J.; Sutton, A.J.; Lambert, P.C. (2004), "What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data", *Statistics in Medicine*. **23** (9): 1351–1375, [doi:10.1002/sim.1761](https://doi.org/10.1002/sim.1761). [PMID 15116347](https://pubmed.ncbi.nlm.nih.gov/15116347). [S2CID 247667708](https://api.semanticscholar.org/CorpusID:247667708)

1. Mansournia, Mohammad Ali; Geroldinger, Angelika; Greenland, Sander; Heinze, Georg (2018). "Separation in Logistic Regression: Causes, Consequences, and Control". *[American Journal of Epidemiology](/source/American_Journal_of_Epidemiology)*. **187** (4): 864–870. [doi:10.1093/aje/kwx299](https://doi.org/10.1093/aje/kwx299). [PMID 29020135](https://pubmed.ncbi.nlm.nih.gov/29020135)

1. Gelman, A.; Jakulin, A.; Pittau, M.G.; Su, Y. (2008), "A weakly informative default prior distribution dor logistic and other regression models", *Annals of Applied Statistics*. **2** (4): 1360–1383, [arXiv:0901.4011](https://arxiv.org/abs/0901.4011). [doi:10.1214/08-AOAS191](https://doi.org/10.1214/08-AOAS191)

## Further reading

- Albert, A. & Anderson, J. A. (1984), "On the existence of maximum likelihood estimates in logistic regression models", *Biometrika*. **71** (1): 1–10, [doi:10.1093/biomet/71.1.1](https://doi.org/10.1093/biomet/71.1.1)
- Kosmidis, I. & Firth, D. (2021), "Jeffreys-prior penalty, finiteness and shrinkage in binomial-response generalized linear models", *Biometrika*. **108** (1): 71–82, [arXiv:1812.01938](https://arxiv.org/abs/1812.01938). [doi:10.1093/biomet/asaa052](https://doi.org/10.1093/biomet/asaa052)
- Davidson, Russell & MacKinnon, James G. (2004). *Econometric Theory and Methods*. New York: Oxford University Press. pp. 458–459. ISBN 978-0-19-512372-2.

---
Adapted from the Wikipedia article [Separation (statistics)](https://en.wikipedia.org/wiki/Separation_(statistics)) by Wikipedia contributors ([contributor history](https://en.wikipedia.org/wiki/Separation_(statistics)?action=history)). Available under [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/). Changes may have been made.