# Variance reduction

> Mediated Wiki article. Canonical URL: https://mediated.wiki/source/Variance_reduction
> Markdown URL: https://mediated.wiki/source/Variance_reduction.md
> Source: https://en.wikipedia.org/wiki/Variance_reduction
> Source revision: 1300775040
> License: Creative Commons Attribution-ShareAlike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/)

Mathematical procedure for reducing the variance of statistical estimators

This article is about a procedure for improving the precision of estimates. For the mathematical optimization method, see [Stochastic variance reduction](/source/Stochastic_variance_reduction).

The variance of randomly generated points within a unit square can be reduced through a stratification process.

In [mathematics](/source/Mathematics), more specifically in the theory of [Monte Carlo methods](/source/Monte_Carlo_method), **variance reduction** is a procedure used to increase the [precision](/source/Precision_(statistics)) of the [estimates](/source/Estimate_(statistics)) obtained for a given simulation or computational effort.[1] Every output random variable from the simulation is associated with a [variance](/source/Variance) which limits the precision of the simulation results. In order to make a simulation statistically efficient, i.e., to obtain a greater precision and smaller [confidence intervals](/source/Confidence_interval) for the output random variable of interest, variance reduction techniques can be used. The main variance reduction methods are

- common random numbers

- [antithetic variates](/source/Antithetic_variates)

- [control variates](/source/Control_variate)

- [importance sampling](/source/Importance_sampling)

- [stratified sampling](/source/Stratified_sampling)

- [moment matching](https://en.wikipedia.org/w/index.php?title=Moment_matching&action=edit&redlink=1)

- [conditional Monte Carlo](https://en.wikipedia.org/w/index.php?title=Conditional_Monte_Carlo&action=edit&redlink=1)

- and [quasi random variables](https://en.wikipedia.org/w/index.php?title=Quasi_random_variables&action=edit&redlink=1) (in [Quasi-Monte Carlo method](/source/Quasi-Monte_Carlo_method))

For simulation with [black-box](/source/Black-box) models [subset simulation](/source/Subset_simulation) and [line sampling](/source/Line_sampling) can also be used. Under these headings are a variety of specialized techniques; for example, particle transport simulations make extensive use of "weight windows" and "splitting/Russian roulette" techniques, which are a form of importance sampling.

## Crude Monte Carlo simulation

Suppose one wants to compute z := E ( Z ) {\displaystyle z:=E(Z)} with the random variable Z {\displaystyle Z} defined on the [probability space](/source/Probability_space) ( Ω , F , P ) {\displaystyle (\Omega ,{\mathcal {F}},P)} . Monte Carlo does this by sampling [i.i.d](/source/Independent_and_identically_distributed_random_variables). copies Z 1 , . . . , Z R {\displaystyle Z_{1},...,Z_{R}} of Z {\displaystyle Z} and then to estimate z {\displaystyle z} via the sample-mean estimator

- z ¯ = 1 n ∑ i = 1 n Z i {\displaystyle {\overline {z}}={\frac {1}{n}}\sum _{i=1}^{n}Z_{i}}

Under further mild conditions such as v a r ( Z ) < ∞ {\displaystyle var(Z)<\infty } , a [central limit theorem](/source/Central_limit_theorem) will apply such that for large n → ∞ {\displaystyle n\rightarrow \infty } , the distribution of z ¯ {\displaystyle {\overline {z}}} converges to a normal distribution with mean z {\displaystyle z} and standard error σ / n {\displaystyle \sigma /{\sqrt {n}}} . Because the standard deviation only converges towards 0 {\displaystyle 0} at the rate n {\displaystyle {\sqrt {n}}} , implying one needs to increase the number of simulations ( n {\displaystyle n} ) by a factor of 4 {\displaystyle 4} to halve the standard deviation of z ¯ {\displaystyle {\overline {z}}} , variance reduction methods are often useful for obtaining more precise estimates for z {\displaystyle z} without needing very large numbers of simulations.

## Common Random Numbers (CRN)

See also: [paired data](/source/Paired_data) and [Paired difference test](/source/Paired_difference_test)

The common random numbers variance reduction technique is a popular and useful variance reduction technique which applies when we are comparing two or more alternative configurations (of a system) instead of investigating a single configuration. CRN has also been called *correlated sampling*, *matched streams* or *matched pairs*.

CRN requires synchronization of the random number streams, which ensures that in addition to using the same random numbers to simulate all configurations, a specific random number used for a specific purpose in one configuration is used for exactly the same purpose in all other configurations. For example, in queueing theory, if we are comparing two different configurations of tellers in a bank, we would want the (random) time of arrival of the *N-*th customer to be generated using the same draw from a random number stream for both configurations.

## Underlying principle of the CRN technique

Suppose X 1 j {\displaystyle X_{1j}} and X 2 j {\displaystyle X_{2j}} are the observations from the first and second configurations on the *j-*th independent replication.

We want to estimate

- ξ = E ( X 1 j ) − E ( X 2 j ) = μ 1 − μ 2 . {\displaystyle \xi =E(X_{1j})-E(X_{2j})=\mu _{1}-\mu _{2}.\,}

If we perform *n* replications of each configuration and let

- Z j = X 1 j − X 2 j for j = 1 , 2 , … , n , {\displaystyle Z_{j}=X_{1j}-X_{2j}\quad {\mbox{for }}j=1,2,\ldots ,n,}

then E ( Z j ) = ξ {\displaystyle E(Z_{j})=\xi } and Z ( n ) = ∑ j = 1 , … , n Z j n {\displaystyle Z(n)={\frac {\sum _{j=1,\ldots ,n}Z_{j}}{n}}} is an unbiased estimator of ξ {\displaystyle \xi } .

And since the Z j {\displaystyle Z_{j}} 's are independent identically distributed random variables,

- Var ⁡ [ Z ( n ) ] = Var ⁡ ( Z j ) n = Var ⁡ [ X 1 j ] + Var ⁡ [ X 2 j ] − 2 Cov ⁡ [ X 1 j , X 2 j ] n . {\displaystyle \operatorname {Var} [Z(n)]={\frac {\operatorname {Var} (Z_{j})}{n}}={\frac {\operatorname {Var} [X_{1j}]+\operatorname {Var} [X_{2j}]-2\operatorname {Cov} [X_{1j},X_{2j}]}{n}}.}

In case of independent sampling, i.e., no common random numbers used then Cov(*X*1*j*, *X*2*j*) = 0. But if we succeed to induce an element of positive correlation between *X*1 and *X*2 such that Cov(*X*1*j*, *X*2*j*) > 0, it can be seen from the equation above that the variance is reduced.

It can also be observed that if the CRN induces a negative correlation, i.e., Cov(*X*1*j*, *X*2*j*) < 0, this technique can actually backfire, where the variance is increased and not decreased (as intended).[2]

## See also

- [Explained variance](/source/Explained_variance)

- [Regularization (mathematics)](/source/Regularization_(mathematics))

## References

1. **[^](#cite_ref-varred17_1-0)** Botev, Z.; Ridder, A. (2017). "Variance Reduction". *Wiley StatsRef: Statistics Reference Online*. pp. 1–6. [doi](/source/Doi_(identifier)):[10.1002/9781118445112.stat07975](https://doi.org/10.1002%2F9781118445112.stat07975). [hdl](/source/Hdl_(identifier)):[1959.4/unsworks_50616](https://hdl.handle.net/1959.4%2Funsworks_50616). [ISBN](/source/ISBN_(identifier)) [9781118445112](https://en.wikipedia.org/wiki/Special:BookSources/9781118445112).

1. **[^](#cite_ref-wolfram_2-0)** Hamrick, Jeff. ["The Method of Common Random Numbers: An Example"](http://demonstrations.wolfram.com/TheMethodOfCommonRandomNumbersAnExample/). *Wolfram Demonstrations Project*. Retrieved 29 March 2016.

- Hammersley, J. M.; Handscomb, D. C. (1964). *Monte Carlo Methods*. London: Methuen. [ISBN](/source/ISBN_(identifier)) [0-416-52340-4](https://en.wikipedia.org/wiki/Special:BookSources/0-416-52340-4). {{[cite book](https://en.wikipedia.org/wiki/Template:Cite_book)}}: ISBN / Date incompatibility ([help](https://en.wikipedia.org/wiki/Help:CS1_errors#invalid_isbn_date))

- Kahn, H.; Marshall, A. W. (1953). "Methods of Reducing Sample Size in Monte Carlo Computations". *[Journal of the Operations Research Society of America](/source/Journal_of_the_Operations_Research_Society_of_America)*. **1** (5): 263–271. [doi](/source/Doi_(identifier)):[10.1287/opre.1.5.263](https://doi.org/10.1287%2Fopre.1.5.263).

- MCNP — A General Monte Carlo N-Particle Transport Code, Version 5 Los Alamos Report LA-UR-03-1987

---
Adapted from the Wikipedia article [Variance reduction](https://en.wikipedia.org/wiki/Variance_reduction) by Wikipedia contributors ([contributor history](https://en.wikipedia.org/wiki/Variance_reduction?action=history)). Available under [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/). Changes may have been made.