# Meta-optimization

> Mediated Wiki article. Canonical URL: https://mediated.wiki/source/Meta-optimization
> Markdown URL: https://mediated.wiki/source/Meta-optimization.md
> Source: https://en.wikipedia.org/wiki/Meta-optimization
> Source revision: 1328884198
> License: Creative Commons Attribution-ShareAlike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/)

Meta-optimization concept.

**Meta-optimization** from [numerical optimization](/source/Numerical_optimization) is the use of one optimization method to tune another optimization method. Meta-optimization is reported to have been used as early as in the late 1970s by Mercer and Sampson[1] for finding optimal parameter settings of a [genetic algorithm](/source/Genetic_algorithm).

Meta-optimization and related concepts are also known in the literature as meta-evolution, super-optimization, automated parameter calibration, [hyper-heuristics](/source/Hyper-heuristics), etc.

## Motivation

Performance landscape for [differential evolution](/source/Differential_evolution).

Optimization methods such as [genetic algorithm](/source/Genetic_algorithm) and [differential evolution](/source/Differential_evolution) have several parameters that govern their behaviour and efficiency in optimizing a given problem and these parameters must be chosen by the practitioner to achieve satisfactory results. Selecting the behavioural parameters by hand is a laborious task that is susceptible to human misconceptions of what makes the optimizer perform well.

The behavioural parameters of an optimizer can be varied and the optimization performance plotted as a landscape. This is computationally feasible for optimizers with few behavioural parameters and optimization problems that are fast to compute, but when the number of behavioural parameters increases the time usage for computing such a performance landscape increases exponentially. This is the [curse of dimensionality](/source/Curse_of_dimensionality) for the search-space consisting of an optimizer's behavioural parameters. An efficient method is therefore needed to search the space of behavioural parameters.

## Methods

Meta-optimization of [differential evolution](/source/Differential_evolution).

A simple way of finding good behavioural parameters for an optimizer is to employ another overlaying optimizer, called the [meta](/source/Meta_(prefix))-optimizer. There are different ways of doing this depending on whether the behavioural parameters to be tuned are [real-valued](/source/Real_number) or [discrete-valued](/source/Discrete_mathematics), and depending on what performance measure is being used, etc.

Meta-optimizing the parameters of a [genetic algorithm](/source/Genetic_algorithm) was done by Grefenstette [2] and Keane,[3] amongst others, and experiments with meta-optimizing both the parameters and the [genetic operators](/source/Genetic_operators) were reported by Bäck.[4] Meta-optimization of the COMPLEX-RF algorithm was done by Krus and Andersson,[5] and,[6] where performance index of optimization based on information theory was introduced and further developed. Meta-optimization of [particle swarm optimization](/source/Particle_swarm_optimization) was done by Meissner et al.,[7] Pedersen and Chipperfield,[8] and Mason et al.[9] Pedersen and Chipperfield applied meta-optimization to [differential evolution](/source/Differential_evolution).[10] Birattari et al.[11][12] meta-optimized [ant colony optimization](/source/Ant_colony_optimization). [Statistical models](/source/Statistical_models) have also been used to reveal more about the relationship between choices of behavioural parameters and optimization performance, see for example Francois and Lavergne,[13] and Nannen and Eiben.[14] A comparison of various meta-optimization techniques was done by Smit and Eiben.[15]

## See also

- [Automated machine learning](/source/Automated_machine_learning) (AutoML)

- [Hyper-heuristics](/source/Hyper-heuristics)

## References

1. **[^](#cite_ref-mercer78adaptive_1-0)** Mercer, R.E.; Sampson, J.R. (1978). "Adaptive search using a reproductive metaplan". *Kybernetes*. **7** (3): 215–228. [doi](/source/Doi_(identifier)):[10.1108/eb005486](https://doi.org/10.1108%2Feb005486).

1. **[^](#cite_ref-grefenstette86optimization_2-0)** Grefenstette, J.J. (1986). "Optimization of control parameters for genetic algorithms". *IEEE Transactions on Systems, Man, and Cybernetics*. **16** (1): 122–128. [doi](/source/Doi_(identifier)):[10.1109/TSMC.1986.289288](https://doi.org/10.1109%2FTSMC.1986.289288). [S2CID](/source/S2CID_(identifier)) [23313487](https://api.semanticscholar.org/CorpusID:23313487).

1. **[^](#cite_ref-keane95genetic_3-0)** Keane, A.J. (1995). "Genetic algorithm optimization in multi-peak problems: studies in convergence and robustness". *Artificial Intelligence in Engineering*. **9** (2): 75–83. [doi](/source/Doi_(identifier)):[10.1016/0954-1810(95)95751-Q](https://doi.org/10.1016%2F0954-1810%2895%2995751-Q).

1. **[^](#cite_ref-back94parallel_4-0)** Bäck, T. (1994). "Parallel optimization of evolutionary algorithms". *Proceedings of the International Conference on Evolutionary Computation*. pp. 418–427.

1. **[^](#cite_ref-Krus03optimizing_5-0)** Krus, PK.; Andersson (Ölvander), J. (2003). "Optimizing optimization for design optimization". *Proceedings of DETC’03 2003 ASME Design Engineering Technical Conferences and Computers and Information in Engineering Conference Chicago, Illinois, USA*.

1. **[^](#cite_ref-Krus13optimizing_6-0)** Krus, PK.; Ölvander(Andersson), J. (2013). ["Performance index and meta-optimization of a direct search optimization method"](https://liu.diva-portal.org/smash/get/diva2:572570/FULLTEXT01.pdf) (PDF). *Engineering Optimization*. **45** (10): 1167–1185. [Bibcode](/source/Bibcode_(identifier)):[2013EnOp...45.1167K](https://ui.adsabs.harvard.edu/abs/2013EnOp...45.1167K). [doi](/source/Doi_(identifier)):[10.1080/0305215X.2012.725052](https://doi.org/10.1080%2F0305215X.2012.725052). [S2CID](/source/S2CID_(identifier)) [62731978](https://api.semanticscholar.org/CorpusID:62731978).

1. **[^](#cite_ref-meissner06optimized_7-0)** Meissner, M.; Schmuker, M.; Schneider, G. (2006). ["Optimized Particle Swarm Optimization (OPSO) and its application to artificial neural network training"](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1464136). *BMC Bioinformatics*. **7** (1): 125. [doi](/source/Doi_(identifier)):[10.1186/1471-2105-7-125](https://doi.org/10.1186%2F1471-2105-7-125). [PMC](/source/PMC_(identifier)) [1464136](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1464136). [PMID](/source/PMID_(identifier)) [16529661](https://pubmed.ncbi.nlm.nih.gov/16529661).

1. **[^](#cite_ref-pedersen08simplifying_8-0)** Pedersen, M.E.H.; Chipperfield, A.J. (2010). "Simplifying particle swarm optimization". *Applied Soft Computing*. **10** (2): 618–628. [CiteSeerX](/source/CiteSeerX_(identifier)) [10.1.1.149.8300](https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.149.8300). [doi](/source/Doi_(identifier)):[10.1016/j.asoc.2009.08.029](https://doi.org/10.1016%2Fj.asoc.2009.08.029).

1. **[^](#cite_ref-mason2017meta_9-0)** Mason, Karl; Duggan, Jim; Howley, Enda (2018). "A Meta Optimisation Analysis of Particle Swarm Optimisation Velocity Update Equations for Watershed Management Learning". *Applied Soft Computing*. **62**: 148–161. [doi](/source/Doi_(identifier)):[10.1016/j.asoc.2017.10.018](https://doi.org/10.1016%2Fj.asoc.2017.10.018).

1. **[^](#cite_ref-pedersen08thesis_10-0)** Pedersen, M.E.H. (2010). [*Tuning & Simplifying Heuristical Optimization*](https://web.archive.org/web/20200213130101/https://pdfs.semanticscholar.org/a5d2/8c26a2e2824170d67b69f14120cf67cabe26.pdf) (PDF) (PhD thesis). University of Southampton, School of Engineering Sciences, Computational Engineering and Design Group. [S2CID](/source/S2CID_(identifier)) [107805461](https://api.semanticscholar.org/CorpusID:107805461). Archived from [the original](https://pdfs.semanticscholar.org/a5d2/8c26a2e2824170d67b69f14120cf67cabe26.pdf) (PDF) on 2020-02-13.

1. **[^](#cite_ref-birattari02racing_11-0)** Birattari, M.; Stützle, T.; Paquete, L.; Varrentrapp, K. (2002). ["A racing algorithm for configuring metaheuristics"](https://www.researchgate.net/publication/220740639). *Proceedings of the Genetic and Evolutionary Computation Conference (GECCO)*. pp. 11–18.

1. **[^](#cite_ref-birattari04thesis_12-0)** Birattari, M. (2004). [*The Problem of Tuning Metaheuristics as Seen from a Machine Learning Perspective*](https://iridia.ulb.ac.be/~mbiro/paperi/BirattariPhD.pdf) (PDF) (PhD thesis). Université Libre de Bruxelles.

1. **[^](#cite_ref-francois01design_13-0)** Francois, O.; Lavergne, C. (2001). "Design of evolutionary algorithms - a statistical perspective". *IEEE Transactions on Evolutionary Computation*. **5** (2): 129–148. [doi](/source/Doi_(identifier)):[10.1109/4235.918434](https://doi.org/10.1109%2F4235.918434).

1. **[^](#cite_ref-nannen06method_14-0)** Nannen, V.; Eiben, A.E. (2006). ["A method for parameter calibration and relevance estimation in evolutionary algorithms"](http://srv.uib.es/pub/1012.pdf) (PDF). *Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation (GECCO)*. pp. 183–190.

1. **[^](#cite_ref-smit09comparing_15-0)** Smit, S.K.; Eiben, A.E. (2009). ["Comparing parameter tuning methods for evolutionary algorithms"](https://www.few.vu.nl/~gusz/papers/2009-CEC-tuning-methods.pdf) (PDF). *Proceedings of the IEEE Congress on Evolutionary Computation (CEC)*. pp. 399–406.

v t e Optimization: Algorithms, methods, and heuristics Unconstrained nonlinear Functions Golden-section search Powell's method Line search Nelder–Mead method Successive parabolic interpolation Gradients Convergence Trust region Wolfe conditions Quasi–Newton Berndt–Hall–Hall–Hausman Broyden–Fletcher–Goldfarb–Shanno and L-BFGS Davidon–Fletcher–Powell Symmetric rank-one (SR1) Other methods Conjugate gradient Gauss–Newton Gradient Mirror Levenberg–Marquardt Powell's dog leg method Truncated Newton Hessians Newton's method Optimization computes maxima and minima. Constrained nonlinear General Barrier methods Penalty methods Differentiable Augmented Lagrangian methods Sequential quadratic programming Successive linear programming Convex optimization Convex minimization Cutting-plane method Reduced gradient (Frank–Wolfe) Subgradient method Linear and quadratic Interior point Affine scaling Ellipsoid algorithm of Khachiyan Projective algorithm of Karmarkar Basis-exchange Simplex algorithm of Dantzig Revised simplex algorithm Criss-cross algorithm Principal pivoting algorithm of Lemke Active-set method Combinatorial Paradigms Approximation algorithm Dynamic programming Greedy algorithm Integer programming Branch and bound/cut Graph algorithms Minimum spanning tree Borůvka Prim Kruskal Shortest path Bellman–Ford SPFA Dijkstra Floyd–Warshall Network flows Dinic Edmonds–Karp Ford–Fulkerson Push–relabel maximum flow Metaheuristics Evolutionary algorithm Hill climbing Local search Parallel metaheuristics Simulated annealing Spiral optimization algorithm Tabu search Software

---
Adapted from the Wikipedia article [Meta-optimization](https://en.wikipedia.org/wiki/Meta-optimization) by Wikipedia contributors ([contributor history](https://en.wikipedia.org/wiki/Meta-optimization?action=history)). Available under [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/). Changes may have been made.