Optimize

stochopy.optimize.minimize(fun, bounds, x0=None, args=(), method='de', options=None, callback=None)

Minimize an objective function using a stochastic algorithm.

Parameters:

• fun (callable) – The objective function to be minimized. Must be in the form f(x, *args), where x is the argument in the form of a 1-D array and args is a tuple of any additional fixed parameters needed to completely specify the function.

• bounds (array_like) – Bounds for variables. (min, max) pairs for each element in x, defining the finite lower and upper bounds for the optimizing argument of fun. It is required to have len(bounds) == len(x). len(bounds) is used to determine the number of parameters in x.

• x0 (array_like or None, optional, default None) – Initial guess. Depending on the solver, array of real elements of size (ndim,) or with shape (popsize, ndim), where ndim is the number of independent variables and popsize is the total population size if the solver is population-based.

• args (tuple, optional, default None) – Extra arguments passed to the objective function.

• method (str, optional, default 'de') –

  Type of solver. Should be one of:

  • ’cmaes’

  • ’cpso’

  • ’de’

  • ’na’

  • ’pso’

  • ’vdcma’

• options (dict or None, optional, default None) –

  A dictionary of solver options. All methods accept the following generic options:

  • maxiter (int): maximum number of iterations to perform

  • seed (int or None): seed for random number generator

  • return_all (bool): set to True to return an array of all the solutions at each iteration

  • verbosity: fraction of population to consider in return_all

• callback (callable or None, optional, default None) – Called after each iteration. It is a callable with the signature callback(X, OptimizeResult state), where X is the current population and state is a partial stochopy.optimize.OptimizeResult object with the same fields as the ones from the return (except "success", "status" and "message").

Returns:

The optimization result represented as a stochopy.optimize.OptimizeResult. Important attributes are:

• x: the solution array

• fun: the solution function value

• success: a Boolean flag indicating if the optimizer exited successfully

• message: a string which describes the cause of the termination

Return type:

stochopy.optimize.OptimizeResult

Result

stochopy.optimize.OptimizeResult()

Represent the optimization result.

stochopy.optimize.x

The solution of the optimization.

Type:

array_like

stochopy.optimize.success

Whether or not the optimizer exited successfully.

Type:

bool

stochopy.optimize.status

Termination status of the optimizer. Its value depends on the underlying solver. Refer to message for details.

Type:

int

stochopy.optimize.message

Description of the cause of the termination.

Type:

str

stochopy.optimize.fun

The solution function value.

Type:

scalar

stochopy.optimize.nit

Number of iterations performed by the optimizer.

Type:

int

Note

There may be additional attributes not listed above depending of the specific solver.

Available optimizers

Competitive Particle Swarm Optimization

stochopy.optimize.cpso(fun, bounds, x0=None, args=(), maxiter=100, popsize=10, inertia=0.7298, cognitivity=1.49618, sociability=1.49618, competitivity=1.0, seed=None, xtol=1e-08, ftol=1e-08, constraints=None, updating='immediate', workers=1, backend=None, return_all=False, verbosity=1.0, callback=None)

Minimize an objective function using Competitive Particle Swarm Optimization (CPSO).

Parameters:

• fun (callable) – The objective function to be minimized. Must be in the form f(x, *args), where x is the argument in the form of a 1-D array and args is a tuple of any additional fixed parameters needed to completely specify the function.

• bounds (array_like) – Bounds for variables. (min, max) pairs for each element in x, defining the finite lower and upper bounds for the optimizing argument of fun. It is required to have len(bounds) == len(x). len(bounds) is used to determine the number of parameters in x.

• x0 (array_like or None, optional, default None) – Initial population. Array of real elements with shape (popsize, ndim), where ndim is the number of independent variables. If x0 is not specified, the population is initialized using Latin Hypercube sampling.

• args (tuple, optional, default None) – Extra arguments passed to the objective function.

• maxiter (int, optional, default 100) – The maximum number of generations over which the entire population is evolved.

• popsize (int, optional, default 10) – Total population size.

• inertia (scalar, optional, default 0.7298) – Inertial weight, denoted by w in the literature. It should be in the range [0, 1].

• cognitivity (scalar, optional, default 1.49618) – Cognition parameter, denoted by c1 in the literature. It should be in the range [0, 4].

• sociability (scalar, optional, default 1.49618) – Sociability parameter, denoted by c2 in the literature. It should be in the range [0, 4].

• competitivity (scalar, optional, default 1.) – Competitivity parameter, denoted by gamma in the literature. It should be in the range [0, 2].

• seed (int or None, optional, default None) – Seed for random number generator.

• xtol (scalar, optional, default 1.0e-8) – Solution tolerance for termination.

• ftol (scalar, optional, default 1.0e-8) – Objective function value tolerance for termination.

• constraints (str or None, optional, default None) –

  Constraints definition:

  • None: no constraint

  • ’Shrink’: infeasible solutions are repaired by shrinking particles’ velocity vector

• updating (str {'immediate', 'deferred'}, optional, default 'immediate') – If 'immediate', the best solution vector is continuously updated within a single generation. This can lead to faster convergence as candidate solutions can take advantage of continuous improvements in the best solution. With 'deferred', the best solution vector is updated once per generation. Only 'deferred' is compatible with parallelization, and is overridden when workers is not 0 or 1 or backend == 'mpi'.

• workers (int, optional, default 1) – The population is subdivided into workers sections and evaluated in parallel (uses joblib.Parallel). Supply -1 to use all available CPU cores.

• backend (str {'loky', 'threading', 'mpi'}, optional, default 'threading') –

  Parallel backend to use when workers is not 0 or 1:

  • ’loky’: disable threading

  • ’threading’: enable threading

  • ’mpi’: use MPI (uses mpi4py)

• return_all (bool, optional, default False) – Set to True to return an array with shape (nit, verbosity * popsize, ndim) of all the solutions at each iteration.

• verbosity (float, optional, default 1.0) – Fraction of population to consider in return_all. If 0.0, returns the best solution at each iteration.

• callback (callable or None, optional, default None) – Called after each iteration. It is a callable with the signature callback(X, OptimizeResult state), where X is the current population and state is a partial stochopy.optimize.OptimizeResult object with the same fields as the ones from the return (except "success", "status" and "message").

Returns:

The optimization result represented as a stochopy.optimize.OptimizeResult. Important attributes are:

• x: the solution array

• fun: the solution function value

• success: a Boolean flag indicating if the optimizer exited successfully

• message: a string which describes the cause of the termination

Return type:

stochopy.optimize.OptimizeResult

References

[1]

10. Kennedy and R. Eberhart, Particle swarm optimization, Proceedings of ICNN’95 - International Conference on Neural Networks, 1995, 4: 1942-1948

[2]

6. Van Den Bergh, An analysis of particle swarm optimizers, University of Pretoria, 2001

[3]

11. Luu, M. Noble, A. Gesret, N. Belayouni and P.-F. Roux, A parallel competitive Particle Swarm Optimization for non-linear first arrival traveltime tomography and uncertainty quantification, Computers & Geosciences, 2018, 113: 81-93

CMA-ES

stochopy.optimize.cmaes(fun, bounds, x0=None, args=(), maxiter=100, popsize=10, sigma=0.1, muperc=0.5, seed=None, xtol=1e-08, ftol=1e-08, constraints=None, workers=1, backend=None, return_all=False, verbosity=1.0, callback=None)

Minimize an objective function using Covariance Matrix Adaptation - Evolution Strategy (CMA-ES).

Parameters:

• fun (callable) – The objective function to be minimized. Must be in the form f(x, *args), where x is the argument in the form of a 1-D array and args is a tuple of any additional fixed parameters needed to completely specify the function.

• bounds (array_like) – Bounds for variables. (min, max) pairs for each element in x, defining the finite lower and upper bounds for the optimizing argument of fun. It is required to have len(bounds) == len(x). len(bounds) is used to determine the number of parameters in x.

• x0 (array_like or None, optional, default None) – Initial mean. Array of real elements of size (ndim,), where ndim is the number of independent variables.

• args (tuple, optional, default None) – Extra arguments passed to the objective function.

• maxiter (int, optional, default 100) – The maximum number of generations over which the entire population is evolved.

• popsize (int, optional, default 10) – Total population size.

• sigma (scalar or array_like, optional, default 0.1) – Initial standard deviation (as a fraction of feasible space defined by bounds).

• muperc (scalar, optional, default 0.5) – Number of parents (as a fraction of total population size).

• seed (int or None, optional, default None) – Seed for random number generator.

• xtol (scalar, optional, default 1.0e-8) – Solution tolerance for termination.

• ftol (scalar, optional, default 1.0e-8) – Objective function value tolerance for termination.

• constraints (str or None, optional, default None) –

  Constraints definition:

  • None: no constraint

  • ’Penalize’: infeasible solutions are repaired and their function values are penalized

• workers (int, optional, default 1) – The population is subdivided into workers sections and evaluated in parallel (uses joblib.Parallel). Supply -1 to use all available CPU cores.

• backend (str {'loky', 'threading', 'mpi'}, optional, default 'threading') –

  Parallel backend to use when workers is not 0 or 1:

  • ’loky’: disable threading

  • ’threading’: enable threading

  • ’mpi’: use MPI (uses mpi4py)

• return_all (bool, optional, default False) – Set to True to return an array with shape (nit, verbosity * popsize, ndim) of all the solutions at each iteration.

• verbosity (float, optional, default 1.0) – Fraction of population to consider in return_all. If 0.0, returns the best solution at each iteration.

• callback (callable or None, optional, default None) – Called after each iteration. It is a callable with the signature callback(X, OptimizeResult state), where X is the current population and state is a partial stochopy.optimize.OptimizeResult object with the same fields as the ones from the return (except "success", "status" and "message").

Returns:

The optimization result represented as a stochopy.optimize.OptimizeResult. Important attributes are:

• x: the solution array

• fun: the solution function value

• success: a Boolean flag indicating if the optimizer exited successfully

• message: a string which describes the cause of the termination

Return type:

stochopy.optimize.OptimizeResult

References

[1]

14. Hansen, The CMA evolution strategy: A tutorial, Inria, Université Paris-Saclay, LRI, 2011, 102: 1-34

Differential Evolution

stochopy.optimize.de(fun, bounds, x0=None, args=(), maxiter=100, popsize=10, mutation=0.5, recombination=0.9, strategy='best1bin', seed=None, xtol=1e-08, ftol=1e-08, constraints=None, updating='immediate', workers=1, backend=None, return_all=False, verbosity=1.0, callback=None)

Minimize an objective function using Differential Evolution (DE).

Parameters:

• fun (callable) – The objective function to be minimized. Must be in the form f(x, *args), where x is the argument in the form of a 1-D array and args is a tuple of any additional fixed parameters needed to completely specify the function.

• bounds (array_like) – Bounds for variables. (min, max) pairs for each element in x, defining the finite lower and upper bounds for the optimizing argument of fun. It is required to have len(bounds) == len(x). len(bounds) is used to determine the number of parameters in x.

• x0 (array_like or None, optional, default None) – Initial population. Array of real elements with shape (popsize, ndim), where ndim is the number of independent variables. If x0 is not specified, the population is initialized using Latin Hypercube sampling.

• args (tuple, optional, default None) – Extra arguments passed to the objective function.

• maxiter (int, optional, default 100) – The maximum number of generations over which the entire population is evolved.

• popsize (int, optional, default 10) – Total population size.

• mutation (scalar, optional, default 0.5) – The mutation constant. In the literature this is also known as differential weight, being denoted by F. It should be in the range [0, 2]. Increasing the mutation constant increases the search radius, but will slow down convergence.

• recombination (scalar, optional, default 0.9) – The recombination constant, should be in the range [0, 1]. In the literature this is also known as the crossover probability, being denoted by CR. Increasing this value allows a larger number of mutants to progress into the next generation, but at the risk of population stability.

• strategy (str, optional, default 'best1bin') –

  The differential evolution strategy to use. Should be one of:

  • ’rand1bin’

  • ’rand2bin’

  • ’best1bin’

  • ’best2bin’

• seed (int or None, optional, default None) – Seed for random number generator.

• xtol (scalar, optional, default 1.0e-8) – Solution tolerance for termination.

• ftol (scalar, optional, default 1.0e-8) – Objective function value tolerance for termination.

• constraints (str or None, optional, default None) –

  Constraints definition:

  • None: no constraint

  • ’Random’: infeasible solutions are resampled in the feasible space defined by bounds

• updating (str {'immediate', 'deferred'}, optional, default 'immediate') – If 'immediate', the best solution vector is continuously updated within a single generation. This can lead to faster convergence as candidate solutions can take advantage of continuous improvements in the best solution. With 'deferred', the best solution vector is updated once per generation. Only 'deferred' is compatible with parallelization, and is overridden when workers is not 0 or 1 or backend == 'mpi'.

• workers (int, optional, default 1) – The population is subdivided into workers sections and evaluated in parallel (uses joblib.Parallel). Supply -1 to use all available CPU cores.

• backend (str {'loky', 'threading', 'mpi'}, optional, default 'threading') –

  Parallel backend to use when workers is not 0 or 1:

  • ’loky’: disable threading

  • ’threading’: enable threading

  • ’mpi’: use MPI (uses mpi4py)

• return_all (bool, optional, default False) – Set to True to return an array with shape (nit, verbosity * popsize, ndim) of all the solutions at each iteration.

• verbosity (float, optional, default 1.0) – Fraction of population to consider in return_all. If 0.0, returns the best solution at each iteration.

• callback (callable or None, optional, default None) – Called after each iteration. It is a callable with the signature callback(X, OptimizeResult state), where X is the current population and state is a partial stochopy.optimize.OptimizeResult object with the same fields as the ones from the return (except "success", "status" and "message").

Returns:

The optimization result represented as a stochopy.optimize.OptimizeResult. Important attributes are:

• x: the solution array

• fun: the solution function value

• success: a Boolean flag indicating if the optimizer exited successfully

• message: a string which describes the cause of the termination

Return type:

stochopy.optimize.OptimizeResult

References

[1]

18. Storn and K. Price, Differential Evolution - A Simple and Efficient Heuristic for global Optimization over Continuous Spaces, Journal of Global Optimization, 1997, 11(4): 341-359

Neighborhood Algorithm

stochopy.optimize.na(fun, bounds, x0=None, args=(), maxiter=100, popsize=10, nrperc=0.5, seed=None, xtol=1e-08, ftol=1e-08, workers=1, backend=None, return_all=False, verbosity=1.0, callback=True)

Minimize an objective function using Neighborhood Algorithm (NA).

Parameters:

• fun (callable) – The objective function to be minimized. Must be in the form f(x, *args), where x is the argument in the form of a 1-D array and args is a tuple of any additional fixed parameters needed to completely specify the function.

• bounds (array_like) – Bounds for variables. (min, max) pairs for each element in x, defining the finite lower and upper bounds for the optimizing argument of fun. It is required to have len(bounds) == len(x). len(bounds) is used to determine the number of parameters in x.

• x0 (array_like or None, optional, default None) – Initial population. Array of real elements with shape (popsize, ndim), where ndim is the number of independent variables. If x0 is not specified, the population is initialized using Latin Hypercube sampling.

• args (tuple, optional, default None) – Extra arguments passed to the objective function.

• maxiter (int, optional, default 100) – The maximum number of generations over which the entire population is evolved.

• popsize (int, optional, default 10) – Total population size.

• nrperc (scalar, optional, default 0.5) – Number of resamplings (as a fraction of total population size).

• seed (int or None, optional, default None) – Seed for random number generator.

• xtol (scalar, optional, default 1.0e-8) – Solution tolerance for termination.

• ftol (scalar, optional, default 1.0e-8) – Objective function value tolerance for termination.

• workers (int, optional, default 1) – The population is subdivided into workers sections and evaluated in parallel (uses joblib.Parallel). Supply -1 to use all available CPU cores.

• backend (str {'loky', 'threading', 'mpi'}, optional, default 'threading') –

  Parallel backend to use when workers is not 0 or 1:

  • ’loky’: disable threading

  • ’threading’: enable threading

  • ’mpi’: use MPI (uses mpi4py)

• return_all (bool, optional, default False) – Set to True to return an array with shape (nit, verbosity * popsize, ndim) of all the solutions at each iteration.

• verbosity (float, optional, default 1.0) – Fraction of population to consider in return_all. If 0.0, returns the best solution at each iteration.

• callback (callable or None, optional, default None) – Called after each iteration. It is a callable with the signature callback(X, OptimizeResult state), where X is the current population and state is a partial stochopy.optimize.OptimizeResult object with the same fields as the ones from the return (except "success", "status" and "message").

Returns:

The optimization result represented as a stochopy.optimize.OptimizeResult. Important attributes are:

• x: the solution array

• fun: the solution function value

• success: a Boolean flag indicating if the optimizer exited successfully

• message: a string which describes the cause of the termination

Return type:

stochopy.optimize.OptimizeResult

References

[1]

13. Sambridge, Geophysical inversion with a neighbourhood algorithm - I. Searching a parameter space, Geophysical Journal International, 1999, 138(2): 479–494

Particle Swarm Optimization

stochopy.optimize.pso(fun, bounds, x0=None, args=(), maxiter=100, popsize=10, inertia=0.7298, cognitivity=1.49618, sociability=1.49618, seed=None, xtol=1e-08, ftol=1e-08, constraints=None, updating='immediate', workers=1, backend=None, return_all=False, verbosity=1.0, callback=None)

Minimize an objective function using Competitive Particle Swarm Optimization (CPSO).

Parameters:

• fun (callable) – The objective function to be minimized. Must be in the form f(x, *args), where x is the argument in the form of a 1-D array and args is a tuple of any additional fixed parameters needed to completely specify the function.

• bounds (array_like) – Bounds for variables. (min, max) pairs for each element in x, defining the finite lower and upper bounds for the optimizing argument of fun. It is required to have len(bounds) == len(x). len(bounds) is used to determine the number of parameters in x.

• x0 (array_like or None, optional, default None) – Initial population. Array of real elements with shape (popsize, ndim), where ndim is the number of independent variables. If x0 is not specified, the population is initialized using Latin Hypercube sampling.

• args (tuple, optional, default None) – Extra arguments passed to the objective function.

• maxiter (int, optional, default 100) – The maximum number of generations over which the entire population is evolved.

• popsize (int, optional, default 10) – Total population size.

• inertia (scalar, optional, default 0.7298) – Inertial weight, denoted by w in the literature. It should be in the range [0, 1].

• cognitivity (scalar, optional, default 1.49618) – Cognition parameter, denoted by c1 in the literature. It should be in the range [0, 4].

• sociability (scalar, optional, default 1.49618) – Sociability parameter, denoted by c2 in the literature. It should be in the range [0, 4].

• seed (int or None, optional, default None) – Seed for random number generator.

• xtol (scalar, optional, default 1.0e-8) – Solution tolerance for termination.

• ftol (scalar, optional, default 1.0e-8) – Objective function value tolerance for termination.

• constraints (str or None, optional, default None) –

  Constraints definition:

  • None: no constraint

  • ’Shrink’: infeasible solutions are repaired by shrinking particles’ velocity vector

• updating (str {'immediate', 'deferred'}, optional, default 'immediate') – If 'immediate', the best solution vector is continuously updated within a single generation. This can lead to faster convergence as candidate solutions can take advantage of continuous improvements in the best solution. With 'deferred', the best solution vector is updated once per generation. Only 'deferred' is compatible with parallelization, and is overridden when workers is not 0 or 1 or backend == 'mpi'.

• workers (int, optional, default 1) – The population is subdivided into workers sections and evaluated in parallel (uses joblib.Parallel). Supply -1 to use all available CPU cores.

• backend (str {'loky', 'threading', 'mpi'}, optional, default 'threading') –

  Parallel backend to use when workers is not 0 or 1:

  • ’loky’: disable threading

  • ’threading’: enable threading

  • ’mpi’: use MPI (uses mpi4py)

• return_all (bool, optional, default False) – Set to True to return an array with shape (nit, verbosity * popsize, ndim) of all the solutions at each iteration.

• verbosity (float, optional, default 1.0) – Fraction of population to consider in return_all. If 0.0, returns the best solution at each iteration.

• callback (callable or None, optional, default None) – Called after each iteration. It is a callable with the signature callback(X, OptimizeResult state), where X is the current population and state is a partial stochopy.optimize.OptimizeResult object with the same fields as the ones from the return (except "success", "status" and "message").

Returns:

The optimization result represented as a stochopy.optimize.OptimizeResult. Important attributes are:

• x: the solution array

• fun: the solution function value

• success: a Boolean flag indicating if the optimizer exited successfully

• message: a string which describes the cause of the termination

Return type:

stochopy.optimize.OptimizeResult

References

[1]

10. Kennedy and R. Eberhart, Particle swarm optimization, Proceedings of ICNN’95 - International Conference on Neural Networks, 1995, 4: 1942-1948

[2]

6. Van Den Bergh, An analysis of particle swarm optimizers, University of Pretoria, 2001

VDCMA

stochopy.optimize.vdcma(fun, bounds, x0=None, args=(), maxiter=100, popsize=10, sigma=0.1, muperc=0.5, seed=None, xtol=1e-08, ftol=1e-08, constraints=None, workers=1, backend=None, return_all=False, verbosity=1.0, callback=None)

Minimize an objective function using VD-CMA.

Parameters:

• fun (callable) – The objective function to be minimized. Must be in the form f(x, *args), where x is the argument in the form of a 1-D array and args is a tuple of any additional fixed parameters needed to completely specify the function.

• bounds (array_like) – Bounds for variables. (min, max) pairs for each element in x, defining the finite lower and upper bounds for the optimizing argument of fun. It is required to have len(bounds) == len(x). len(bounds) is used to determine the number of parameters in x.

• x0 (array_like or None, optional, default None) – Initial mean. Array of real elements of size (ndim,), where ndim is the number of independent variables.

• args (tuple, optional, default None) – Extra arguments passed to the objective function.

• maxiter (int, optional, default 100) – The maximum number of generations over which the entire population is evolved.

• popsize (int, optional, default 10) – Total population size.

• sigma (scalar or array_like, optional, default 0.1) – Initial standard deviation (as a fraction of feasible space defined by bounds).

• muperc (scalar, optional, default 0.5) – Number of parents (as a fraction of total population size).

• seed (int or None, optional, default None) – Seed for random number generator.

• xtol (scalar, optional, default 1.0e-8) – Solution tolerance for termination.

• ftol (scalar, optional, default 1.0e-8) – Objective function value tolerance for termination.

• constraints (str or None, optional, default None) –

  Constraints definition:

  • None: no constraint

  • ’Penalize’: infeasible solutions are repaired and their function values are penalized

• workers (int, optional, default 1) – The population is subdivided into workers sections and evaluated in parallel (uses joblib.Parallel). Supply -1 to use all available CPU cores.

• backend (str {'loky', 'threading', 'mpi'}, optional, default 'threading') –

  Parallel backend to use when workers is not 0 or 1:

  • ’loky’: disable threading

  • ’threading’: enable threading

  • ’mpi’: use MPI (uses mpi4py)

• return_all (bool, optional, default False) – Set to True to return an array with shape (nit, verbosity * popsize, ndim) of all the solutions at each iteration.

• verbosity (float, optional, default 1.0) – Fraction of population to consider in return_all. If 0.0, returns the best solution at each iteration.

• callback (callable or None, optional, default None) – Called after each iteration. It is a callable with the signature callback(X, OptimizeResult state), where X is the current population and state is a partial stochopy.optimize.OptimizeResult object with the same fields as the ones from the return (except "success", "status" and "message").

Returns:

The optimization result represented as a stochopy.optimize.OptimizeResult. Important attributes are:

• x: the solution array

• fun: the solution function value

• success: a Boolean flag indicating if the optimizer exited successfully

• message: a string which describes the cause of the termination

Return type:

stochopy.optimize.OptimizeResult

References

[1]

25. Akimoto, A. Auger and N. Hansen, Comparison-Based Natural Gradient Optimization in High Dimension, Proceedings of the 2014 conference on Genetic and evolutionary computation, 2014, 373-380