Adaptive Hooke-Jeeves-evolutionary algorithm for linear equality constrained problems

This paper proposes a novel hybrid algorithm called Genetic Algorithm based Simplex Adaptive Hooke and Jeeves (GA-SAHJ) method for solving equality constrained non-linear optimization problems. The proposed hybrid technique uses Genetic Algorithm (GA) as the global optimizer and a modified Hooke and Jeeves method for further refinements of the current solution within the landscape of a feasible region. The convergence proof of the modified approach is also provided. The effectiveness of the proposed GA-SAHJ method is demonstrated by applying it on six test instances each involving at least one equality constraint. The results witness that the proposed hybrid approach is capable of producing highly accurate and fully feasible solutions of the considered problems.


Introduction
*The general optimization problem can be written as follow: subject to h j ( x ) = 0, (j = 1,2, ⋯ , J) g k ( x ) ≤ 0, (k = 1,2, ⋯ , K) where f( x ), h j ( x ) and g k ( x ) are functions of the design vector: x = (x 1 , x 2, ⋯ , x d ) T and x ∈  d . Here each component x i of x are called design or decision variables, and they can be real continuous, discrete, or a mix of these two. The functions f(x) is called the objective function or simply cost functions. The space spanned by the decision variables is called the design space or search space, whereas the space formed by the objective function values is called the solution space or response space. The equalities for h j ( x ) and inequalities for g k ( x ) are called constraints (Koziel and Yang, 2011).
It has been reported that the approaches like gradient based methods, linear programming and sequential quadratic programming perform poorly due to non-availability of derivatives, discontinuities and multiple local minima (Miettinen et al., 2003). The derivative free methods, Hooke-Jeeves, Nelder-Mead Simplex and multidirectional search methods were used to obtain the optimal truss designs . Tabassum et al. (2015) optimized the design of oxygen production system by using the Hooke-Jeeves, Nelder-Mead Simplex method (Tabassum et al., 2015) and the same problem by differential evolution along with 2-paramer exponential penalty function approach that gave slightly better results . There exist some examples on which the Nelder-Mead Simplex Algorithm method failed to find optimal solutions. Ali et al. (2017) proposed a simplex volume based novel strategy for rescuing the method from stagnations or complete failures. There are examples where these methods have successfully been applied to the problems involving variables up to few hundreds. In recent decades, DFMs have successfully been applied in multiple areas of engineering design, scheduling, bio-systems, molecular biology, neural networks, decision making and Image processing problems. Some well-known DFMs can be found in (Conn et al., 2009;Belegundu and Chandrupatla, 2011;Deb, 2004;Rao, 1996). These methods highly depend on provided initial guess to start and aim at local search. Moreover their performances depend strongly on the dimensions of optimization problems. Generally derivative free algorithms are designed for unconstrained optimization; therefore, they cannot be applied to constrained problems directly. The application of DFMs to an unconstrained problem (A) is possible only if an equivalent unconstrained is formed. Among the several methods of constraint handling approaches, penalties functions are the most common approaches in the evolutionary algorithms to handle almost all of the types of constraints (equality and inequality). This approach embeds constraints into the objective function and designs an equivalent unconstrained problem. The embedded part is comprised of positive multiples of degrees of constraint violations act as penalization to the algorithm for exploring any infeasible solution (Bertsekas, 1982;Joines and Houck, 1994;Michalewicz and Janikow, 1991).
Penalty functions also inherit some drawbacks, such as non-optimal tuning of penalty parameters and raised complexity, high non-linearity, nonconvexity or non-differentiability of resulting objective function. For example the main drawback of both of annealing penalty (Kirkpatrick et al., 1983) and static penalty function is a high number of parameters which require careful fine tune for better results (Michalewicz, 1995). In dynamic penalties the quality of the obtained solution is very sensitive to changes in its fixed constants and there are no clear guidelines regarding the sensitivity of the approach to different values of constants (Smith and Coit, 1997). The drawback of adaptive penalties is the issue of choosing the generational gap that provides reasonable information to guide the search (Rasheed, 1998). In the practical applications of coevolutionary penalties in EAs, there arise necessities of four additional highly sensitive parameters (Coello, 1999;2000), making it difficult to use. Death penalty is confined by not explaining and exploiting any kind of information from the infeasible points that can be produced by the EA to supervise and lead the search (Coit and Smith, 1996). In In genetic algorithms a 2-paramer exponential penalty function approach was used for handling equality and inequality constraints which provided comparable results on various well known constraint optimization problems (Chaudhry et al., 2009). Boudjehem et al. (2011) gave the idea to reduce the dimension of the optimization problem to a monodimensional that equipped new algorithm with the ability to determine a narrow space around the global optimum by reducing the number of evaluations.
Genetic algorithm (GA) and Hooke and Jeeves (HJ) method both are not free from encountering similar drawbacks when applied to the penalized objective function, especially when linear equality constraint is accompanied. Linear equality constraint constitutes narrow feasible region and hence appears to be resistant to global explorations of GA and local explorations of the moves of HJ method which can be noticed from Fig. 1. In this way GA and HJ may fail without making a single improvement in the objective function value.
To overcome the shortcomings of HJM and GA on equality constrained problems their hybrid approach is needed to be constructed. Hybridization of different algorithms by uniting their strengths is an effective approach in meta-heuristics, especially in evolutionary algorithms. The examples of such hybridization can be found in (Miettinen et al., 2003;Bertsekas, 1982;Coit and Smith, 1996;Hooke and Jeeves, 1961;Coelho and Mariani, 2009). Pandian (2010) proposed a meta-hetaeristic approach by hybridizing simulated annealing and genetic algorithm to search for the best feasible solutions to the decision variables for solving a nonlinear objective function in industrial management problems.

Fig. 1: Stagnation of HJM to a non-optimal point
In this paper we modify HJM by utilizing advantages of equality constraints rather than eliminating it. The modified HJM is named as Simplex Adaptive HJM (SAHJ) method. The proposed SAHJM is further hybridized with GA for global search and the resulting method is named as Genetic Algorithm based Simplex Adaptive Hooke-Jeeves (GA-SAHJ) method.
Rest of the paper is organized as follows. Section 2 exhibits the basic concepts, definitions and the construction of SAHJ algorithm, in Section 3 the convergence proof of SAHJ for unconstrained optimization problems is presented; Section 4 consists of flowchart of the proposed hybrid GA-SAHJ algorithm and numerical examples are presented in Section 5. In the last section, some concluding remarks are also presented.

Simplex adaptive Hooke and Jeeves method (SAHJ)
The proposed modification concerns with the constrained problems involving a linear equality constraint of the form: Such a constraint defines hyper-plane with a normal u in the search space. For each of such hyper planes one or more members of the basis of search space become redundant. Therefore, the effective dimensions of the feasible region determined by (1) become lesser than those of the optimization problem. We require a set of linearly independent vectors which span the feasible region and lie in planes parallel to (1). Once an initial guess lying in the feasible region is found then the proposed scheme guarantees that no infeasible point is explored. Following assumptions are important for practicability of the scheme.

Assumption 2.2:
The normal u is such that u i ≠ 0 ∶ 1 ≤ i ≤ n.
The assumption 2.1 is quite possible because otherwise a parallel hyper-plane with arbitrary nonzero b can be used. The proposed modified algorithm requires bases of the hyper-plane through origin and parallel to (1). We will refer such bases as parallel bases. The assumption 2.2 is concerned with the case when any u i = 0 which will be addressed in coming sections. The equality constraint (1) can be expressed as: Assumptions 1 and 2 guarantee the existence of distinct points These points along with the origin form a nondegenerate simplex in R n . The face opposite to the origin is a part of the hyper-plane (1). Replacing the origin by centroid of its opposite face we get a degenerate simplex in R n but it is a non-degenerate n-simplex in hyper-plane (1) whose active dimensions are (n-1).
Definition 2.1: By inactive search direction we mean any direction d such that Inactiveness is based on the fact that such a direction does not help in reducing the efforts of exploration of the points in the feasible region of the problem involving (1).

SAHJ algorithm
The following steps interpret the exploitation of the sub-simplex for determination of search directions which form the active spanning set of the hyper-plane (1).
Step 2: (a) Translate x (o) to the hyper-plane by (d) Computev i = G − P (i) , 1 ≤ i ≤ n and construct n possible parallel bases of ( n − 1) distinct i .
Step 3: Choose m th parallel basis and apply Gram-Schmidt process to get d i ∶ 1 ≤ i ≤ (n − 1) .
Step 5: Perform an exploratory move on x b to get x.
If f( x ) < f( x b ), set x e = x and Go to step 6; else set = . Set m = mod(m, n) + 1 and go to Step 3.
Step 6: Findx p = x b + λ ( x e − x b ) and apply another exploratory move with x p as the base point. Suppose x E is the result. Go to step 7.
Step 7: If f( x E ) < f( x e ), set x b = x e , x e = x E and Go to step 6; else set x b = x e , = α and Go to step 4.
The proposed SAHJ differs from HJ method in following three aspects.
(i) The step length reduction parameter α is chosen randomly in the range [0.5, 1). (ii) The search directions may not be along coordinate axes and are fewer than the dimensions. (iii) The set of search directions is updated after every failed exploratory move at step 5.

Convergence of SAHJ method
Before establishing main convergence results of the algorithm we state and prove two important results which will be helpful in proving convergence of the proposed algorithm. Which shows that every v i lies in some hyperplane parallel to (1).

Claim 4.2:
The algorithm SAHJ can generate the points lying only in the hyper-plane (1).

Proof:
The step 2 (a), 2(b) ensure that SAHJ starts with a feasible initial guess. Consider the initial guess be x b . Claim 3.1 guarantees that step 2 (d) generate only active search directions. On contrary suppose that there is a step length ∆ k and a direction d k such that the point x b + ∆ k d k generated by exploratory move does not lie in the feasible region of (7), i.e., That is contradiction to the activeness of d k . Now we step forward to our main convergence results with following two assumptions.  Proof: The initial guess provided to the algorithm lies within the box constraints.
Step 2 (a) translates the initial guess to the feasible region. By claim 3.1 the algorithm generates only active search directions and claim 3.2 ensures that the algorithm can generate only feasible points. Algorithm generates a set of (n -1) linearly independent active search direction. This set can be extended to a basis of by including only one element, namely = | | . Every exploratory move along with non-zero step length generates an infeasible point. Using penalized objective function such a point results in an unsuccessful exploratory move. Suppose at any iteration the current base point be ( ) .
The proof is dependent on the successful and unsuccessful exploratory moves. Suppose that the exploratory moves are unsuccessful for iterations + 1, + 2, + 3, …. Then for any j > , the current base point ( ) is the same as ( ) and is such that: as → ∞ then ( −1) → 0 this implies that from (2) and (3) ( ( ( ) )) . = 0 , = 1, 2, 3, … , ( − 1) Now we consider that ( ) is not a stationary point. Then after a finite number 1 of iterations, the exploratory move must be successful otherwise a reduction in step length parameter occurs. Continuing the process we get a strictly decreasing sequence { ( ) } such that Since ( )is bounded below so due to monotonicity ( ( ) ) converges to its lower bound. The compactness of feasible region implies that there is a feasible point * such that Considering the unsuccessful exploratory moves with base point * and by the virtue of (2) and (3) we arrive at: Which shows that ( ( * )) is orthogonal to every active direction. Hence there is a non-zero constant * such that This proves the existence of constrained stationary point of the problem.
We have shown that in case of all failed exploratory moves the algorithm converges to a stationary point and successful exploratory moves provide descent change which leads to existence of a stationary point.

Extension to penalized objective function
The algorithm SAHJ can converge to a constrained minimum point of the said problem when the assumption 2.2 is not fulfilled but the search directions are constructed in a special way. For illustration consider = 0 for some ∈ {1, 2, 3, … , } then the hyper-plane (1) is parallel to ℎ coordinate axis. In this case step 2 (d) can generate only (n -1) search directions. The ℎ search direction is calculated as: provided ( ) is the unit vector along ℎ coordinate axis. Along with such extensions the above convergence results are also valid.
The unconstrained penalized objective function used here is of the form (Bertsekas, 1982;Runarsson and Yao, 2000): where, ( )is the penalty function and ℛ is the penalty factor. For every feasible solution ℱ( )coincides with ( )and for every infeasible solution ℱ( ) > ( ).
The convergence for penalized function ℱ( ) can also be proved. There is coincidence between ℱ( )and ( )for explorations along all the active search directions. While using penalized function the exploration is made along the members of a basis of the search space which can be obtained by including a unit vector ̂ along in the parallel basis of the hyper-plane through origin. The resulting linear basis and the counter parts of its members form a maximal positive basis for the search space. In this way a finite set { : 1 ≤ ≤ } of positive bases is formed and exploited between steps 3 to step 5. The proof of existence of a stationary point for penalized function ℱ( ) is based on its differentiability and use of finite set of maximal positive bases. The proof of convergence of SAHJ on ℱ( ) coincides with the one convergence proof of directional direct search method (Conn et al., 2009).

Genetic algorithm based simplex adaptive Hooke and Jeeves algorithm (GA-SAHJ)
Genetic Algorithm (GA) is a well-practiced evolutionary global search algorithm. We associate GA with SAHJ to design global search method which ultimately explores the local search to converge to the global solution. The resulting combined approach is named as Genetic Algorithm based Simplex Adaptive Hook and Jeeves (GA-SAHJ) method for further use. The flow chart of the hybrid algorithm is given in Fig. 2.
In the following subsection we present numerical results for evaluating the efficiency of the proposed algorithm on selected test cases.

Numerical performances
For comparisons of performances of GA-SAHJ and GA-HJ we considered some test problems from the literature. For both of the methods the penalized objective functions given by (10) were constructed by using approach in (Michalewicz and Schoenauer, 1996) for inequality constraints and a square penalty function given in (Deb, 2004) for equality constraint. For numerical implementations of GA-HJ and GA-SAHJ methods, step length used were Δi=1, the step length reduction parameter α = 0.5 for test problems 1-6 and two termination criteria (i) ξ = 10 −20 and (ii) number of function evaluations becomes larger than 10000 × (the number of variables).
GA is implemented through optimization toolbox in MATLAB (Chipperfield et al., 1994) with parameters: Population size = 50, crossover fraction = 0.8, mutation rate = 0.01, elite count = 2. Remaining parameters used are same as the default settings. Six test problems of constrained minimization were taken from (Hock and Schittkowski, 1981;Himmelblau, 1972). GA is run for 20 to 30 generations and the obtained final point was retained. 200 iterations of both of SAHJ and HJ were recorded with retained point as initial guess. The Figs. 3-8 show the convergence of three approaches GA-SAHJ, GA-HJ and GA.

Conclusion
This paper presented a convergent hybrid optimization method namely Genetic Algorithm based Simplex Adaptive Hooke and Jeeves Algorithm (GA-SAHJ) to efficiently solve the equality constrained optimization problems. The proposed algorithm guarantees to return a stationary point of the equality constrained problems. The effectiveness of the GA-SAHJ has been demonstrated through six benchmark problems taken from the literature. The convergence curves in Figs. 3-8 show the superior convergence speed of GA-SAHJ over GA and GA-HJ. Moreover, the statistical results in Table 1 witness that best and mean fitness values produced by GA-SAHJ are highly better than those of GA-HJ under the similar implementation conditions. These results emphasize the capabilities of the GA-SAHJ for such problems and encourage its further implementation to other fields as well as engineering problems. As a future direction, it is intended to hybridize the proposed SAHJ with modern metaheuristics for solving economic dispatch problem with valve point loading effect resembling with the considered test instances. Additionally, suitable linearization schemes can be used to solve nonlinear equality constrained problems with the proposed scheme.