An innovative approach to automatically identify control point set for model deformation rectification

Rectifying 3D model deformation based on control point set is one of the most important problems frequently applied in virtual reality fields, in which control point set is the key for implementing deformed manipulation. This paper presents a technique to automatically identify control point set by analyzing the changes of each point in a 3D model through its variations, then gathering clusters and selecting important points to become a control point set. Our proposed approach was tested and proved to be effective in our practical experiments with a deforming technique based on the interpolation of the radial basis function


Introduction
*Object deformation and rectifying the deformation has been one of the interesting research topics attracting the special attention of several researchers worldwide and there have been many different approaches existing. For instance, to test a design in a crash experiment, technicians tend to create its digital 3D models which can be used several times by adjusting its key parameters. The 3D models are then accordingly tested in the computational environment several times to measure their deformation before a real sample is experimented in practice. This will save a lot of resources and efforts. Or, cinematographers can usually create 3D computational scenes and environments, perform complex explosions and special tricks on computers and then merge scenes together as well as transform some actors/ actresses into other characters or even make certain transformations on the actors/ actresses themselves (Ansari and Abdel-Mottaleb, 2003;Hwang et al., 2011;Lee et al., 1995;Zhang et al., 2002;Fua, 1997;Blanz and Vetter, 1999;Akimoto et al., 1993;Ip and Yin, 1996;Lee et al., 2004;Lin et al., 2002). Moreover, there are still many professional applications relating to manipulating the object deformation and rectification (Fan et al., 2017;Rezende et al., 2016;Rock et al., 2015). Such practical problems have well resulted in the special research interests in computer graphics in general and rectifying 3D object deformation in particular.
One of the most frequently used to rectify the deformation of 3D models is control-point based approach (Cerveró et al., 2016;Jacobson et al., 2011;Hwang et al., 2012). The 3D object surface will be characterized and controlled by a set of points (control point sets), when we transform this set of points, the data of the object will change accordingly. This approach is usually done in two phases: the first phase is to determine the control point and the relationship between them and the entire data of the object, the second phase is to rectify the changes. In the rectification phase, a transformation function is firstly identified from the transformation of the control points; then, the function is applied to the entire object to obtain a rectified model. Hence, in the approach, control points play a crucial role. Consequently, it is really important to effectively select control points that are suitable for the type of objects that need to be modelled and modified.
Practically, the selection of suitable control points is based on the knowledge of experts who have made certain studies on the nature of the object of interest; for example, when studying scanned classes of medical photographs, the construction of points that show specific subjects such as bone sections, joints, brain, heart, etc. needs to have anatomically knowledgeable people and specialists who know where specific positions need special attention. Thus, depending on practical problem, different information from the control set can be exploited. For example, in the study of building facial animation from videos, Cao et al. (2014) used face control points not only in 3D modelling problem, but also to calibrate the camera matrix to increase the accuracy of the system. Similar studies can be found in (Hwang et al., 2012;Hwang et al., 2011;Ansari and Abdel-Mottaleb, 2003). Specifically, a training data set should include a set of object images marked with control points describing the instance of the interest objects in the images. Experts will use a computational tool to perform the mark-up on all images existing in the training set. In fact, in addition to this manual way, there are other approaches that can be automatic or semi-automatic; for example, when a relatively well-trained model set is obtained, it can be used to identify relevant control points in new images which can be then tested for their accuracy by an expert and some proper corrections can be done to improve the performance.
Typically, the control point set of an object is understood as a set of points that are distinct from other points and have consistency on different observations of an object. For example, if we focus on eyes in a face model, the control points preferably selected include eye corners which can be easily identified and marked. The nature of control points on a specific type of object requires certain knowledge of the object itself; thus, the selection is usually based on the expertise of a knowledgeable person who has a good understanding of the selected object.
The control point set is stored in MPEG-4 format which is an object-based multimedia data compression standard (Pandzic and Forchheimer, 2003). In image processing of facial models, as shown in Fig. 1, MPEG-4 characterizes faces with 84 feature points along with the activation parameters which well correspond to the actual actions of the faces and result in deformation of the face model compared to their neutral statuses. Presenting the deformation procedure of the facial model with some activation parameters within a time frame results in a sequence of facial animations. Key facial features correspond to the main positions on a human face usually considered include the eye muscles, eye position, nose and mouth. This control point set is chosen to reflect the effective movement of the human face. The key features are often arranged in groups such as cheeks, eyes, mouth, etc. The proposed control point sets have been successfully applied in many studies and practical applications on model modification. However, the control points are usually marked manually. To overcome this shortcoming, this paper aims to propose an innovative approach to automatically select control points based on observational patterns of object transformation. Hence, the next section of this paper briefly presents our proposed technique in clustering and selecting control points based on their transformation while Section 3 shows our empirical results obtained from practical tests.

Identifying control point set for model deformation rectification
Our practical observations show that when there is a change in a model, the transformation of the control points is interdependent, i.e. there are some points to be significantly changed and there are some with little change. For example, when we grab our skin and pull it up, the points near to grabbed points are varied the most whereas those far away from the grabbed point are varied the least. Thus, it can be considered that the most varied points will pull other points and the least varied points will retain other ones. From such observation, our approach will first find points with similar variations and thereby determine the strongest and weakest varied points to put into the control point set. Thus, the i th point of the 3D object will have M values corresponding to M observations. We call Ti = {ai0, ai1, ..., aiM-1} as the variable trajectory of the i th point in the observation set S. Thus, Ti correspond to the variations of the i th point in the observation set S of the 3D object. The determination of points with strong and weak changes will be done through the calculation of the trajectory of each point in the model.

Clustering transformation trajectory
From the set M, we calculate a set of variable trajectory of points in the model. In particular, each transformation trajectory respectively represents transformation of a point. We can then cluster the obtained trajectories into groups of points that have similar variations. Clustering techniques are built based on the algorithm K-means which is commonly used in clustering problems (Okabe and Yamada, 2018). Specifically, first, we calculate the distance between two transformation trajectories: Function: euclidDistPointTrajectory Input: 2 variable transformation trajectories T1, T2 Output: distance d Process:

d: = 0 2. foreach i in [0, |T1|) 3. d = d + euclid_distance(T1[i], T2[i]) 4. endfor 5. d: = d/|T1|
In each iteration, we need to recalculate the representation. To ensure the representative point be a point of the model, from the representative point calculated, for each cluster, we choose one point as a representative based on a specified criterion that it is the closest point to the calculated representative point. The distance is determined by the distance between the two mentioned trajectories. Specifically, the clustering algorithm of trajectories is as follows:

Selecting control points
After obtaining the clusters of model points, with each cluster, we choose points with the strongest and weakest changes as control points. The threshold is calculated based on a reference model. Thus, each point of the reference model corresponds to a variable trajectory. Corresponding to each point, we construct vector of deviation calculated by the deviation of each point in the trajectory against the corresponding point on the reference model. The threshold of the changes of each point is calculated with Euclid standard of the deviation vector corresponding to the point.

Combining with deformation rectification algorithm
The goal of identifying the control points is to serve the deformation rectification. Specifically, to select a good control point set, we need to incorporate an algorithm for deformation rectification and a set of variable models to serve the quality evaluation of the deformation rectification. General algorithm is as follows: In the algorithm, the function errs morph performs an error assessment between the target model and the transformation model from the reference model based on the given control set.

Experimental results
Before the practical experiments were conducted, the following components should be available: a rectification technique based on control point set, a 3D object for reference, a set of its variations to calculate and select control points and 1 set of object variations to evaluate the quality of selected control point set. The rectification technique used in our experiments is an approach using radial basic function (RBF) (Buhmann, 2003). And, we conducted experiments with many different values of K so that we can evaluate the model errors. Our experiments were conducted in two cases: (1) 3D objects which are spherical models created by computer graphical techniques, and (2) 3D models created from corresponding data of real human faces.

Spherical 3D models
3D object used in this tests is a spherical model with 482 vertices and 960 surfaces as shown in Fig.  3. Fig. 4 presents its distinct variants created by a 3D object transformation based on the random reference in the RBF rectification technique with a randomly generated control point set. Specifically, after a control point set is selected, coordinates of the control points are randomly transformed along the straight line connecting it to the center of the sphere in a certain proportional range. In our experiments, the proportion to radius of the sphere is chosen within [0.6, 1.4].
With the variant set for calculating and selecting control points, the trajectories of the points were first calculated before they are clustered with different K values to determine each region with similar variation. Fig. 5 briefly shows the clusters obtained in our experiments where points in the same cluster are marked with the same color.
With each cluster obtained from each K value, perform control point calculations and apply control points to transform the 3D model referenced by each sample in the set of variations for quality evaluation result. Each model rectified by RBF technique is then compared with the target model to calculate the model error. As the nature of the data in the 3D model and the trajectory of the points are all 3D point sets, the errors between the rectified models and the target one are also calculated through the distance function between the two variable trajectories. The error values of the control point set on the variable set for the model evaluation are calculated by the average error of each considered sample. Consequently, Fig. 6 presents the correlation between the errors and K values, where we can find that the errors are significantly reduced when the K is increased while Fig. 7 displays the rectification results based on our proposed approach with the spherical 3D object.

3D faces
3D object used in these tests is a face model with 3448 vertices and 6736 surfaces created from the samples in JAFFE database by Lyons et al. (1998). A typical example of the 3D face model is shown in Fig.   8. JAFFE database contains 213 images of 7 different facial expressions (6 basic facial expressions including joy, sadness, surprise, anger, indignation, fear, and a neutral status) recorded on 10 Japanese female models and taken at the Department of Psychology at Kyushu University. 60 Japanese students were asked to evaluate the images with 6 emotional states based on a 5-point scale where 5 denotes the highest level and 1 denotes for the lowest one. Corresponding to the 213 images in JAFFE dataset, we have 213 models of 3D faces. These models were then randomly divided into two distinct groups to be used in our tests which were carried out similarly to spherical 3D models mentioned in Section 3.1. Specifically, Fig. 9 shows some sample variants of the 3D faces created whereas Fig. 10 briefly shows the clusters obtained in our experiments where points in the same cluster are marked with the same color. In addition, Fig. 11 presents the correlation between the errors and K values while Fig. 12 displays the rectification results based on our proposed approach with the 3D faces.   From the above results, we can conclude that when increasing the value of K, the accuracy of our transformation and rectification is significantly improved regardless of simulated 3D spherical models or with 3D faces created from real-life photographs. As such, our proposed approach is not only effective in identifying reasonable control points and rectifying deformation within a given error range but also applicable to different variations on different types of objects.

Conclusion
Transforming 3D models based on control points has been widely applied in several practical industries and how to effectively identify control points to serve the rectification of model deformation is still an open issue that has attracted the special attention of researchers and practitioners in the field of image processing and virtual reality. This paper proposes an innovative technique to automatically identify control points based on an algorithm incorporated with RBF interpolation technique. Our proposed approach proves its good performance in our practical experiments with different databases. Thus, our future research will further validate its performance with more collectable deformations of more types of objects that can be modelled in 3D objects and combine with some other interpolation techniques.