Metamodeling techniques for CPU-intensive simulation-based design optimization: a survey

Abstract

In design optimization of complex systems, the surrogate model approach relying on progressively enriched Design of Experiments (DOE) avoids efficiency problems encountered when embedding simulation codes within optimization loops. However, an efficient a priori sampling of the design space rapidly becomes costly when using High-Fidelity (HF) simulators, especially in high dimension. On the other hand, in applications such as aeronautical design, multiple simulation tools are frequently available for the same problem, generally with a degree of precision inversely proportional to the CPU cost. Thus, the concept of multi-fidelity proposes to merge different levels of fidelity within a single model with controlled variance. Based on recent Reduced-Order Modeling (ROM) techniques, an alternative approach allows to pursue the objective of mastering the simulation budget by replacing costly models with their approximate full-field counterparts, providing additional insight to scalar surrogates built directly from the Quantities of Interest (QoI). Both approaches: multi-fidelity and ROM, may be combined, allowing for additional flexibility in choosing the degree of fidelity required in different zones of the design space. This paper reviews the strategies that seek to improve surrogate-based optimization efficiency, including ROM, multi-fidelity metamodeling, and DOE enrichment strategies.

Introduction

Several computational methods with varying fidelity¹ have been developed over the past decades to simulate fluid dynamics [97]. High-fidelity (HF) simulation models such as 3D-Reynolds Averaged Navier-Stokes (RANS) and Large Eddy Simulations (LES) have reached a level of maturity that allows them to be sufficiently predictive for use in aeronautical part design optimization loops [158]. However, these imply extensive computer resource utilization ranging from hours to full days of computation on supercomputer architectures and the generation of several gigabytes of data. On the other hand, Low-Fidelity (LF) models such as simplified physics, categorization [72, 151], mesh coarsening [20, 115] or relaxed convergence criteria [52], can be used to compute fluid flow approximations using limited resources. The Multi-Fidelity (MF) optimization approach combines the LF’s and HF’s respective advantages regarding cost and precision within a common framework. The first category of MF Models (MFMs) consists of exploring the LF values to determine the most interesting zones in the design space (for example the region of interest for an optimization problem) to reduce the calls to the expensive HF solver. This type of methods is not properly referred to as MFMs, since the HF and LF levels are not merged into a single model. The MF combination techniques use most of the time a surrogate model to integrate multiple fidelity levels.

Surrogate modeling consists in building a regression model² from a set of available samples obtained from a design of experiments (DOE) allowing to predict the values of the function at interesting points. It avoids repeated calls to the simulation software in the design loops. Applications are uncertainty analysis, statistical inference (data-driven) [53, 220] or multi-disciplinary [48, 50, 132, 202], shape [90, 199, 226] or topology optimization [142]. Reviews on Surrogate-Based Optimization (SBO) are provided by [73, 163, 189, 195] where references on DOE techniques, surrogate modeling, and SBO applications are detailed. Later, SBO techniques applied to the aerodynamic field were reviewed by [232], and a more general review was published by [231] on surrogates, DOE, and adaptive strategies in engineering applications. In the present review, we focus on the construction of surrogate models when multiple levels of fidelity are available, in the SBO context.

Alternatively to the SBO frameworks, the evaluation costs may be also reduced by two families of Reduced-Order Models (ROMs): the intrusive projection-based models [8, 18, 238] and Non-Intrusive ROMs (NIROMs) [65, 224]. These methods have provided important savings in the computational cost of CFD models, but also in Numerical Structural Mechanics (CSM) especially in optimization [225], Uncertainty Quantification (UQ) [43, 114] or inverse resolution [62]. Other approaches combine the MFM and ROM within a surrogate to solve costly optimization [20, 60, 103]. The use of multiple fidelities has been introduced to temper the cost of high-dimensional multi-disciplinary design breaking the ”curse of dimensionality” [189, 209]. This concept allows to predict improved design location by taking into account a large amount of LF cheap data and few expensive HF available data.

However, both ROM and MFM based SBO raise predictability issues when ROM-based surrogates rely on offline sampling, leading to costly optimization procedures [45, 82, 204]. To address this issue, offline-online strategies, also called adaptive strategies, are employed to adapt the ROM as the optimization search progresses. These approaches have recently been highlighted by [231] in its state-of-the-art paper on SBOs as one of the current solutions to the challenges of large-scale modeling. The enrichment criterion uses a merit function to predict both the improvement in accuracy and the optimization objective. The next point to be evaluated is selected to maximize this criterion iteratively until an adequate stopping condition is reached. The goal is to choose the most relevant points in order to reduce the number of calls to solvers, that would answer a given design problem. The strategies for selecting sampling points can be classified as infill sampling [82, 207] adaptive [82, 123, 219], response-based [27], a posteriori, sequential and online approaches. Contrary to the domain-based and space-filling approaches where the training database is selected according to its point-to-point distance distribution, the adaptive selection is based on the surrogate model’s information.

The solutions to the CPU challenge of repetitive costly simulations reviewed in this paper aim at reducing the training and evaluation costs of metamodels. On the one hand, the MFMs combine variable simulation costs and, on the other hand, the ROMs allow to handle high-dimensional outputs while taking into account the vector representation of the simulation data (typically the discretized solutions of fluid state variables in CFD). Compared to scalar approximation models of Quantities of Interest (QoI), they are supposed to provide better insight into the physical model [47]. Also, sampling methods can be used to minimize the number of calls to expensive solvers while maintaining sufficient representativeness of the simulated physical model.

This paper reviews scalar and full-field SBO single- and MF frameworks. “Overview on surrogate-based optimization” section presents the SBO methods. Scalar MFMs are presented in “Multi-fidelity management” section, ROMs in “Surrogate modeling for full-field computations” section, and finally, vectorial MFMs based on ROMs in “Multi-fidelity reduced-order methods” section. For each surrogate, the corresponding adaptive sampling strategies are overviewed.

Overview on surrogate-based optimization

This section presents surrogate-Based optimization (SBO). The surrogate models are presented on “Surrogate modeling” and “Off-line design of experiment” sections introduces general and non-adaptive sampling methods and “Adaptive sampling” section the adaptive sampling approaches.

In the context of industrial design, the Quantities of Interest (QoI) are used to determine whether the chosen technological solution meets the functional requirements. A performance criterion generally referred to as an objective function $\mathcal {J}(\varvec{\chi })$ can be evaluated at the design point ${\varvec{\chi }}^T=(\chi _1\,... \,\chi _d)$, where d is the design space dimension. This function is minimized under one or more constraints to meet the design specification. The scalar QoIs (lift, drag,...) are post-processed from the simulator output $f(\varvec{\chi })$ (velocity, pressure field,...). The classical optimization problem is defined in a design space $\mathcal {D}$ by the objective $\mathcal {J}$, equality and inequality constraints h and g respectively

$$\begin{aligned} \begin{array}{cl} \displaystyle \quad \varvec{\chi }^{*}=\mathop {{{\,\mathrm{\hbox {arg}\,\hbox {min}}\,}}}\limits _{\varvec{\chi } \in \mathcal {D}} &{} \mathcal {J}(f(\varvec{\chi }))\\ \text {s.t.} &{} g_{i}(f(\varvec{\chi })) \le 0,\quad i=1,...,p\\ &{} h_{j}(f(\varvec{\chi }))=0, \quad j=1,...,q\\ &{}\\ \end{array} \end{aligned}$$

(1)

where $\varvec{\chi }$ is the design variable, $\mathcal {D}\subset \mathbb {R}^{d}$, $\varvec{\chi }^{*}$ is the optimum of the objective function $\mathcal {J} \in \mathbb {R}$ subjected to the constraints g, h.

In simulation-based optimizations, the QoI are evaluated by post-processing simulation results, requiring repetitive calls to time-consuming software within the optimization loops. Therefore, the SBO consists in solving the approximate problem defined by the approximate quantities noted as $\widetilde{\mathcal {J}}$, $\widetilde{g}$, and $\widetilde{h}$, respectively

$$\begin{aligned} \begin{array}{clclcl} \displaystyle \quad \varvec{\tilde{\chi }}^{*}=\mathop {{{\,\mathrm{\hbox {arg}\,\hbox {min}}\,}}}\limits _{\varvec{\chi } \in \mathcal {D}} &{} \widetilde{\mathcal {J}}(\varvec{\chi })\\ \text {s.t.} &{} \widetilde{g_{i}}(\varvec{\chi }) \le 0,\quad i=1,...,p\\ &{} \widetilde{h_{j}}(\varvec{\chi })=0, \quad j=1,...,q\\ {} &{} \\ \end{array} \end{aligned}$$

(2)

The SBO steps are illustrated in Fig. 1. First, is the off-line phase, samples are evaluated $\varvec{X} = \{\varvec{\chi }^{(0)}\,,... \,,\varvec{\chi }^{(M)}\}$ using design of experiments (DoE) methods as reviewed by [77, 195] or, in the specific field of aerodynamic applications, by [232]. Once the simulation has evaluated the samples $\textit{f}(\varvec{\chi }^{(i)})$ with $i=\{1,\,... \,, M\}$, the surrogate model is built for the QoI $\mathscr {Q} = \{\widetilde{\mathcal {J}},\widetilde{g},\widetilde{h}\}$ and the optimization is performed in the online phase.

Fig. 1

The offline SBO strategy

Surrogate modeling

The approximated quantities are modeled with a surrogate model, or metamodels, for which common techniques include

• polynomial regression [192, 194],

• Gaussian Process Regression (GPR) including Kriging [55, 107, 134], developed in the geostatistical domain by [111],

• Radial Basis Functions (RBF) [68, 81, 160],

• Moving Least Squares (MLS) [48, 113],

• Support Vector Regression (SVR) [201],

• Artificial Neural Networks (ANN) [37].

Although polynomials are extensively cited in the literature, their use is mainly limited to low-dimensional, linear or quadratic cases [189]. RBF and Kriging metamodels can handle non-linear engineering approaches. While RBF imply isotropic kernels, Kriging models that are based on the optimization of anisotropic ones requiring more hyper-parameters to tune and becoming untractable for high-dimensionality [171]. Excellent reviews (see “Introduction” section) are available for response surface modeling, however, the validation of response surfaces is necessary to ensure surfaces are representative of the underlying physical model. In simulation based applications, the number of data is generally insufficient to create distinct training validation sets. Specific approaches to model validation have therefore been proposed, some of them being independent of the surrogate method at hand. In the learning phase, an error estimator is needed to optimize the hyper-parameters of the approximate model. Gaussian-processes are commonly tuned by the likelihood maximization, employing evolutionary or gradient-based algorithms [168, 213]. Methods estimating the predictor error without generating an additional set of observations, include CV [58] and the bootstrap [69]. Kohavi [109] compares bootstrap to CV and concludes to better performance for ten-fold CV, on the specific case considered. The overall surrogate error is usually validated by Cross-Validation (CV) or sensitivity estimation [73]. K-fold-CV consists in successively removing k samples from the training set and estimating the model accuracy on these experiments. Setting k=1 yields the Leave-One-Out (LOO) procedure. Such methods allow to build the validation test from available data, to quantify the overall error typically with the Mean Square Error (MSE), the Root MSE (RMSE) and the Integrated MSE (IMSE) [115]. A more local approach provides a partition of the design space using the Variable Error Value with Sampling Points (VESP) [46] assigned to different partitioned regions. Thereafter, surrogate models are built iteratively using different subsets of samples and tested on the remaining points. Local accuracy can also be quantified by the Maximum Absolute Error (MAE). The correlation between two function responses can be measured by the coefficient R [78] when the relationship between the predicted response and the actual response is linear, one can also find r, the Pearson correlation coefficient [205] or its alternatives for non-linear models [200].

Off-line design of experiment

Sampling techniques for numerical experiments can be grouped into two main categories, namely Classical DoE (CDoE) and Modern DoE (MDoE) or Design and Analysis of Computer Experiments (DACE) [129, 196]. Giunta et al. [77] presents general summaries of CDoE and MDoE methods used in aeronautics. To obtain a good quality approximation on the whole domain, the points must be distributed as evenly as possible in space. Classical DoE methods for physical (in vitro/in vivo) experiments tend to allocate the sample points in the way to minimize the effect of the random error term. They include the full factorial and the fractional factorial design [133], taking samples from regularly spaced sites. Its main limitation is that the total number of design points increases exponentially with the problem dimension [19]. The central composite [144] and the Behnken Box design [28] follow a similar principle but replicate samples are taken making these methods less interesting for computer (in silico) experiments considered as deterministic. A widely used, random sampling methods for parameter estimation, albeit costly, is the Markov chain Monte Carlo (MCMC) [151] algorithms, however the number of simulations involved rapidly becomes prohibitive since they are based on the Markov chain process which requires a very high sample size. Quasi-random methods generate well-distributed sets of points with a controlled size. Various criteria are used to fill the DoE, such as Minimax and Maximin models, Kullback–Leibler, Audze–Eglais and Maximum Entropy Sampling (MES) [190]. Another approach maximizes Euclidean distance between all points in the DoE [141]. Among the modern DoE methods, [77], one of the most commonly used is the Latin Hypercube Sampling (LHS) [135], distributing a fixed number of samples independently of the DoE dimensions. The Voronoi Latinized Centroid Tesselation is an extended version of the LHS technique [185], improving its coverage. Sampling can also be constructed after an Analysis Of VAriance (ANOVA), or using Sobol [203] and Halton [87] sequences, often used to identify the parameters that most influence the output values [98]. Above space-filling techniques attempt to distribute the sampling points evenly in the design space. However, there is no general definition of an a priori number of samples in aerodynamic problems resulting in the desired precision [232], as QoI are often non-linear or discontinuous. Therefore, adaptive sampling approaches are developped to progressively improve the quality of the surrogates.

Adaptive sampling

Sacks et al. [184] introduced sequential sampling using the model information to improve the sampling’s quality with posterior data. In the literature, a sequential strategy is said to be adaptive if it improves the DoE without considering the experiment outputs. Sample location has been found to have a significant impact on the estimated error [34, 80] and is improved when adaptive sampling methods are used to select new samples in areas of the design space where the error estimate is higher. Liu et al. [123] present a comprehensive review of global surrogate sampling, classifying the methods into one-shot, sequential, sequential adaptive, and multiple sampling categories. Another review presents methods tackling high-dimensional black-box problems by [189]. This section presents some of the existing enrichment criteria applied to the SBO framework. The objective of adaptive procedures is to find a sufficiently representative model allowing for an efficient convergence of the optimization problem.They are generally based on an infill criterion (Fig. 2), selecting new sample $\varvec{\chi }$ to be added to the current training set $\varvec{X} \subset \mathcal {D}$.

Fig. 2

The adaptive SBO strategy

The most common infill sampling criteria are built from stochastic metamodels in a Bayesian scheme. Zilinskas and Brochu et al. [30, 239] select the evaluation points sequentially taking into account a compromise between exploration of high uncertainty areas and intensification to improve over the current best observation. Regarding single-objective bound-constrained optimization, the Expected Improvement (EI) was popularised by [101] in the Efficient Global Optimization (EGO) framework. Later, the EI criterion has been extended to handle constraints [149, 150] and to address multi-objective problems.³ In the general EGO framework, [157] compared EI, Augmented EI [95] and the Weighted MSE Criterion for a noisy optimization benchmark. The weighted EI extension is proposed by [207] in order to control the balance between exploitation and exploration in a constrained optimization framework. An EGO handling constraints was also applied by [17, 156]. Scott et al. [186] generalized the EGO for multidimensional variables the Gaussian process and gradient-knowledge for noisy QoIs framework. Another variants of the EGO were proposed by [40, 118, 161]. These strategies are also categorized by the survey [123] as variance-based adaptive sampling approaches including:

• Minimization and maximization of the predicted variance or Mean Square Error [124],

• Lower Confidence Bounds (LCB) [30],

• Upper Confidence Bounds (UCB) [11]

• Gaussian Process-UCB [206],

• EI [101],

• Probability of Improvement (PI)[30],

• Entropy search [93, 217].

Currin et al. [57] applied a two-stage optimal design based on the maximum entropy criterion to gain insight into a circuit-simulator example. Sacks et al. [184] used two-stages optimal design based on Integrated Mean Squared Error (IMSE) criterion for the same example. Mackman and Allen [127] compared sequential sampling methods for the generation of surrogate aerodynamic models. Regis [169] coupled the Trust Region method (TR) to the EGO and compared to the standard EI and an adaptive criterion based on the RBF [171] on several test problems including the groundwater application based on a 36-dimensional simulation concluding that the proposed strategy is most efficient in the high-dimensional configuration. Li et al. [120] developed an adaptive DOE method based on accumulative error inspired by the greedy algorithm’s principles [147, 216], and favoring regions of space where the outputs are non-linear. Continuous And Multi-Modal regions of the design space corresponding to multi-modal, noisy responses with a tendency to abrupt variation are identified and sampled until all suspected Continuous Multi-Modal regions are explored. This method was compared to the maximum entropy design and the maximum distance metric. It was concluded that it outperforms the other methods in terms of RMSE in the majority of the engineering examples presented.

Multi-fidelity management

The MF approach consists in merging Low-Fidelity (LF) with High-Fidelity (HF) data to produce surrogate models at affordable costs. A study of MF methods has been published by [151] in the context of uncertainty propagation, inference, and optimization. It was followed by an overview [75], which questions the added value of MF approaches and how they can be used most effectively. MF Models (MFMs) have been grouped by [72] into three categories, the first where only HF quantities are estimated by a surrogate model, secondly, approaches where both levels are combined within the same metamodel, and finally, the category in which LF is only an enabler of HF modeling. This last category is related to the filtering approaches as it consists in interrogating LF to decide when to use HF models. A more general review is completed by [232] on design space sampling, model selection, ROM, and assisted surrogate models in aeronautics. MF is presented as one of the recent solutions to improve simulation efficiency. This section gives an overview of the main MF approximation methods in order to determine each method’s applicability according to our knowledge of the problem at hand.

Multi-fidelty data fusion

Corrective approaches The concept of MF was originally introduced in linear additive [108] and multiplicative [36, 85] approaches used in gradient-based optimization and was applied to a high speed civil transport aircraft wing. Hutchison et al. [96] applied a polynomial correction term on aerodynamic drag approximation. The main idea is to consider the LF model as a general trend to exploit by adding approximation of the difference (or ratio) between the LF and HF quantities. The corrective methods relie on the assumption of a relationship between different levels of fidelity, allowing to better approximate a fine HF model with its associated coarse LF using additive [44, 177, 208]

$$\begin{aligned} \widetilde{\mathcal {J}}_{MF}(\varvec{\chi }) = \mathcal {J}_{LF}(\varvec{\chi }) + \widetilde{\varepsilon }(\varvec{\chi })\,, \end{aligned}$$

(3)

multiplicative [4, 15]

$$\begin{aligned} \widetilde{\mathcal {J}}_{MF}(\varvec{\chi }) = \widetilde{\beta }(\varvec{\chi }) \mathcal {J}_{LF}(\varvec{\chi }) \end{aligned}$$

(4)

or hybrid (also called comprehensive) corrections [235, 236]

$$\begin{aligned} \widetilde{\mathcal {J}}_{MF}(\varvec{\chi }) = \widetilde{\beta }(\varvec{\chi }) \mathcal {J}_{LF}(\varvec{\chi }) + \widetilde{\varepsilon }(\varvec{\chi })\,. \end{aligned}$$

(5)

where $\mathcal {J}_{HF}, \mathcal {J}_{LF}$ and $\widetilde{\mathcal {J}}_{MF}$ are exact HF, LF and MF objective functions, $\widetilde{\varepsilon }(\varvec{\chi })$ and $\widetilde{\beta }(\varvec{\chi })$ are surrogate models of the difference and ratio between $\mathcal {J}_{HF}(\varvec{\chi })$ and $\mathcal {J}_{LF}(\varvec{\chi })$.

Figure 3 illustrates a non-linear function form in the review of MFMs methods [75]. The comprehensive approximation is closer to the HF targeted function and appears to be the most accurate. Ng and Eldred [146] originally proposed the MF Polynomial Chaos Expansion (PCE) in the UQ field [63, 64] using an additive correction between fidelity levels. An improvement of the HK has been proposed by [148] through the use of PCE and Kriging as a HF surrogate model, then extended by additional polynomial terms to improve the accuracy within an inviscid RAE 2822 CFD application. The addition of a first order polynomial to the HK model has been shown to improve predictive performance. Lickenbrock et al. and Rumpfkeil et al. [122, 183] proposed a MF model based on a scattered PCE in the context of a CFD case illustrated in Fig. 4. This bi-fidelity CFD case demonstrated that the MF PCE was able to capture the lift coefficient trends.

Fig. 3

Illustration of Multi-Fidelity scaling (or corrective) approaches by [75]

Fig. 4

Baseline of an efficient supersonic air vehicle (ESAV) model [122]

Space-Mapping (SM) methods, developed by Bandler et al. in 1994 in cartography reviewed by [16] and applied to the microwave circuit design make corrections at the input of a model, rather than at its output. They are based on the assumption that the set of entry points of a HF model is a geometric transformation of the HF models [14, 119]. The main idea is to optimize the link or mapping between the spatial parameters LF and the HF model’s spatial parameters to meet optimization specifications. The parameters are obtained by minimizing the difference between the available HF values and those of the metamodel. HF points generally update this approximation. This method was coupled with the Trust Region Model Management (TRMM) by [176]. The parameters, defined on different design spaces, are linked by a corrected SM. It is used with a TRM of sequential-quadratic programming for two design optimization problems related to aerospace. These approaches allow different input dimensions [130] in the LF and HF models.

Fusion models are partly known by co-Kriging models [74, 104, 110, 143, 153, 180]. The multi-fidelity co-Kriging extension was originally based on a linear multi-fidelity regression formulation [54], that was improved by [56] through a Bayes linear formulation in a UQ framework. The co-Kriging multi-fidelity extension, introduced by [104], is based on a scaled LF objective function $\mathcal {J}_{LF}$ with a factor $\rho$ added to the Kriging-interpolated correction function $\varepsilon$

$$\begin{aligned} \mathcal {J}_{HF}(\varvec{\chi }) = \rho \mathcal {J}_{LF}(\varvec{\chi }) + \widetilde{\varepsilon }(\varvec{\chi }), \, \varvec{\chi }\in \mathcal {D} \end{aligned}$$

(6)

where $\mathcal {J}_{HF}(\varvec{\chi })$ and $\widetilde{\varepsilon }(\varvec{\chi })$ are respectively the exact HF objective and the interpolation of $\varepsilon (\varvec{\chi })=\mathcal {J}_{HF}(\varvec{\chi })-\mathcal {J}_{LF}(\varvec{\chi })$.

Le Gratiet et al. [115] proposed a recursive formulation of co-Kriging to tackle the complexity reduction related to the inverse of the covariance matrices using a recursive multi-level formulation. The Hierarchical Kriging (HK) developed by [89] redefines co-Kriging considering the LF function’s Kriging as the trend of the Kriging model for the expensive HF function avoiding the difficulty associated with building cross-covariance. The complexity related to higher dimensional Kriging is typically tackled through the use of alternative metamodels such as RBF of diffuse approximation where the covariance matrice and hyperparameters are less computationally demanding. A MF extension of the RBF regression model is introduced by [173] for experimental and computational integrated data for a missile configuration, and was later applied by [175] for the surface pressure of aircraft wings. Durantin et al. [66] developed a co-RBF extension yelding an accuracy of the same order as that of co-Kriging, while reducing its training cost, especially in high dimension.

Filtering approaches

To manage MF models in the optimization context, hierarchical methods use the LF model to filter out unpromising points before evaluating the remaining points with the HF model [44, 59]. This group of methods that may be considered as filtering techniques [151] consisting in using the LF to explore the design space may also be coupled to the trust-region framework [51]. Alexandrov et al. [4] has associated a tuned LF with a numerical optimizer subject to a confidence region constraint, where the LF model is tuned using the HF model and applied to a wing design optimization problem. Giunta et al. [76] applied the MFM technique to the multidisciplinary design of high-speed civil transport where the LF is used to define a sub-region of the most likely optimal location. The HF is then applied to improve the fidelity level.

Dedicated infill

The efficiency of MFMs is conditioned by the correlation between levels of fidelity and their evaluation cost. A comparison of surrogate quality criteria [94] concludes MSE to be the most efficient in the MFM framework, followed by LOO and Predictive Estimation of Model Fidelity (PEMF) error. a MFM metamodel quality metric involving PEMF proposed by [136] chooses between multiple fidelity levels. Song et al. [205] compared the commonly used $R^2$ metric (or performance criterion) to the Pearson Correlation Coefficient (PCC) in a context of multiple levels of fidelity in order to introduce the correlation impact on MFMs. The PCC measure between two random variables describing the correlation between HF and LF functions is defined by [212] for MFMs:

$$\begin{aligned} r^2 = \left( \frac{\sum _{i=1}^M (\mathscr {Q}_{HF}(\varvec{\chi }^{(i)}) - \bar{\mathscr {Q}}_{HF}(\varvec{\chi }^{(i)}) )(\mathscr {Q}_{LF}(\varvec{\chi }^{(i)}) - \bar{f}_{LF}(\varvec{\chi }^{(i)}) )}{\sqrt{\sum _{i=1}^M (\mathscr {Q}_{HF}(\varvec{\chi }^{(i)}) - \bar{\mathscr {Q}}_{HF}(\varvec{\chi }^{(i)}) )^2 \sqrt{\sum _{i=1}^M (\mathscr {Q}_{LF}(\varvec{\chi }^{(i)}) - \bar{\mathscr {Q}}_{LF}(\varvec{\chi }^{(i)}) )^2 }}}\right) ^2 \end{aligned}$$

(7)

where M is the number of observations of the LF and HF solvers $\mathscr {Q}_{LF}$ and $\mathscr {Q}_{HF}$, respectively and $\bar{\mathscr {Q}}_{LF}$ and $\bar{\mathscr {Q}}_{HF}$ the corresponding means. Toal [212] compares the impact of RMSE and the correlation metrics of the LF and HF functions. It has been observed that the $r^2$ metric is better suited than RMSE to identify the LF-HF correlation. Using the correlation measures, it was shown that the amount of LF data must be greater than that of HF and the LF simulation budget must be no greater than 80% of the total simulation budget but at least higher than 10% to be worthwhile to use a MF representation. Thus, merging existing LF data with expensive HF data is not always cheaper than a single-fidelity method [75]. Correlation, cost ratio and other error metrics are crucial to measure the benefit of MF models, which depends on the context (optimization, UQ, etc.). The efficiency of MFMs depends on the trade-off between cost and accuracy [75]. Dedicated criteria are used to maximize the benefit of MF methods in the context of optimization. One concept of MF infill strategies is to establish a criteria to predict the most promising locations for new HF samples from a LF model.Other criterion use multiple fidelity levels to estimate the most promising fidelity level and location within the design space. Some of these methods take into account the HF/LF cost ratio to control the overall budget. In the following paragraphs, we provide an overview of the main adaptive approaches in these two categories.

Trust-Region MF management

Trust-Region Methods (TRMs) have been used to locate promising points in the MFM framework [1], with derivative-based infill criteria [3] and extended to be used without gradients by [130, 131] and in a multi-objective framework [61, 110]. Rodriguez et al. [178] coupled a corrected LF model with a TR constrained numerical optimizer. They proposed a MF optimization framework based on a TR algorithm, the gradient of the objective function computed using the LF model allows the solution to converge to the HF model’s optimum. The TR MFM uses a ratio between LF and HF, indicating the region of confidence [151]. The MFMs used by [151] are inspired by the approximation model management for the optimization in [2] which considers the approximation model as a level of fidelity in its own right. Corrective methods are used to evaluate the sub-models within the confidence interval at each iteration k and to improve LF QoIs

$$\begin{aligned} \mathscr {Q}_{MF}(\varvec{\chi }) = \mathscr {Q}_{LF}(\varvec{\chi })+e(\varvec{\chi }_{k})+\nabla e(\varvec{\chi }_{k})^{T}(\varvec{\chi }-\varvec{\chi }_{k}) \end{aligned}$$

(8)

where $e(\varvec{\chi }_{k})$ is the MFM correction model at iteration k, $\mathscr {Q}_{LF}$ and $\mathscr {Q}_{MF}$, respectively the LF and MFM QoI.

Adaptive MF Kriging

The other way to efficiently manage the optimization in a MFM framework is the EGO (see “Adaptive sampling” section). The MF co-Kriging provides the variance that can be directly exploited to enrich the model. Le Gratiet and Le Gratiet and Grainer [115, 116] proposed a sequential version of recursive co-Kriging that uses IMSE and takes into account both the computational cost ratios between code fidelity levels and their respective contributions to the total variance. This method is used to predict a turbulence model of a particular gas composition where different refinements mesh define two levels of fidelity. The strategy, called sequential Kriging is used in an optimization framework to select new points of interest. It has been extended to parallel computing and adapted to the vectorial MF optimization application of a reservoir by [210] (see “Adaptive reduced-order models” section). In the same framework, [112] present a two-step space-filling strategy considering the information gain in the variance and the cost of each level of output fidelity. This strategy was tested with analytical MF extension of the 2D Rosenbrock function, a constrained optimization problem, and an aerodynamic optimization test case. The MFM was demonstrated more efficient than the single-fidelity EGO. The approach adopted is the merging of L fidelity levels for which the level $l_{n+1}$ of the next simulation at design point $\varvec{\chi }_{n+1}$ is

$$\begin{aligned} l_{n+1}=argmin_{l \in {1,...,L}} \quad \frac{C_{l}(\varvec{\chi }_{n+1})}{\bar{\sigma }_{l}^{2}(\varvec{\chi }_{n+1})-\widetilde{\sigma }_{l}^{2}(\varvec{\chi }_{n+1})}, \end{aligned}$$

(9)

where $C_{l}(\varvec{\chi }_{n+1})$ is the evaluation of the $(n+1)^{th}$ predicted point cost with the simulator of level l, $\bar{\sigma }_{l}^{2}(\varvec{\chi }_{n+1})$ and $\widetilde{\sigma }_{l}$ are respectively the mean and the variance of the MF surrogate model at $l^{th}$ fidelity level. The objective is to find the lowest output cost with the highest information gain.

An adaptation of EI to MF is applied to HK by [234]. The HK scheme of [91] is used to choose the fidelity level at each iteration. The adapted EI expression includes the fidelity level l

$$\begin{aligned} EI_{vf}(\varvec{\chi },l) = \left\{ \begin{array}{l@{\quad }l} 0 &{} \text{ if } \quad \sigma _{l}(\varvec{\chi }) = 0, \\ (\mathscr {Q}_{best}-\widetilde{\mathscr {Q}}(\varvec{\chi }))\Phi \left( \frac{\mathscr {Q}_{best}-\widetilde{\mathscr {Q}}(\varvec{\chi })}{\sigma _{l}(\varvec{\chi })}\right) + \sigma _{l}(\varvec{\chi })\phi \left( \frac{\mathscr {Q}_{best}-\widetilde{\mathscr {Q}}(\varvec{\chi })}{\sigma _{l}(\varvec{\chi })}\right) &{} \text{ if } \quad \sigma _{l}(\varvec{\chi }) > 0, \end{array} \right. \end{aligned}$$

(10)

where $\phi (.)$ and $\Phi (.)$ denote respectively the standard normal probability density and distribution functions, $\widetilde{\mathscr {Q}}(\varvec{\chi })$ the posterior mean of $\mathscr {Q}$ and $\sigma _{l,l\in \left[ 1,2\right] }$ is the variance of the LF and HF models respectively.

Guo et al. [80] applied $EI_{vf}$ and observed that compared to a single-fidelity EI, the search for the optimum was made efficient by LF exploration, were fewer HF calls were needed to solve the optimization problem. Hao et al. [92] developed an extension of this $EI_{vf}$ criterion to Gradient-Enhanced Kriging (GEK). LF sample points are first used to represent the HF function trend close to the optimal solution, then HF samples are added. It is demonstrated on analytical test functions that $EI_{vf}$ adaptively determines both the location and the fidelity level of a new design point. Other variance-based criteria have been adapted to the variable fidelity framework. A comparative study was carried out on the LCB [99] criterion between single-fidelity, MF EI, and other EGO criteria [234]. Eight numerical examples, along with one physics-based aircraft fuselage design, resulted in 25% and 45% savings thanks to the VF-LCB criterion which showed an improved exploration/exploitation trade-off. Also, the PI criterion has been adapted to MFMs as an extended PI

$$\begin{aligned} EPI(\varvec{\chi },l) = PI_{vf}(\varvec{\chi })\times Corr(\varvec{\chi },l)\times CR(l) \times \eta (\varvec{\chi },l), \end{aligned}$$

(11)

where $PI_{vf}$ is the PI built from the MFM co-Kriging of [181], $l = 1,...,\, L$, is the fidelity level, $Corr(\varvec{\chi }, l)$ denotes the posterior correlation coefficient between the HF and fidelity level l at input $\varvec{\chi }$. CR(l) is the cost ratio between HF and $l^{th}$ fidelity model, $\eta (\varvec{\chi },l)$ is the sample density function adopted by [123] to avoid clustering of training data. This criterion was compared to the Augmented Expected Improvement (AEI) of [95], $EI_{vf}$ of [234], another MF PI variant, and a standard single-fidelity PI. An application to structural design optimization of the micro-aerial vehicle fuselage and stiffened cylindrical shell was carried out and led to the conclusion that the proposed VF-PI method is most efficient when the correlation between the HF and LF models is high. In contrast, the VF-EI method is even more efficient when the correlations remain low. In the same group of sampling methods using GPs, the GP-UCB, also known as UCB in the Bayesian framework [30] was adapted to MFM by [102]. The MF adaptation uses the largest LF uncertainty to add the next sample. This criterion was compared to the single-fidelity GP-EI, GP-UCB criteria, and the algorithm MF Mutual Information Greedy (MF-MI-Greedy) proposed by [187] where fidelities are assumed independent. This algorithm maximizes the amount of mutual information in order to take advantage of LF. MFM is modeled using additive Gaussian processes based on shared relationships with the target HF function, and the variable costs of different fidelities are taken into account. The MF-MI-Greedy was demonstrated to effectively reduce the uncertainty.

Adaptive RBF models

Another extension of the EI was proposed by [152] for MF UQ and adapted to RBF surrogate modelling. The enrichment aims to minimize both the uncertainty and the objective function. Several criteria are compared, including the Maximum Uncertainty Adaptive Sampling (MUAS), evaluated from the MF metamodel and minimized, the MFEI, and Aggregate Criteria Adaptive Sampling (ACAS) consisting in minimizing the difference between the MF prediction and the associated uncertainty. Finally, Multi-Criteria Adaptive Sampling (MCAS) allows identifying new training points by minimizing the objective function constrained by a maximal prediction uncertainty. This adaptive sampling strategy was tested on a dynamic Stochastic Radial Basis Functions (SRBF) [218]. It has been observed that MUAS and MCAS were more exploratory than MFEI and ACAS, and HF evaluations were well distributed within the design space. MFEI and ACAS concentrate HF evaluations in the optimum region. The overall results lead to the conclusion that MFEI are the most efficient adaptive sampling techniques for the proposed CFD shape optimization test case of a NACA hydrofoil. Such bayesian EGO framework is used with the the Gaussian Process based model so-called Kriging because it is able to provide variance and mean values to guide the DoE search. Extension of such framework were adapted to a reduced complexity surrogate RBF by [26], allowed [66] to obtain time savings for comparable optimization results. The same conclusion was drawn by [215], following the co-RBF comparison with a Random Forest co-surrogate. Gutmann [81] used RBF metamodels in an EGO approach with a measure of interpolation error, referred to as the power function. This criterion was modified by [170] to improve its local search property. The adaptive MF Sequential Radial Basis Optimization (MFSRBO) method was developed by [174] and tested on an aerodynamic application. The compromise criterion between fidelity levels [104] of additive MF surrogate is adapted to an RBF metamodel. This scheme uses the difference between HF and l fidelity levels. The AEI criterion proposed by [95] for co-Kriging is adapted to the MF RBF at the $l^{th}$ fidelity levels as

$$\begin{aligned} EI(\chi ,l) = EI*\alpha _{1}(\chi ,l)*\alpha _{2}(\chi ,l)*\alpha _{3}(\chi ,l), \end{aligned}$$

(12)

with $\alpha _{1}(\chi ,l)$ a discontinuous function of the local prediction error, $\alpha _{2}(\chi ,l)$ the relative reduction in the posterior standard deviation and $\alpha _{3}(\chi ,l)$ is the ratio of the l fidelity to the highest HF costs $\frac{C_{l}}{C^{HF}}.$

Cai et al. [34] adapted the CV metric associated with the Voronoi tesselation [229] to partition the design space for RBF-based MF metamodeling. Recently, [219] proposed a MF metamodel based on the stochastic RBF (SRBF) with a MF adaptive sampling based on CV error and the Voronoi partition, sampling both LF and HF.

Surrogate modeling for full-field computations

The optimization problem formulation (2) and consequently the surrogate approaches presented in previous sections consider QoI functionals obtained by postprocessing simulation results corresponding in aerospace applications to detailed velocity, pressure, stress, or thermal fields defined on a finite element/finite volume mesh. The full-field approach consists in building a Reduced-Order Model (ROM) $\varvec{\widetilde{f}}(\mathbf{x} ,\varvec{\chi })$ of the solver and to express, objective function and constraints in terms of the approximate field. This approach yields a better insight into detailed characteristics of the flow at every optimization iteration. The corresponding optimization problem is stated as

(13)

where $\mathbf{x}$ is the set of points defining the space discretization $\Omega \subset \mathbb {R}^n$ with n the number of nodes, $\varvec{\chi }$ the design variable, $\in \mathcal {D}\subset \mathbb {R}^{d}$, and $\varvec{\chi }^{*}$ is the optimum configuration of the objective function $\mathcal {J} \in \mathbb {R}$ respecting constraints $\mathbf{g}$, $\mathbf{h}$.

The different steps of the full-field SBO are illustrated in Fig. 5. First, the DoE $\varvec{X} = \{\varvec{\chi }^{(0)}\,,... \,,\varvec{\chi }^{(M)}\}$ is generated in $\mathcal {D}$ with one of the a priori sampling methods cited in “Off-line design of experiment” section. Once the fields $\varvec{f}(\mathbf{x} ,\varvec{\chi }^{(i)})$, $i=\{1,\,... \,, M\}$ have been computed, the ROM $\varvec{\widetilde{f}}(\mathbf{x} ,\varvec{\chi })$ is built and the optimization is performed with the functionals $\mathscr {Q}(\varvec{\widetilde{f}(\mathbf{x} ,\chi )})= \{\mathcal {J}(\varvec{\widetilde{f}(\mathbf{x} ,\chi )}),\varvec{g(\widetilde{f}(\mathbf{x} ,\chi ))},\varvec{h(\widetilde{f}(\mathbf{x} ,\chi ))}\}$.

Fig. 5

The full-field SBO strategy

“Reduced-order modeling” section reviews the ROM techniques; associated adaptive strategies are presented in “Adaptive reduced-order models” section.

Reduced-order modeling

While the surrogate modeling techniques cited in the previous Sections can exploit integrated quantities, ROM can capture spatial information in a model approximation by expressing the solution as a linear combination of a limited number of modes. Thus assuming that few independent modes govern the system dynamics, a significant reduction in the solution’s computational cost can be achieved [47]. Applications to the description of turbulent flow have demonstrated the capture of non-linear features. Aubry et al. [9] constructed a 5-mode system that qualitatively reproduces the turbulent boundary layer discontinuity. This approach was extended to diverse types of flows [79, 167, 214]. The ROM can be intrusive or non-intrusive (direct or indirect) depending on whether the physical model is considered as a black box or the reduced Partial Differential Equation (PDE) is solved online.

Intrusive ROM (IROM) reproduces the physical behavior of a system by approximating the governing equations and source code and is typically derived by POD and Galerkin projection. Therefore, it retains much of the physical characteristics of the original system due to its intrusive nature. Physical models and engineering problems are described by Partial Differential Equations (PDEs). These equations usually depends on a number of parameters, corresponding to physical or geometrical properties of the model as well as to initial or boundary conditions of the problem, and can be deterministic or subject to some source of uncertainty. The IROM requires code modifications seeking to reproduce these physical PDEs to approximate there solutions $\varvec{f}(\mathbf{x} ,\mathbf{t} )$ spatial distribution $\mathbf{x}$ at time $\mathbf{t}$ (such as displacement or velocity spatial distribution). The PDE to approximate may be non-linear and depending on time $\varvec{t}$, space $\mathbf{x}$ and physical parameters $\varvec{\chi }$. The approximated solution $\widetilde{\varvec{f}}(\mathbf{x} ,\mathbf{t} )$ may be obtained, for example, by Galerkin projection onto a subspace spanned by a set of modes [5, 6, 227]

$$\begin{aligned} \widetilde{\varvec{f}}(\mathbf{x} ,\mathbf{t} ) = \sum _{k=1}^{m}\alpha _k (\mathbf{t} )\Phi _k (\mathbf{x} ) \end{aligned}$$

(14)

where $\alpha _k (\mathbf{t} )$ are temporal coefficients that capture the time dynamics, $\Phi _k (\mathbf{x} )$ are spatial modes. The modes may be obtained by Proper Orthogonal Decomposition (POD) also known as Karhunen-Loève Expansion (KLE) [126] in the field of turbulence or Principal Component Analysis (PCA) [100] in machine learning applications. The POD relies on the extraction of the reduced basis by truncated Singular Value Decomposition (SVD) of the snapshots matrix. This technique produces a low rank global basis of the most impacting modes [198] and has been shown to be appropriate for coherent flow structures that can be sorted by their energy content [25]. Intrusive POD has been applied by [121] to estimate state variables and associated variables in transonic flows in gradient-based aerodynamic shape optimization, but also in mechanical systems [125]. Other approaches are Krylov subspace [13] used on large scale systems structural dynamics or turbulence in CFD, Fourier well suited for periodic signals series as in Harmonic Balance Method (HBM) [86, 230], or Voltera Series [182] applied to structural, aeroelasticity problem, control engineering, or electrical engineering. The non-linear engineering applications were reviewed by [38], by [23] for dynamical systems projection-based approaches and by [193] for aeroelastic identification and more general detailed references about aeroelastic application [7]. IROMs may cover different kinds of separated representations of the involved unknown fields as well as different constructors able to address a variety of linear and non linear models, elliptic,parabolic, and hyperbolic. Extensions were proposed to general linear evolution schemes such as finite volume schemes for parabolic and hyperbolic evolution equations [84]. In a non-exhaustive way, one can also find Reduced-Basis Method (RBM) [179], Empirical Interpolation Method (EIM) and Discrete EIM (DEIM) [238]. The evaluation cost of the projected non-linear model is not always more interesting than that of the full model in representing non-linear dynamics. This limitation can be addressed by several existing approaches and ROM extensions, including POD-DEIM for which the linear terms may be treated in POD-Galerkin type models, an additional POD basis is introduced to represent the non-linear terms in the ROM. The location of the non-linear interpolation points are determined based on a DEIM correction algorithm, which minimizes the non-linear residuals. Three different model reduction techniques are compared by [197] the Galerkin projection onto a POD basis representing the principal variable, the Galerkin projection onto two separate POD bases, one basis for the principal variable, a second basis for the nonlinear convection term, and the DEIM for evaluating the projected nonlinear terms were applied to a mode equation and the Navier–Stokes equations. The nonlinearity arises from the convection phenomenon, the solution exhibits advection, diffusion, and instability, and the system exhibits oscillatory behavior that prevails. Provided the number of POD modes is large enough, Galerkin projections have been shown to perform better, with a reduced nonlinear term that still preserves energy. The Galerkin projection with the second basis that represents the nonlinearities also performs well, but the energy preservation criterion is significantly impaired. The DEIM model reduction technique succeeds for wind flow in the limit cycle regime, but fails for wind flow in the transient regime, producing reduced order models that exhibit a fine time singularity. Proper Generalized Decomposition (PGD) [41, 42] as well covers a variety of applications from structural optimization to CFD optimization. Under the assumption of the influence of PDE uncertainties, the polynomial chaos model (PCE) can also be used. It is a polynomial of random variables based on the homogeneous chaos theory of Wiener [106], based an expansion of chaos using Hermite polynomials. These direct approaches are closer to the behavior of the simulated system since they are directly applied to the physical equations. Nevertheless, such methods are prone to instability and non-linearity efficiency problems, and they are also difficult to implement, as they require a modification of the source code [222].

Non-intrusive ROMs

To avoid this problem, non-intrusive approaches have been introduced, such as Non-Intrusive POD (NIPOD), also known as Galerkin-free [191] or POD with Interpolation (PODI) [31]. Unlike direct ROM approaches, indirect ROMs do not require calls to the differential equation solver to perform the interpolation. As in the intrusive ROM, the POD snapshot method, developed by [198], allows to empirically build the basis vectors keeping the modes whose sum of energies represents at least 90% of the total energy, while the neglected modes represent less than 1% of the main mode’s energy. This POD technique was applied by [31] as an interpolation of known designs for inverse design purposes. Coelho et al. [47] successfully applied a two-step NIPOD on a 3D flexible wing shape optimization, involving weak code coupling between the solid and fluid models. On the one hand, the coupling variables were reduced by the POD, expressing the interface pressure as a linear combination of a few modes. On the other hand, the scalar coefficients obtained by the linear expansion of the POD modes were approximated by polynomial response surfaces and moving least squares surrogates. The idea of a Non-Intrusive POD basis in the offline phase, is to store the full-field output (pressure, velocity field, etc.) and use it to predict the vectors online. In the online phase, the solution vectors are approached with the linear combination of the basis vectors. First M vectors are sampled and stored in the so-called snapshot matrix $\varvec{S}$. The DoE $\mathcal {X} = \{\varvec{\chi }^{(1)},\ldots ,\varvec{\chi }^{(M)}\}$ is generated using an ad hoc sampling method (LCVT, hypercube,etc.) and $\varvec{S}$ is constructed with the $\varvec{f}$ function evaluating the DoE $\varvec{\mathcal {X}}$

$$\begin{aligned} \varvec{S} = \left[ \varvec{f}(\mathbf{x} ,\varvec{\chi }^{(1)})\, \ldots \, \varvec{f}(\mathbf{x} ,\varvec{\chi }^{(M)})\right] \end{aligned}$$

(15)

where $\mathbf{x}$ is the discretised space vector of the size $dim\times n$, $dim\{1,2,3\}$ is the physical space dimension. When $n\ge M$ is considered in the following, the snapshots matrix may be defined as the sum of the mean $\bar{\varvec{s}}$ of all the snapshots

$$\begin{aligned} \bar{\varvec{s}} =\frac{1}{M} \sum _{i=1}^{M} \varvec{f}(\mathbf{x} ,\varvec{\chi }^{(i)}) \end{aligned}$$

(16)

and the matrix of fluctuations $\mathcal {S}$

$$\begin{aligned} \varvec{S}=\bar{\varvec{s}}\cdot \begin{pmatrix} 1\\ \vdots \\ 1 \end{pmatrix}^{T}_{M\times 1} + \, \mathcal {S}. \end{aligned}$$

(17)

Note the difference between $\varvec{S}$ and $\mathcal {S}$ symbols, denoting snapshot and fluctuation matrices, respectively. To eigen decompose the covariance matrix $\varvec{C} = \varvec{S}\varvec{S}^T$, the SVD of a rectangular matrix $\varvec{S}$ of the size $n\times M$ may be considered.

$$\begin{aligned} \mathcal {S} = \varvec{\Phi } \varvec{\Sigma } \varvec{\Psi }^{T} \end{aligned}$$

(18)

where $\varvec{\Phi }$ and $\varvec{\Psi }^T$ matrices are respectively composed of the left and right singular vectors, of size $n\times M$ and $M\times n$, $\varvec{\Sigma }$ is a diagonal matrix containing the singular values of $\mathcal {S}$ and equivalent when squared to eigen values of the “big” $(n\times n)$ $\varvec{C} = \mathcal {S}\mathcal {S}^T$ or “small” $M\times M$ correlation matrix $\varvec{c} = \mathcal {S}^T\mathcal {S}$. Note that the decomposition may be performed by different approaches, the most common being “economical” SVD. In the offline phase, this method is used to obtain the decomposition for the initial dataset, and once in the online phase, its incremental version [155] uses only the newly added points instead of the overall DoE at each iteration. Often the basis is truncated to lower the computational burden thus speed-up computation. The idea behind the truncation is to consider the most relevant modes only. The latter are determined by minimizing the relative Frobenius error norm of the difference between the original $\varvec{S}$ and its reconstruction $\varvec{S}_{m_l,LF}$ with $m_l<M_l$ basis vectors

$$\begin{aligned} \varepsilon (m)=\frac{||\varvec{S} - \widetilde{\varvec{S}}_{m}||_{F}}{||\varvec{S} ||_{F}}. \end{aligned}$$

(19)

The m optimal value $m^{*}$ is evaluated from the minimization of $\varepsilon (m)$

$$\begin{aligned} m^{*} = \mathop {{{\,\mathrm{\hbox {arg}\,\hbox {min}}\,}}}\limits _{m\in \{1\,,\,... \,,\, M\}}\varepsilon (m). \end{aligned}$$

(20)

Once the NIPOD basis $\Phi$ is obtained, the snapshots matrix $\varvec{S}$ is projected into this basis to determine the scalar coefficients $\alpha (\varvec{\chi }^{(i)})$ for the DoE points belonging to $\mathcal {X}$

$$\begin{aligned} \alpha ^i(\varvec{\chi }) = \varvec{\Phi }(\mathbf{x} )^{T}(f(\mathbf{x} ,\varvec{\chi }_i)-\bar{\varvec{s}}), \,\varvec{\chi }_i \in \mathcal {X}, \, i= 1,\,...\,,\, M \end{aligned}$$

(21)

The $\widetilde{\varvec{f}}(\mathbf{x} ,\varvec{\chi })$ can then be approximated for an arbitrary point $\varvec{\chi }$ by

$$\begin{aligned} \widetilde{\varvec{f}}(\varvec{\chi },\mathbf{x} ) = \bar{\varvec{s}} + \varvec{\Phi }(\mathbf{x} )\widetilde{\alpha }(\varvec{\chi },\varvec{\theta }),\, \forall \varvec{\chi }\in \mathcal {D} \end{aligned}$$

(22)

with $\widetilde{\alpha }(\varvec{\chi })$ the surrogate model of the projection coefficients $\alpha (\varvec{\chi })$ that requires model paramaters (hyperparameters) $\varvec{\theta }$, to be identified in the offline phase.

The obtained reduced order field is used to evaluation quantities to optimize, or used in other contexts where the calls to solver and computional ressourses demand is too high, such as UQ application. In the context of structural highly contrained optimization, the idea of CPOD has been introduced by [226, 227] in order to insure the precise estimation of the quantities of interest of all the training experiments while truncating the POD basis. This method is an extension of the POD in the context of a search for the representation of objectives and constraints, aiming to take into account constraints in the search for the optimal POD basis. The constrained orthogonal projection uses $\Psi$ defined from the QR decomposition and the SVD. Another NIROM extension takle the local non-linearity for CFD applications, the Local Decomposition Method (LDM), is proposed by [65] as an extension of NIPOD and inspired by the mixture of experts and local reduced-order dynamic modeling. It consists in using multiple POD bases adapted to delimited regions of the design space, distinguished by the sampled snapshots features. This separation allowed one to adapt the POD approximations to each physical regime, to capture local non-linearities. Another well-known NIPOD extension is the Gappy-POD dedicated originally to human face image reconstruction from incomplete data [70]. Such an approach has been applied by [31], to fill in incomplete data fields from known snapshots. Gappy-POD has been used to reconstruct fields in fluid and structural applications where distributed sensors provide limited field measurements [49]. Another approach uses the concept of shape manifold for a reduced order representation. Meng et al. [137] reviews the shape manifold designed for ROM representation of complex shapes encountered in mechanical problems. The general idea is to define the shape space, known as POD-morphing [166], in which the structure boundary evolves. The reduced representation [164] is obtained by determining the intrinsic dimensionality of the problem, independent of the original design parameters, and by approximating a hyper surface, i.e., a shape manifold, connecting all the known admissible shapes using level set functions. Moreover, an optimal parameterization can be obtained for arbitrary shapes, where the parameters must be defined a posteriori. The family of manifold step algorithms for predictor–corrector optimization in a reduced shape space guarantees the admissibility of the solution without additional constraints. Applications include structural optimization [165, 166], springback minimization in metal forming [117], microstructural design of materials [223], and inverse problems [138].

Adaptive reduced-order models

The ROM models presented in the former section are often improved during the optimization process in an attempt to target the regions of interest more efficiently. In the ROM context, adaptive approaches can refer to the procedure of orthogonal basis adaptation with each new vector, adding as few samples as possible to build the whole basis. As for the scalar SBO, there are also approaches aiming at choosing samples to reduce the calls to expensive solvers. A range of these approaches is presented in the following paragraphs.

Trust-Region Methods Among intrusive ROMs, [24] applied POD Reduced-Order Models and TR method to minimize the mean drag under rotary control of a cylinder wake in the laminar regime reducing its relative mean drag by 30%. The Sequential sampling schemes TR used on the scalar SBO (see “Dedicated infil” section) were coupled to ROM to solve constrained and unconstrained optimization [172].

Error-based infill methods Among error-based infill methods, a LOO (see “Overview on surrogate-based optimization” section) approach for POD-based models was proposed by [29] along with an error estimate of the POD model to select new sample locations. This strategy was compared to classical a priori uniform sampling and tested on an analytical test case and the 2D turbulent flow around a RAE2822 airfoil. It was concluded that the convergence of the LOO-CV adaptive-based procedure was faster. This infill criterion was applied to the NIPOD approach by [82] where an improvement of the modal coefficients was obtained by taking into account each snapshot’s influence to represent the full-field model. Two cases based on the RAE2822 airfoil demonstrated the error control over the whole parametric space even in the non-linear transonic region.

Greedy approaches The principle of a greedy algorithm is to choose the most promising elements (or to suppress the least promising ones) iteratively to obtain the best prediction. Once the models are built in the off-line phase, the most (or least) predictive features may be selected during the online phase based on error criteria. The ROM database model’s accuracy can be adjusted in the offline phase, where a local ROM database is built by a greedy procedure [162]. The greedy algorithm is used for the selection of snapshots for the approximation of the parameterized PDE solution [147, 216] and in a non-intrusive framework [67]. Such approach was applied by [10] to time-dependent non linear problems using RBF-based NIROM. This model adaptively generates snapshots based on a greedy approach to minimize the overall computational cost.

Adaptive Gappy POD [32] use an adaptive Gappy POD (GPOD) (see “Reduced-order modeling” section) for two-dimensional airfoil inverse design. This approach aims at finding the optimal sensor locations and adaptively improve the POD basis. At each iteration, the criterion used to achieve the selection is based on the Gappy POD maximum error. The complete model information is then generated by solving the system between known, and unknown data at this position, the database is enriched using the resulting snapshot, and the reduced basis is updated. These steps are then repeated, using the updated model to calculate the reduced states at each locations around the airfoil.

Active-learning approaches The NIPOD was adapted to an active-learning⁴ framework using the partitioning approach in the LDM method [65]. The experimental design is divided into several parts corresponding to values associated with a shock sensor allowing to identify the areas where discontinuous structures will likely appear. Once targeted, these regions are clustered and enriched using error, sensitivity, and global entropy metrics. The procedure’s application reveals a capacity to better take into account the regions of high gradient and discontinuity, in the case of a shock problem in a turbulent flow around the RAE2822 wing in transonic regime.

PCA-based infill Within the UQ framework, the implementation of [88] is a non-intrusive method where corrective sampling is used to improve stability and precision. PCA allows building a new basis that describes the shapes globally. The Bayesian optimization is performed in the space of reduced dimension of the active components, complemented by a random embedding in the space of the remaining components, to approach the optimization problem in reduced dimension.

Multi-fidelity reduced-order methods

Traditional multi-fidelity surrogate models are based on integrated scalar data from vector QoI. The correlation between the different levels of solvers is not always meaningful through these integrated quantities, thereby missing valuable information. This is why the approaches have recently been extended to a vector representation using ROMs. Several approaches can be distinguished between those that combine multiple level of fidelity on the ROM model, which are categorized as fusion models, and models using the MF correction approaches. The ROM methods of the last reviewed section outperform scalar approaches in complex non-linear optimizations. The MFM extension of these methods are promising, there references are presented in the present section.

“Multi-fidelity vectorial modeling” section reviews the MF ROM extensions; associated adaptive strategies are presented in “Adaptive multi-fidelity vector models” section.

Multi-fidelity vectorial modeling

MF fusion

Toal [211] adapted the Gappy POD (see “Reduced-order modeling” section) to a MF surrogate-based optimization and UQ framework. A bi-variate optimization of a NACA 0012 airfoil is performed by predicting the two-dimensional HF pressure distribution from its computed LF counterpart. In such framework, [211] demonstrated better accuracy of the MF Gappy POD than the equivalent Kriging and co-Kriging models of predicting the drag polar of an airfoil over a bivariate aerodynamic design space. Improving the accuracy of the drag variation in the design space of an airfoil offers a significant improvement in optimization performance. The approach has also shown promising results in reducing the cost of predicting nodal uncertainties in a thermo-mechanical gas turbine rotor model. Mifsud et al. [139] developed a bi-fidelity POD extension using two discretization levels of numerical experiments. It consists on building an optimal set of orthogonal basis vectors from the two fidelity snapshots subsets. The set of basis are used to construct the overall response, then predict snapshots in unknown locations. This approach is tested for flow prediction using RANS at two fidelity levels of mesh refinement. The numerical examples led to the conclusion that the efficiency depends heavily on the correlation between the two fidelity levels. Inspired by the CPOD of [22, 226] developed the MFNIPOD extension based on QR decomposition allowing to build a bi-fidelity orthogonal basis. This $\varvec{\Phi }_{MF}$ basis is constructed from the concatenation of the high and low fidelity bases $\varvec{\Phi }_{HF}$ and $\varvec{\Phi }_{LF}$ respectively, allowing the HF interpolation. The final interpolation of the field is obtained by projecting the MF field on a reduced MF basis $\varvec{\Phi }_{MF}=[\varvec{\Phi }_{HF}|\varvec{\Phi }_{LF}]$. Unlike the MF NIPOD extension of [139], the MFNIPOD enriches the reduced space obtained from scarce HF information with LF related orthogonal modes, giving a hierarchical ROM. The method was compared to a model based on a single-fidelity RBF Neural Network (RBFNN) and co-Kriging on a constrained 2D analytical optimization and the 19-D optimization of an industrial compressor blade and its non-axisymmetric hub. The results show that MFNIPOD outperforms models based on co-Kriging and RBFNN in terms of costs and accuracy. Combining different data simulations from multiple fidelity snapshots presents difficulties because the two sets of fields lie most likely in different spaces. When fidelity levels are based on different mesh refinement, this problem may be tackled by projecting the LF outputs onto the finer HF grid [20]. Problems of inconsistency were also addressed by [154] who applied a MF ROM using a common subspace, to a transonic wing problem. A reduction in computational training cost between 10% and 73% was observed compared to a single-fidelity approach of comparable accuracy. In a different approach, the MFROM and MFNIPOD extensions inspired the development of a model based on MF PCA by [33]. The Shared Principal Component (SPC) MF surrogate model consists in building the POD-based model separately with the sampled HF and then LF vectors and linking them assuming that the HF variations comply with those observed on the LF functions. Consequently, LF sampled vectors are similar to the principal components of the HF samples. The separation allows having two different mesh configurations with different fidelity levels. This method allows the fusion of LF and HF samples within a common metamodel and is applied to the design of turbomachinery blades. The SPC reduced the error to about 78% with respect to single-fidelity surrogates. The Space-Mapping surrogate presented the lowest error ratio among other tested methods. SPC appeared more efficient than space mapping, co-Kriging, and MFNIPOD [22]. Yonda et al. [233] used an MF ROM extensions based on the PODI [31] and compared it to non-linear least squares ROM [83, 237]. Two level fidelity simulations of 2D and 3D turbine blades are tested and used in an optimization loop. The POD-based method directly decomposes LF flow fields into main components covering a reduced solution subspace, where HF values are interpolated. Compared to a Kriging model based on design space variables, this model is smaller in size. The POD-based method showed a faster convergence than scalar and single-fidelity optimization while reaching the same profile efficiency and similar design. Thenon [210] proposed another MF co-Kriging based on the POD equivalent approach, the Principal Component Analysis (PCA), from machine learning community. The LF sampling vectors are directly projected on the reduced basis issued from the HF vectors. The co-Kriging approximation is performed among both Low- and High-fidelity level of projected coefficients. The method was applied to pressure maps outputs of a two-level oil tank model. The MF vectorial metamodel appeared effective with a condition on the correlation between the fidelity levels of the simulation outputs. Xiao et al. [228] proposed an alternative efficient approach that deals with the dynamic outputs based on a Galerkin projection in which a weight is assigned to two and three levels of MF data constructed by varying the number of basis vectors. Numerical tests show better results for three-level MF Kriging than single and two-level Kriging models. Following an online POD based topological optimization, [227] presented a MF approach to build a POD based ROM incrementally from successive variable-fidelity approximations of the global physical equations. Considering a non-deterministic framework [39] propose a sparse non-intrusive PCE adapted to MFM. Kriging and orthogonal polynomial covariance function are used to build the full-field model and an iterative scheme detects the optimal PCE basis in each fidelity level. The Kriging POD was also adapted by [140] to two fidelity levels. In the same spirit as PCE-Kriging, the basis functions are evaluated from a stochastic field where it is not necessary to calculate the Kriging trend functions separately.

Corrective MF

Poethke et al. [159] presented a method combining MF and POD and compared it to the scalar co-Kriging and single-fidelity Kriging. Instead of using the scalar objective value, the correction to the lower fidelity model is applied to the computational domain directly prior to any form of post-processing. The comparison between Gappy POD and single-fidelity Kriging was performed through UQ application for a number of gas turbine airfoils subject to different flow regimes. Wang et al. [221] proposed a multi-fidelity ROM for the reconstruction of a steady flow field. It performs a HF-LF corrective mapping of the bi-level ROM modal coefficients separately. This approach was applied to predict the flow with shock waves of a NACA0012 airfoil achieving better accuracy than the traditional Kriging ROM. Similarly, [128] proposed a multi-fidelity POD representation by computing the coarser and finer mesh CFD solutions $\varvec{S}_{LF}$ and $\varvec{S}_{HF}$ respectively, and add the POD interpolated field solution difference $\Delta \varvec{S}_{POD}$

$$\begin{aligned} \varvec{S}_{MF} = \varvec{S}_{LF} + \Delta \varvec{S}_{POD} \end{aligned}$$

(23)

Adaptive multi-fidelity vector models

Multifidelity offers an alternative to oversampling the HF response when seeking to represent meaningful surrogates. To address this challenge, methods to reduce computational cost involve creating many low-fidelity samples to supplement the few HF samples. The LF response has some error compared to the HF response, however, the samples are obtained at a much lower computational cost. These methods have reduced the computational cost or reduced the substitution error, or both. Sampling strategies have been adapted to the MF ROM to take advantage of the reduced cost of LF sampling or to adapt the MF ROM to HF. The adaptive ROMS and MFMs have been demonstrated to be a promising perspective to MFM ROM models [22, 233]. This section explores adaptive schemes involving MFM vector models.

Error-based sampling

An extension of the DoE Voronoi cells is performed in the framework of sequential sampling by [210] selecting the points in multiple Voronoi cells associated with the highest CV errors. Then, for each level of fidelity, a point is selected iteratively from the highest to the lowest corresponding error in each cell, until the calculation’s budget is reached. The fidelity level is chosen according to the IMSE weighted by the evaluation costs. In this paper, it has been observed that the two levels of fidelity should be sufficiently correlated. Besides, the LF computational cost should be limited to increase the method’s efficiency. The method has been applied to a POD co-Kriging and is compared to a MF scalar approach. It was shown that the POD vectorial approach improved the prediction performance of scalar models even with a poor HF-LF correlation. The corrective MF approach have been coupled to Gappy POD by [211] allowing to reconstruct the HF discrete gappy dataset with LF data. Benamara et al. [21] presents a MF adaptive approach based on the Gappy POD identifying a HF subspace to be covered by the LF vectors.

MF Bayesian framework

The MFNIRB proposed by [105] as an extension to the Non-Intrusive Reduced-Basis (NIRB) method of [35] takes into account constraints and shows promising results on a 2D analytical test case [22]. The method was compared to a scalar single-fidelity Kriging and used the constrained EI [12] criterion to sample iteratively the design space. However, this attempt is based on HF sampling on low-dimensional parameter space. The EI criterion was also applied by [145] in a MF reduced-order context. The expression of the EI criterion is evaluated using the LF data variance and mean. This modified EI is used to add LF and HF data, generated from LATIN Proper Generalized Decomposition (PGD) model reduction framework [41], assisting a mechanical part optimization problem.

Conclusion

This paper presents a non-exhaustive review of surrogate methods that address simulation cost issues in optimizing physical systems modeled with computer codes focusing on fluid/solid mechanics in potentially multidisciplinary settings. The reviewed methods’ common feature is the construction of surrogate models based on sequentially updated Design Of Experiments. The techniques reviewed include scalar, vector, and multi-fidelity surrogates, along with their associated infill strategies. Surrogate vectorial multi-fidelity models based on reduced-order approaches appear to outperform response surface approximations and control the overall simulation budget. However, the multi-fidelity approach’s feasibility depends on the quality and cost of physical models available for a given application. The trade-off between cost and accuracy in the multi-fidelity model is conditioned by the correlation between LF (supposed cheap) and HF (supposed expensive) simulators and their CPU ratio. The development of dedicated infill techniques requires criteria allowing both for the site’s choice and the new simulation’s fidelity level under the overall simulation budget constraint.