Efficient solvers for timedependent problems: a review of IMEX, LATIN, PARAEXP and PARAREAL algorithms for heattype problems with potential use of approximate exponential integrators and reducedorder models
 Florian De Vuyst^{1}Email author
DOI: 10.1186/s403230160063y
© De Vuyst. 2016
Received: 13 November 2015
Accepted: 2 March 2016
Published: 15 March 2016
Abstract
In this paper, we introduce and comment some recent efficient solvers for time dependent partial differential or ordinary differential problems, considering both linear and nonlinear cases. Here “efficient” may have different meanings, for instance a computational complexity which is better than standard time advance schemes as well as a strong parallel efficiency, especially parallelintime capability. More than a review, we will try to show the close links between the different approaches and set up a general framework that will allow us to combine the different approaches for more efficiency. As a complementary aspect, we will also discuss ways to include reducedorder models and fast approximate exponential integrators as fast global solvers. For developments and discussion, we will mainly focus on the heat equation, in both linear and nonlinear form.
Keywords
IMEX LATIN PARAEXP PARAREAL Performance Reduced order modelBackground
This paper gives an overview of recent alternative time advance schemes with interesting algorithmic features, including the possibility of parallel computations. First for linear problems, we will introduce the PARAEXP algorithm based on a superposition principle for achieving parallelintime computation. For nonlinear problems, the iterative LATIN method is a kind of splitting approach by alternating global linear solutions and local nonlinear projections. We will then discuss more general fixed point algorithms with a special focus on Newton and quasiNewton methods, separation of linear terms and nonlinear residuals in an implicitexplicit discretization strategy, then time subdomain decomposition and parallelintime computing involving coarse global and fine local propagators in the PARAREAL method.
The PARAEXP algorithm
 1.
First, define a partitioning of the time domain [0,T] into p time subintervals \([T_{j1},T_j]\), \(j=1,...,p\), \(0=T_0<T_1<...<T_p=T\);
 2.For each \(j=1,...,p\), solve the initial zero value problem$$\begin{aligned} {\dot{\varvec{v}}_j}(t) = A \varvec{v}_j(t) + \varvec{f}(t), \quad \varvec{v}_j(T_{j1})=0,\quad t\in [T_{j1}, T_j]; \end{aligned}$$(6)
 3.For each \(j=1,...,p\), solve the homogeneous problem(with the notation \(\varvec{v}_0(T_0):=\varvec{u}^0\)).$$\begin{aligned} {\dot{\varvec{w}}}_j(t) = A \varvec{w}_j(t), \quad \varvec{w}_j(T_{j1})=\varvec{v}_{j1}(T_{j1}), \quad t\in [T_{j1},T] \end{aligned}$$(7)
Nonlinear problems: an implicitexplicit IMEX time advance scheme
Iterative methods: the LATIN approach
A usual way to deal with nonlinear equations numerically is to use an iterative process within a fixed point algorithm. The LATIN (LArge Time INcremental) method pioneered by Ladevèze [15] and since then broadly used in computational structural Mechanics and material science (see [16] for a recent reference) solves timedependent problems (linear or nonlinear) according to a twostep iterative process. To separate the difficulties, equations are partitioned into two groups: (i) a group of equations being local in space and time, possibly nonlinear (representing equilibrium equations for example); (ii) a group of linear equations, possibly global in the spatial variable. Then adhoc spacetime approximations methods are used for the treatment of the global problem. Of course, spacetime local equations can be solved in parallel, what makes the LATIN method efficient and suitable for today’s HPC facilities. Let us emphasize that with LATIN, it is possible to solve hard nonlinear mechanics problems including thermodynamics irreversible problems (plasticity, friction as examples).
 1.
Initialization (\(k=0\)): let \(u_{(0)}\in L^2((0,T), H^1(\Omega ))\) an approximate solution (in space and time) of the nonlinear problem (it can be an approximate solution obtained with a coarse solver for example); compute \(\tilde{\kappa }_{(0)}=\kappa (u_{(0)})\);
 2.Iterate k, step 1 (global linear solution). Solve the linear problemwith given initial and boundary conditions.$$\begin{aligned} \partial _t u_{(k+1)}  \nabla \cdot (\tilde{\kappa }_{(k)} \nabla u_{(k+1)}) = f \end{aligned}$$
 3.Iterate k, step 2 (local projection over the admissible manifold). Compute$$\begin{aligned} \tilde{\kappa }_{(k+1)} = \kappa (u_{(k+1)}). \end{aligned}$$
 4.
Check convergence, \(k\leftarrow k+1\) if not and go to 2.
Newton and quasiNewton approaches
Spectral structure of the linearized problem
QuasiNewton approach
The PARAREAL method
 1.
Define a partition in time \([T_{j1},T_{j}]\), \(0=T_0<T_1<...<T_p=T\);
 2.
Define a cheap coarse propagator \(\mathscr {G}\) and a fine propagator \(\mathscr {F}\).
 3.
Initialization (\(k=0\)): \(\varvec{u}_{(0)}^0 = \varvec{u}^0\), \(\varvec{u}_{(0)}^{j+1} = \mathscr {G}(\varvec{u}_{(0)}^j)\);
 4.Loop on the iterates k:$$\begin{aligned} \varvec{u}_{(k+1)}^{j+1} = {\mathscr {G}(\varvec{u}_{(k+1)}^j)} \ + \ {\left( \mathscr {F}(\varvec{u}_{(k)}^j)  \mathscr {G}(\varvec{u}_{(k)}^j) \right) } \end{aligned}$$(17)
 5.
Check convergence, test the stop criterion.
One can imagine different choices of coarse solvers: loworder accurate time advances schemes, simplified equations, simplified models, discretizations on coarser meshes, etc. Reference papers like Bal and Maday [4] and Baffico et al. [3] show general convergence theorems for nonlinear ordinary differential systems using coarse time integrators as coarse solvers. Gander and Hairer in [9] also show a superlinear convergence of the parareal algorithm.
Putting all together
The Newton method to handle nonlinear terms with ROMs of dynamical systems

reducedorder models are expected to reproduce the stability of the system (for instance in the sense of Lyapunov, see [14] on this subject);

the local dynamics has to be reproduced, at least “at first order”, involving a compatibility of the spectral properties between full and reduced systems;

the area visited by the trajectories into the statespace may be defined over a nonlinear manifold rather than in a linear subspace. Thus nonlinear dimensionality reduction methods would be better candidates for reduction.
For timedependent problems, one can adopt a greedy incremental strategy during time by adapting/enriching the lowdimensional subspace when the principal components are changing during time. But the price to pay is to online evaluate some (highdimensional) nonlinear terms to control the error, what can be a penalizing factor of performance. If there is no other choice, parallelintime computing once again appears to be a complementary tool to keep global performance of the method.
Newton method and Galerkin projection method
 1.
(initialization). Use a coarse solver and compute \(\varvec{u}_{(0)}\). Loop over (k):
 2.
Compute M principal components \(\varvec{w}_{(k)}^m\), \(m=1,\ldots ,M\) or a suitable reduced basis from the knowledge of \(\varvec{u}_{(k)}\).
 3.
Assemble and compute in parallel \(\tilde{A}_{(k)}^M(t)\) and \(\varvec{r}_{(k)}^M(t)\) at all the discrete times.
 4.Solve the linear problemand compute$$\begin{aligned}&{\dot{ \varvec{a}}}_{(k+1)}^M = \tilde{A}_{(k)}^M(t)\,\varvec{a}_{(k+1)}^M + \varvec{r}_{(k)}^M(t), \quad t\in (0,T]\\&\varvec{a}_{(k+1)}^M(0) = \varvec{a}^0_{(k+1)}\in \mathbb {R}^M, \end{aligned}$$$$\begin{aligned} \varvec{u}_{(k+1)}^M(t) = \sum _{m=1}^M a^m_{(k+1)}(t)\, \varvec{w}^m_{(k)}. \end{aligned}$$
 5.
Test convergence after iterate k.
Remark 1
For the computation of the basis functions \(\varvec{w}_{(k)}^m\), one can of course use Proper Orthogonal Decomposition (POD) [22] or any other dimensionality reduction method. The update the reduced basis may also be done by incrementing the basis set within an adaptive learning algorithm.
Remark 2
In the step 3, it is assumed that both \(\tilde{A}_{(k)}(t)\) and \(\varvec{r}_{(k)}(t)\) have to be assembled and computed at all the discrete times. Of course, that may appear too penalizing for achieving high performance. Actually, one can consider additional reduction strategies for approximating both Jacobian matrix and right hand sides. This will be the aim of the following “Discussion” section.
Discussion about further reduction
There are many options to improve the whole numerical complexity of the algorithm using some additional approximations or reduction strategies.
Freezing up the Jacobian matrices
Adding coarse models
Achieving dimensionality reduction for \(\varvec{f}\)
If possible, one can also use a reducedorder approximation for \(\varvec{f}\). If the iterative algorithm is expected to converge towards a solution that has the same order of accuracy than the original one, one have to consider an accurate reducedorder model for \(\varvec{f}\). Once the empirical interpolation method may help us for that. However, if a globalintime reduction strategy is considered, it is possible that the dimension M of the loworder vector space becomes too large, leading to a degradation of the whole performance.
An alternative approach would be to consider a family of localintime empirical interpolation methods for \(\varvec{f}\). In this case, we should also consider local models \(\varvec{f}_{(k)}^j\) available in the time slice \([T_j, T_{j+1}[\) which can also be updated at each k from a learning process.
Approximate exponential integrators
In order to make the PARAEXP algorithm globally efficient, it is essential to compute fast and accurate approximate exponential integrators. In the case of the linear heat equation, we have to compute the exponential of a large scale, symmetric sparse matrix A. More precisely, for the the problem \({\dot{\varvec{u}}} = A\varvec{u}\) with initial data \(\varvec{u}(0)=\varvec{u}^0\), we have to compute the solution \(\varvec{u}(t)=\exp (tA)\varvec{u}^0\) for any \(t\in [0,T]\).
Closing discussion
From this review on efficient timeadvance solvers including IMEX, LATIN, PARAEXP and PARAREAL algorithms, we try to show the different ways and tracks to deal with largescale dynamical systems, linear and/or nonlinear terms. For the sake of an easy discussion, we have taken the example of the heat equation (linear or nonlinear). We are aware that this may be too restrictive and nonlinear computational mechanics including for example thermodynamics irreversible problems need more efforts and technical developments. Among the methods discussed above, some of them have been designed to address these problems. This is the case for the LATIN approach for example.
Time parallelization appears to be a promising key element of speedup. For problems with a small Kolmogorov width, reducedorder modeling may be a supplementary methodology to accelerate the whole time advance solution. For numerous reasons, it is interesting to cast a nonlinear problem into a sequence of linear problems within an iterative process. Linear problems are easier to deal with, and there are dedicated tools like the parallelintime PARAEXP method. On the other hand, an iterative process allows for achieving multifidelity adaptive solvers, using incremental, greedy or learning algorithms. Of course, we have to keep in mind that iterative methods may not converge. So in the design process of the numerical approach, one has to answer to the following questions: is the whole iterative process stable, is it possible to prove the convergence ? If the method is convergent, what is the rate of convergence ? Is it possible to accelerate the convergence ? At convergence, is it sure that the iterative algorithm converges to the solution obtaines with the accuracy we paid at the finest level ? For parallel algorithms, what is the effective speedup ?
Last but not least, managing multifidelity models and multilevel reducedorder models as well as parallelintime algorithms and learning algorithms implemented on distributed memory computer architecture necessarily require data management efforts and smart software engineering.
Conclusions
The first aim of this paper is to review different efficient timeadvance solvers (including IMEX, PARAEXP, LATIN, PARAREAL) and show connections between them. We also try to show the links with quasiNewton approaches and relaxation/projection methods to deal with nonlinear terms. Parallelintime algorithms appear to be a complementary and promising framework for the fast solution of timedependent problems. Finally, reducedorder models (PODbased, principal eigenstructure, a priori reduced bases, ...) can be possibly included to achieve better performance. In a future paper, we will achieve numerical experiments on different hybrid approaches.
Declarations
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Authors’ Affiliations
References
 Antoulas AC. An overview of approximation methods for largescale dynamical systems. Annu Rev Control. 2005;29:181–90.MathSciNetView ArticleGoogle Scholar
 Audouze C, De Vuyst F, Nair PB. Nonintrusive reducedorder modeling of parametrized timedependent partial differential equations. Numeri Methods Partial Differen Equ. 2013;29(5):1587–628.MathSciNetView ArticleMATHGoogle Scholar
 Baffico L, Bernard S, Maday Y, Turinici G, Zrah G. Parallelintime molecular dynamics simulations. Phys Rev E. 2002;66(5):057701.View ArticleGoogle Scholar
 Bal G, Maday Y. A “parareal” time discretization for nonlinear PDEs with application to the pricing of an American put. Recent developments in domain decomposition methods. Berlin: Springer; 2002. p. 189–202.View ArticleGoogle Scholar
 Chaturantabut S, Sorensen DC. Nonlinear model reduction via discrete empirical interpolation. SIAM J Sci Comput. 2010;32(5):2737–64.MathSciNetView ArticleMATHGoogle Scholar
 Chinesta F, Ammar A, Lemarchand F, Beauchène P, Boust F. Parallel time integration and high resolution homogenization. CMAME. 2008;197(5):400–13.MATHGoogle Scholar
 Cortial J, Fahrat C, Guibas LJ, Rajashekhar M. Compressed sensing and timeparallel reducedorder modeling of structural health monitoring using DDDAS. Computational scienceICCS 2007. Berlin: Springer; 2007. p. 1171–9.View ArticleGoogle Scholar
 Filbet F, Negulescu C, Yang C. Numerical study of a nonlinear heat equation for plasma Physics. Int J Comp Math. 2012;89(8):1060–82.MathSciNetView ArticleMATHGoogle Scholar
 Gander MJ, Hairer E. Nonlinear convergence analysis for the PARAREAL algorithm. In: Domain decomposition methods in Science and Engineering. 2008. vol. 60, p. 4556.Google Scholar
 Gander MJ, Güttel S. PARAEXP: a parallel integrator for linear initialvalue problems. SIAM J Sci Comput. 2013;35(2):C123–42.MathSciNetView ArticleMATHGoogle Scholar
 Gander MJ. 50 years of time parallel time integration. Multiple shooting and time domain decomposition. Berlin: Springer; 2015.Google Scholar
 Hochbruck M, Lubich C. On Krylov subspace approximations to the matrix exponential operator. SIAM J Num Anal. 1997;34(5):1911–25.MathSciNetView ArticleMATHGoogle Scholar
 Kalashnikova I, Barone MF. On the stability of a Galerkin reduced order model (ROM) of compressible flow with solid wall and farfield boundary treatment. IJNME. 2010;83:1345–75.MathSciNetView ArticleMATHGoogle Scholar
 Kalashnikova I, Barone MF, Arunajatesan S, von Bloemen Waanders BG. Construction of energystable projectionbased reduced order models. Appl Math Comp. 2014;249:569–96.MathSciNetView ArticleMATHGoogle Scholar
 Ladevèze P. Non linear computational structural mechanics new approaches and nonincremental methods of calculation. New York: SpringerVerlag; 1999.MATHGoogle Scholar
 Ladevèze P, Passieux JC, Néron D. The LATIN multiscale computational method and the proper generalized decomposition. Comput Methods Appl Mech Eng. 2010;199(21):1287–96.MathSciNetView ArticleMATHGoogle Scholar
 Lions J, Maday Y, Turinici G. A “parareal” in time discretization of PDE’s. Comptes Rendus de l’Académie des Sciences, Séries I, Mathematics. 2001;332(7):661–8.MathSciNetView ArticleMATHGoogle Scholar
 Maday Y, Nguyen NC, Patera AT, Pau GSH. A general multipurpose interpolation procedure: the magic points. Commun Pure Appl Anal. 2009;81:383–404.MathSciNetMATHGoogle Scholar
 Minion M. A hybrid parareal spectral deferred corrections method. Commun Appl Math Comput Sci. 2010;5(2):265–301.MathSciNetView ArticleMATHGoogle Scholar
 Nievergelt J. Parallel methods for integrating ordinary differential equations. Comm ACM. 1964;7:731–3.MathSciNetView ArticleMATHGoogle Scholar
 Prud’homme C, Rovas D, Veroy K, Maday Y, Patera AT, Turinici G. Reliable real time solution of parametrized partial differential equations: reducedbasis output bound methods. J Fluids Eng. 2002;124(1):70–80.View ArticleGoogle Scholar
 Willcox K, Peraire J. Balanced model reduction via the proper orthogonal decomposition. AIAA J. 2002;40(11):2323–30.View ArticleGoogle Scholar