So what is loop unrolling then?

Loop unrolling is one of the oldest tricks in the optimizing-compiler playbook, and the idea is disarmingly simple. Instead of executing a loop body once per iteration and paying the overhead of a branch and a counter update every single time, you write out the body two, four, or eight times per iteration, advancing the counter by that many steps at once.

Take a trivial example, nothing to do with our Poisson problem yet:

do i = 1, n
   a(i) = 2.0_dp * a(i)
end do

Unrolled by a factor of two, this becomes:

do i = 1, n, 2
   a(i)   = 2.0_dp * a(i)
   a(i+1) = 2.0_dp * a(i+1)
end do

modulo a bit of bookkeeping if n is odd, which we’ll conveniently ignore for now. Fewer branches, fewer counter increments, and — crucially — twice as much independent work exposed to the compiler within a single loop trip. That last point is really the whole appeal: with two statements sitting side by side and no dependency between them, the compiler is free to interleave them, pack them into a single vector instruction, or schedule them however the hardware likes. Unrolling doesn’t make the CPU do less work; it makes the scheduler’s job easier by giving it more to look at at once. So, naturally, the question we should ask is: does the same trick save Gauss-Seidel?

A tempting shortcut that doesn’t work

Let’s try the obvious thing and unroll our Gauss-Seidel inner loop by two, changing nothing else:

do i = 2, n-2, 2
   u(i, j)   = 0.25_dp*(dx2*b(i,j)   + u(i-1,j)   + u(i+1,j)   &
                                     + u(i,j-1)   + u(i,j+1))
   u(i+1, j) = 0.25_dp*(dx2*b(i+1,j) + u(i,j)     + u(i+2,j)     &
                                     + u(i+1,j-1) + u(i+1,j+1))
end do

Look closely at the second statement: it reads u(i,j) which is exactly the value the first statement just finished computing, two lines above. We haven’t removed the loop-carried dependency at all. We’ve just moved a copy of it inside the loop body, where it’s now a same-iteration dependency instead of a cross-iteration one. The compiler still can’t touch these two statements independently, still can’t vectorize them together, and still has to execute them strictly in order. We’ve paid for the bookkeeping of unrolling (larger code, an odd-n remainder to handle) and bought ourselves precisely nothing.

This is worth sitting with for a second, because it’s a genuinely common trap: unrolling looks like it should help with dependencies, since it’s so often reached for as a vectorization enabler, but by itself it does nothing whatsoever to break a true dependency chain. It’s very good at exposing parallelism that already exists (as in our trivial a(i) = 2*a(i) example) but is completely powerless against parallelism that has to be created. Gauss-Seidel falls squarely in the second category: the dependency isn’t a scheduling accident the compiler failed to notice, it’s baked into the mathematics of the update rule.

So if unrolling alone is a dead end, what’s missing? The answer is that the dependency itself has to be rewritten away before unrolling can do any good. That means reaching for a bit of algebra, not just a bit of loop restructuring.

Fair enough, let’s do that for Gauss-Seidel!

To keep the algebra honest and readable, let’s forget about the full 2D grid for now, and work with the 1D Poisson equation, , discretized on a uniform grid as

Solving for gives the update rule

so, unsurprisingly, the constant playing the role of our earlier is now . Sweeping left to right and updating in place, Gauss-Seidel reads:

do i = 2, n-1
   u(i) = c*(u(i-1) + u(i+1) + dx2*b(i))
end do

Exactly one neighbor here is a problem: u(i-1) was overwritten by the previous loop trip, while u(i+1) is still the untouched, old value (the sweep hasn’t reached it yet). That’s the whole dependency, laid completely bare with nothing else to distract from it.

Writing it as a recursion

Group everything that doesn’t depend on the sweep’s own progress into a single per-column quantity,

which is fully known before the block starts (it’s old data, untouched by anything we’re computing right now). The update collapses to a clean, one-term affine recursion:

where um is shorthand for the last already-updated value feeding into this block. This is precisely the shape that makes the dependency chain visible as just a recursion in — nothing more, nothing less.

Solving the recursion two steps ahead

Rather than computing and then from it, we solve the recursion explicitly for both, purely in terms of quantities that existed before the block started:

Look at what happened: no longer references at all. Both outputs are now pure functions of um, , and . All quantities that existed before this pair of updates began. The intra-pair dependency that plain unrolling couldn’t touch (the “unroll-only” kernel from the previous section) has been algebraically dissolved.

In code, defining , this gives the following kernel:

do i = 2, n-2, 2
   um   = u(i-1)
   t0   = u(i+1) + dx2*b(i)
   t1   = u(i+2) + dx2*b(i+1)
   u(i)   = c *(um + t0)
   u(i+1) = c2*um + c2*t0 + c*t1 ! Does not depend on u(i) !
end do

Two independent expressions, no read of one output by the other’s computation. Exactly what the compiler needs to schedule them concurrently instead of serially.

Back to 2D: nothing new needed

This is where the 1D detour pays off. In the real 2D stencil, the north, south, and source contributions play no role whatsoever in creating the i-dependency — they’re either values from a row Gauss-Seidel has already finished, or values from a row it hasn’t touched yet, but never anything the i-loop itself is computing. So they simply fold into the same lumped we just defined:

and the exact same two-line substitution applies, verbatim, giving you the unrolled kernel:

real(dp), parameter :: c  = 0.25_dp
real(dp), parameter :: c2 = 0.0625_dp
do j = 2, n-1
   do i = 2, n-2, 2
      um   = u(i-1, j)
      tmp1 = dx2*b(i, j)   + (u(i+1, j) + u(i, j+1) + u(i, j-1))
      tmp2 = dx2*b(i+1, j) + (u(i+2, j) + u(i+1, j+1) + u(i+1, j-1))
      u(i, j)   = c *(um + tmp1)
      u(i+1, j) = c2*um + (c2*tmp1 + c*tmp2)
   end do

   ! Handles the case where an odd number of grid points is used.
   if (mod(n, 2) == 1) then
      tmp1 = dx2*b(n-1, j) + (u(n, j) + u(n-1, j+1) + u(n-1, j-1))
      u(n-1, j) = c*(u(n-2, j) + tmp1)
   end if
end do

The trailing if block just mops up a leftover column when n is odd, i.e. the same remainder bookkeeping any unrolled loop needs, nothing conceptually new.

Does it actually work?

Ok, so the new kernel looks a bit more complicated than our original one. But mathematically, it’s all the same thing, so is it truly more efficient? Time to check the theory against the machine. Same workflow as before: compile with -O3 -march=native -S, mark the inner loop, and let OSACA loose on it.

                                     Port pressure in cycles                                      
     |  0   - 0DV  |  1   |  2   -  2D  |  3   -  3D  |  4   |  5   |  6   |  7   ||  CP  | LCD  |
--------------------------------------------------------------------------------------------------
  74 |             |      |             |             |      |      |      |      ||      |      |   .L5:
  75 |             |      | 0.50   0.50 | 0.50   0.50 |      |      |      |      ||      |      |   vmovsd 16(%rax), %xmm1
  76 |             |      | 0.50   0.50 | 0.50   0.50 |      |      |      |      ||      |      |   vmovsd 8(%rax), %xmm0
  77 | 0.00        | 0.00 |             |             |      | 0.50 | 0.50 |      ||      |      |   addq $16, %rdx
  78 | 0.50        | 0.50 | 0.50   0.50 | 0.50   0.50 |      |      |      |      ||  4.0 |      |   vaddsd 8(%rax,%rsi,8), %xmm1, %xmm1
  79 | 0.50        | 0.50 | 0.50   0.50 | 0.50   0.50 |      |      |      |      ||      |      |   vaddsd (%rax,%rsi,8), %xmm0, %xmm2
  80 | 0.50        | 0.50 | 0.50   0.50 | 0.50   0.50 |      |      |      |      ||  4.0 |      |   vaddsd 8(%rax,%rcx,8), %xmm1, %xmm1
  81 | 0.50        | 0.50 | 0.50   0.50 | 0.50   0.50 |      |      |      |      ||      |      |   vaddsd (%rax,%rcx,8), %xmm2, %xmm2
  82 | 0.00        | 0.00 |             |             |      | 0.50 | 0.50 |      ||      |      |   addq $16, %rax
  83 | 0.50        | 0.50 | 0.50   0.50 | 0.50   0.50 |      |      |      |      ||  4.0 |      |   vfmadd231sd -8(%rdx), %xmm5, %xmm1
  84 | 0.50        | 0.50 | 0.50   0.50 | 0.50   0.50 |      |      |      |      ||      |      |   vfmadd231sd -16(%rdx), %xmm5, %xmm2
  85 | 0.50        | 0.50 |             |             |      |      |      |      ||  4.0 |      |   vmulsd %xmm4, %xmm1, %xmm1
  86 | 0.50        | 0.50 |             |             |      |      |      |      ||      |      |   vaddsd %xmm3, %xmm2, %xmm0
  87 | 0.50        | 0.50 |             |             |      |      |      |      ||      |      |   vmulsd %xmm4, %xmm0, %xmm0
  88 | 0.50        | 0.50 |             |             |      |      |      |      ||  4.0 |      |   vfmadd132sd %xmm6, %xmm1, %xmm2
  89 | 0.50        | 0.50 |             |             |      |      |      |      ||  4.0 |  4.0 |   vfmadd132sd %xmm6, %xmm2, %xmm3
  90 |             |      |             |             |      | 1.00 |      |      ||  1.0 |      |   vunpcklpd %xmm3, %xmm0, %xmm0
  91 |             |      | 0.00        | 0.00        | 1.00 |      |      | 1.00 ||  0.0 |      |   vmovupd %xmm0, -16(%rax)
  92 | 0.00        | 0.00 |             |             |      | 0.00 | 1.00 |      ||      |      |   cmpq %rax, %rdi
  93 |             |      |             |             |      |      |      |      ||      |      | * jne .L5

       5.50          5.50   4.00   4.00   4.00   4.00   1.00   2.00   2.00   1.00    29.0    4.0

If you look at the last two columns, two things should stand out. The critical path CP got a bit longer, but the loop-carried dependencies (LCD) dropped from 12 to 4 cycles per loop trip — and since each trip now covers two elements, that’s roughly a 6× drop per grid point.

That’s the static prediction. Time to see if the actual machine agrees — and, more interestingly, to see why it agrees, one compilation flag at a time.

Experiment 1 — baseline, no optimization

First, a sanity check. Compiled with -O0, so no scheduling cleverness of any kind gets to play a role:

Two things worth checking here before moving on. First, all three solvers reach the solution in the same number of iterations as the earlier posts — confirming the unrolled kernel really is computing the same thing as the textbook one, algebra and all, not some subtly different (and subtly wrong) approximation. Second, at -O0, the unrolled kernel is already reliably faster than the textbook kernel, by almost exactly the ratio convergence theory predicts for Gauss-Seidel over Jacobi (2×) — a nice confirmation that even without any compiler scheduling to exploit, exposing two independent computations per loop trip buys you something, purely from halving the loop overhead. But nobody ships -O0 code, so this is a baseline, not the result.

Experiment 2 — the critical one

Now compile with real optimization, but deliberately keep vectorization switched off (-O3 -march=native -mtune=native -fno-tree-vectorize), so that any gains we see can’t be attributed to SIMD at all:

This is the number that matters. With vectorization explicitly disabled, the unrolled kernel is still six times faster per sweep than the textbook kernel, and lands within 5% of Jacobi’s per-sweep cost. Since neither Gauss-Seidel kernel can be vectorized under this flag anyway, this gain has nothing to do with packing multiple grid points into one instruction. It comes entirely from something more subtle: reducing the loop-carried dependency lets the CPU’s out-of-order scheduler overlap the scalar work of consecutive iterations, filling execution ports that the textbook kernel’s 12-cycle dependency chain left idle. Fewer cycles spent waiting, same number of instructions issued. This is the empirical confirmation of the LCD-versus-throughput story from the OSACA table above, isolated from vectorization entirely.

Experiment 3 — turning vectorization back on

Finally, the same three kernels, same flags, but with vectorization allowed:

Both Gauss-Seidel kernels are completely unchanged, down to the microsecond. That’s expected once you know the assembly: gfortran never vectorizes either GS kernel regardless of this flag, so there’s nothing for the flag to act on. Vectorization was never the mechanism behind experiment 2’s speedup. It’s a separate story that only applies to Jacobi.

And even for Jacobi, notice the gain is real but modest: 166 µs → 115 µs, about 1.4×, not the ~4× you might expect from packing four doubles into a single AVX register. This is the by-product the LCD story predicted but didn’t fully explain on its own: vectorization only pays off to the extent that a kernel’s compute is the bottleneck, and Jacobi’s inner loop, despite having no loop-carried dependency at all, is largely memory-bound at this problem size. It streams through the whole grid every sweep with essentially no data reuse, so DRAM bandwidth caps the achievable speedup well below what the instruction-level analysis alone would suggest.

Put the three experiments side by side and the conclusion is unambiguous: it’s the absence (or reduction) of a loop-carried dependency, not vectorization, that lets the scheduler do its job. Vectorization is a nice bonus when it’s available. It does still help Jacobi, just less than you’d naively guess. But it was never the reason Gauss-Seidel was slow, and reducing the LCD was already enough, on its own, to close almost the entire performance gap with Jacobi. Actually, this unrolled Gauss-Seidel kernel hands us the solution almost 3 seconds faster than the Jacobi one. Mission accomplished! Or so it appears…

But look closely at the mnemonics

Here’s the detail that’s easy to miss if you only look at the CP/LCD numbers and declare victory. If you look at the far right of the OSACA output, that is the assembly for inner-most loop (i) of our kernel. Every single arithmetic instruction in that table is still a scalar op — vaddsd, vmulsd, vfmadd231sd, vfmadd132sd, all operating on individual doubles in %xmm registers, not a single packed pd instruction among them. Compare that to Jacobi’s inner loop, built entirely out of vaddpd/vmulpd/vfmadd213pd on 256-bit %ymm registers, four grid points at a time. The compiler never vectorized this loop. It didn’t need to, to hit Jacobi’s throughput. It got there by exposing two independent scalar chains (one for u(i,j), one for u(i+1,j)) that fill different execution ports at the same time, rather than by packing four elements into one instruction the way Jacobi does. The one and only place the two lanes actually meet is right at the end, at line 90: a vunpcklpd stitching the two independently-computed scalars into a single 128-bit register just before the store. That’s not vectorized computation. It’s a vectorized write of two scalar results, a small, free bonus riding on top of the real work, not the source of the speedup itself.

This distinction matters more than it might look like at first glance, and it’s the seed of the next post’s cliffhanger: because the compiler is achieving Jacobi-level throughput through instruction-level parallelism across ports rather than true SIMD, this kernel still can’t be handed a wider vector width for free, and it certainly can’t be split across threads. That tiny 4-cycle pair-to-pair dependency, however cheap, is still a sequential handoff that no do concurrent or OpenMP directive could safely break apart. We’ve matched (and even slightly exceeded) Jacobi’s single-core speed, but not its parallelizability. Those turn out to be two different prizes entirely. And only one of them is in hand.

Can we go even further?

Could we push the substitution one step further? Unroll by 4 instead of 2, and squeeze out even more performance? Not really. We’d be at risk of marginally reducing an LCD that’s already a non-issue, while greatly increasing the pressure on the CPU ports.

Here’s why. Going back to the general recursion from before,

each additional element you fold into the substitution costs you one more multiply-add. By the time you reach the degree-4 unrolling, you’re resolving a 4-term cascading sum, every term weighted by its own power of . Averaged across the four outputs, that’s roughly twice the multiply-add work per element compared to degree 2. And recall where we ended up last section: the throughput floor is already the governing bound at degree 2 (2.75 cycles/element, versus an LCD of only 2.0). Doubling the arithmetic per element doesn’t touch the LCD’s role in the story at all. It just raises the throughput floor further, from the bound that’s actually constraining us. You’d be spending real port-pressure budget to shave cycles off a dependency chain that was already comfortably hidden behind other work. A losing trade, and one you can see coming from the algebra alone, without even needing to compile anything.

So degree 2 isn’t an arbitrary stopping point. It’s exactly the amount of unrolling needed to push the LCD below the throughput floor, and not a bit more. Past that point, every extra unit of unrolling is pure cost with no corresponding benefit. At least for Gauss-Seidel.

So, where does that leave us?

We made it: a Gauss-Seidel kernel that keeps the mathematical convergence advantage over Jacobi. Same iteration count as the textbook version, while very nearly matching Jacobi’s raw computational efficiency per sweep. A serial win on both fronts at once, which is exactly the “have your cake and eat it too” outcome we set out chasing at the start of this post.

But, there’s always a but, it’s a serial win only. The kernel is still built entirely out of scalar instructions. The compiler (gfortran 15.3) never vectorized it for my particular CPU. It got to Jacobi-level throughput through a different route (independent scalar chains sharing execution ports, not packed SIMD lanes). Worse, that tiny residual pair-to-pair dependency, however cheap, is still a genuine sequential handoff. No do concurrent, no OpenMP directive, no compiler flag can safely split this loop across threads or hand it a wider vector register, because the math still insists that pair cannot start until pair has written its result.

For a single core on a laptop, that’s a complete success. But it’s a dead end for anything that needs to scale. And scaling is, after all, the entire point of numerical computing once your grids get big enough to matter. If we want a Gauss-Seidel kernel that vectorizes and multithreads, unrolling and clever algebra won’t get us there. We’ll need to change the order in which we visit grid points altogether, which is exactly where red-black ordering comes in, next time.

Acknowledgement – I have to give credit where credit is due. It is Ivan Pribec who showed me this particular unrolling trick for Gauss-Seidel. It is also he who put osaca under my radar, so none of what I presented here would have been possible without him. Thanks a lot!

If you want to read more of my stuff

• Gauss-Seidel is faster than Jacobi. Or is it?
• Jacobi method: From a naïve implementation to a modern Fortran multithreaded one
• Is Fortran better than Python for teaching the basics of numerical linear algebra?

Solving the 2D Poisson equation with the Gauss-Seidel method

Consider a system of linear equations

where is an invertible matrix. In the rest of this post, we’ll assume that this problem arises from the finite-difference discretization of the two-dimensional Poisson equation on the unit-square (see here for more details). If you’ve ever had a numerical linear algebra, the Jacobi method is probably the first iterative method you’ve seen to solve this kind of problems. You may even remember your lecturer telling you that, even though easy to implement and analyse, the Jacobi method can be terribly slow. And this is usually the stepping stone for introducing the Gauss-Seidel method.

Numerical experiments with Fortran

Alright! So now that we know the Gauss-Seidel method is theoretically sound, let us implement it in Fortran and see how good it is in practice. For that purpose, we’ll use the same test case as before: the 2D Poisson equation discretized with the standard five-point Laplacian approximation along with homogeneous Dirichlet boundary conditions. For more details on the derivation of the discrete problem, you can go back to the post on the Jacobi method (here).

integer, parameter :: dp = selected_real_kind(15, 307)

pure subroutine textbook_kernel(nx, ny, u, b, dx)
    implicit none (type, external)
    integer, intent(in) :: nx, ny
    real(dp), intent(inout) :: u(nx, ny)
    real(dp), intent(in) :: b(nx, ny), dx
    integer :: i, j
    do j = 2, ny-1
        do i = 2, nx-1
            u(i, j) = 0.25_dp * (b(i, j)*dx**2 + u(i+1, j) + u(i, j+1) &
                                               + u(i-1, j) + u(i, j-1))
        enddo
    enddo
end subroutine

function solver(b, tol, maxiter) result(u)
    implicit none
    real(dp), intent(in)  :: b(:, :), tol
    integer, intent(in)   :: maxiter
    real(dp), allocatable :: u(:, :)
    ! Internal variables.
    integer               :: nx, ny, i, j, iteration
    real(dp), allocatable :: v(:, :)
    real(dp)              :: dx, l2_norm

    ! Initialize variables.
    nx = size(b, 1) ; ny = size(b, 2)
    if (nx /= ny) then
        error stop "Number of points in each direction need to be equal."
    endif
    
    dx = 1.0_dp / (nx -1)
    allocate(u(nx, ny), v(nx, ny), source=0.0_dp)
    l2_norm = 1.0_dp
    iteration = 0

    ! Beginning of the Gauss-Seidel iterative method.
    do while ((iteration < maxiter) .and. (l2_norm > tol))
        ! Save previous solution for norm correction computation.
        if (mod(iteration, 1000) == 0) v = u
        ! Gauss-Seidel kernel.
        call textbook_kernel(nx, ny, u, b, dx)
        ! Compute norm of the correction.
        if (mod(iteration, 1000) == 0) l2_norm = norm2(u - v)
        ! Update iteration counter.
        iteration = iteration + 1
    enddo
end function

pure subroutine jacobi_kernel(nx, ny, u, v, b, dx)
    implicit none (type, external)
    integer, intent(in)   :: nx, ny
    real(dp), intent(out) :: u(nx, ny)
    real(dp), intent(in)  :: v(nx, ny), b(nx, ny), dx
    integer               :: i, j
    do concurrent(j=2:ny-1, i=2:nx-1)
        u(i, j) = 0.25_dp*(b(i, j)*dx**2 + v(i+1, j) + v(i-1, j) &
                                         + v(i, j+1) + v(i, j-1))
    enddo
end subroutine

! Jacobi update
u(i, j) = 0.25_dp*(b(i, j)*dx**2 + v(i+1, j) + v(i, j+1) &
                                 + v(i-1, j) + v(i, j-1))

! Gauss-Seidel update
u(i, j) = 0.25_dp*(b(i, j)*dx**2 + u(i+1, j) + u(i, j+1) &
                                 + u(i-1, j) + u(i, j-1))

u(i, j)   = 0.25_dp * (u(i+1, j) + u(i-1, j) + u(i, j+1)   + u(i, j-1)  )
u(i+1, j) = 0.25_dp * (u(i+2, j) + u(i, j)   + u(i+1, j+1) + u(i+1, j-1))
u(i+2, j) = 0.25_dp * (u(i+3, j) + u(i+1, j) + u(i+2, j+1) + u(i+2, j-1))
u(i+3, j) = 0.25_dp * (u(i+4, j) + u(i+2, j) + u(i+3, j+1) + u(i+3, j-1))

u(i, j)   = 0.25_dp * (v(i+1, j) + v(i-1, j) + v(i, j+1)   + v(i, j-1)  )
u(i+1, j) = 0.25_dp * (v(i+2, j) + v(i, j)   + v(i+1, j+1) + v(i+1, j-1))
u(i+2, j) = 0.25_dp * (v(i+3, j) + v(i+1, j) + v(i+2, j+1) + v(i+2, j-1))
u(i+3, j) = 0.25_dp * (v(i+4, j) + v(i+2, j) + v(i+3, j+1) + v(i+3, j-1))

Is all hope lost?

Not at all! The failure of textbook Gauss-Seidel on modern hardware is not a mathematical defect. It is a dependency and parallelism problem. To make Gauss-Seidel fast on modern CPUs, we must eliminate the loop-carried dependency without losing its superior convergence rate. This brings us to grid reordering strategies:

1. Red-Black (Checkerboard) Ordering: By splitting the grid into alternate red and black nodes, we can update half the domain in parallel using pure SIMD sweeps, recovering full SIMD vectorization while keeping the Gauss-Seidel convergence rate.
2. Domain Decomposition & Multi-Scale Layouts: Partitioning the domain into independent blocks allows us to execute localized Gauss-Seidel sweeps inside L1/L2 caches before exchanging boundary data.

In the next post, we will go through the details of the Red-Black Gauss-Seidel, implement it in Fortran and explore whether we can recover Gauss-Seidel’s convergence advantage without sacrificing the hardware efficiency of Jacobi.

Abstract

Effective control of fluid flows is critical across transportation, energy and medicine, where it can increase lift, reduce drag, enhance mixing and attenuate noise. Yet fluids are notoriously difficult to control because they involve high-dimensional, nonlinear and multiscale dynamics that resist conventional approaches. Reinforcement learning has driven remarkable progress in fields such as protein folding and complex games, which have shared benchmarks and standaridized environments. Fluid dynamics has lacked such infrastructure, so each controller is typically tuned to a single geometry and operating condition, making progress difficult to accumulate, transfer and compare. Here we introduce HydroGym, a solver-independent reinforcement learning platform providing more than 60 validated, openly available flow control environments spanning from canonical laminar flows to complex turbulent flows, with systematic progression in the Reynolds number up to , and Mach number variations in two and three dimensions. Across these environments, agents repeatedly discover robust control principles, including boundary layer manipulation, disruption of acoustic feedback and reorganization of turbulent wakes. Critically, we demonstrate a proof of concept of zero-shot transfer, in which agents that are trained exclusively in inexpensive surrogate environments are deployed to challenging real-world scenarios such as three-dimensional wing section. We achieve a 38% reduction in local skin friction while reducing exploration costs by four orders of magnitude compared with direct on-wing optimization. As this transfer exploits shared near-wall physics, the breadth of generalization remains open, suggesting a new pathway for research toward policy generalization across computationally prohibitive simulation environments. By offering a common, extensible foundation for reproducible research, HydroGym moves flow control from isolated case studies toward a cohesive community effort.

Links

Published paper

Abstract

Direct solvers for linear algebraic systems scale cubically in the problem’s dimension, rapidly becoming intractable for large-scale problems, while sparse factorization may still require quadratic storage du fill-in. Krylov techniques avoid these costs by needing only a routine that computes a matrix-vector product, iteratively building a subspace from which the solution is obtained. LightKrylov is a Fortran package providing a suite of such Krylov methods along with an easy-to-use high level API based on abstract types. It is primarily intended for applications where the linear operator of interest is only available implicitly via a matrix-vector subroutine and enables users to maximally re-use existing components of their code base (including parallelization), thus requiring a minimal set of changes without sacrificing computational performance.

Links

Published paper

Solving a linear system with the Jacobi method?

Consider the system of linear equations

where is an invertible matrix. If you ever had a course on numerical linear algebra, you have seen various algorithms to solve this problem. These are divided in two categories: direct solvers targetting small- to medium-sized dense matrices, and iterative solvers for large sparse matrices.

Among the zoo of iterative methods, the Jacobi method is probably the first one you’ve encountered. There are two reasons for that:

1. It is easy to implement, no matter the programming language.
2. Its theoretical analysis is rather simple, even for undergrad students.

It does come with its limitations though: it does not work for all possible matrices and the convergence is rather slow (i.e. it requires many iterations). Because of these, the Jacobi method is not a viable alternative compared to the (preconditioned) conjugate gradient or multigrid methods and is thus hardly used in production codes. It is however, in my opinion, a fantastic learning example. So, how does it work?

The Poisson’s equation on the unit square

The Poisson’s equation is an elliptic partial differential equation (PDE) appearing in numerous fields of physics. Let’s parse what this means:

• PDE – The solution of the equation depends on more than one variable, where the variables are typically the different spatial dimensions.
• Elliptic – The solution exhibits a certain notion of smoothness, whatever that means mathematically. Typically, it implies that the solution will not exhibit any discontinuities or very steep fronts in contrast to what you may see for hyperbolic equations (think shock waves for instance).

Mathematically, it reads

along with appropriate boundary conditions. In the rest of this post, we’ll consider one of its simplest variations. The domain is the unit square, i.e.  and we will consider only homogenous Dirichlet boundary conditions, i.e.  on the boundaries of the square. Our problem thus is

Note that the problem is sufficiently simple that you can express its analytical solution using Fourier series. But we are computational scientists, so we’ll solve the problem numerically.

Let Fortran shine!

Alright! It’s time for what you all expected: the Fortran implementation. We’ll start with a simple translation to Fortran of the pseudo-code. This implementation will be our baseline. We will then incrementally improve it by using various tips and tricks with a particular constraint: use only standard-compliant Fortran code. I’ll try to explain the rationale behind every decision I make along the way. By the end of our journey, we’ll have a standard-compliant implementation which can naturally leverage multithreaded computations without having have to write a single openMP pragma. Performance-wise, we’ll end up with a 20x to 30x speed-up compared to our baseline implementation without every leaving the realm of Fortran. Too good to be true? Bare with me then!

integer, parameter :: dp = selected_real_kind(15, 307)

pure subroutine textbook_kernel(nx, ny, u, v, b, dx)
    implicit none
    integer, intent(in) :: nx, ny
    real(dp), intent(out) :: u(nx, ny)
    real(dp), intent(in) :: v(nx, ny), b(nx, ny), dx
    integer :: i, j
    do j = 2, ny-1
        do i = 2, nx-1
            u(i, j) = 0.25_dp*(b(i, j)*dx**2 - v(i+1, j) - v(i-1, j) &
                                             - v(i, j+1) - v(i, j-1))
        enddo
    enddo
end subroutine

function textbook_solver(b, tol, maxiter) result(u)
    implicit none
    real(dp), intent(in) :: b(:, :), tol
    integer, intent(in) :: maxiter
    real(dp), allocatable :: u(:, :)
    ! Internal variables
    integer :: nx, ny, i, j, iteration
    real(dp), allocatable :: v(:, :)
    real(dp) :: dx, l2_norm
    ! Initialize variables
    nx = size(b, 1); ny = size(b, 2); dx = 1.0_dp/(nx - 1)
    if (nx /= ny) then
        error stop "Number of points in each direction need to be equal."
    endif
    allocate (u(nx, ny), source=0.0_dp)
    allocate (v(nx, ny), source=0.0_dp)
    l2_norm = 1.0_dp
    iteration = 0
    ! Begining of the Jacobi iterative method.
    do while ((iteration < maxiter) .and. (l2_norm > tol))
        ! Jacobi iteration.
        call textbook_kernel(nx, ny, v, u, b, dx)
        ! Compute error norm.
        l2_norm = norm2(u - v)
        ! Update variable.
        u = v
        ! Update iteration counter.
        iteration = iteration + 1
    end do
    print *, "Textbook solver :"
    print *, "    - Number of iterations :", iteration
    print *, "    - l2-norm of the error :", l2_norm
end function

    u = v

function nocopy_solver(b, tol, maxiter) result(u)
    implicit none
    real(dp), intent(in) :: b(:, :), tol
    integer, intent(in) :: maxiter
    real(dp), allocatable :: u(:, :)
    ! Internal variables
    integer :: nx, ny, i, j, iteration
    real(dp), allocatable :: v(:, :)
    real(dp) :: dx, l2_norm
    ! Initialize variables
    nx = size(b, 1); ny = size(b, 2); dx = 1.0_dp/(nx - 1)
    if (nx /= ny) then
        error stop "Number of points in each direction need to be equal."
    endif
    allocate (u(nx, ny), source=0.0_dp)
    allocate (v(nx, ny), source=0.0_dp)
    l2_norm = 1.0_dp
    iteration = 0
    ! Begining of the Jacobi iterative method.
    do while ((iteration < maxiter) .and. (l2_norm > tol))
        ! Jacobi iteration (no copy).
        call textbook_kernel(nx, ny, v, u, b, dx)
        call textbook_kernel(nx, ny, u, v, b, dx)
        ! Compute error norm.
        l2_norm = norm2(u - v)
        ! Update iteration counter.
        iteration = iteration + 2
    end do
    print *, "No-copy solver  :"
    print *, "    - Number of iterations :", iteration
    print *, "    - l2-norm of the error :", l2_norm
end function

pure subroutine doconcurrent_kernel(nx, ny, u, v, b, dx)
    implicit none(external)
    integer(ilp), intent(in) :: nx, ny
    real(dp), intent(out) :: u(nx, ny)
    real(dp), intent(in) :: v(nx, ny), b(nx, ny), dx
    integer :: i, j
    do concurrent(i=2:nx - 1, j=2:ny - 1)
        ! Jacobi update.
        u(i, j) = 0.25_dp*(b(i, j)*dx**2 - v(i + 1, j) - v(i - 1, j) &
                                         - v(i, j + 1) - v(i, j - 1))
    end do
end subroutine doconcurrent_kernel

    l2_norm = norm2(u-v)

    if (mod(iteration, 1000)) l2_norm = norm2(u - v)

if (mod(iteration, 1000)) then
    l2_norm = 0.0_dp
    do concurrent(i=2:nx-1, j=2:ny-1) reduce(+:l2_norm)
        l2_norm = l2_norm + (u(i, j) - v(i, j))**2
    enddo
    l2_norm = sqrt(l2_norm)
endif

function doconcurrent_solver(b, tol, maxiter) result(u)
    implicit none(external)
    real(dp), intent(in) :: b(:, :), tol
    integer, intent(in) :: maxiter
    real(dp), allocatable :: u(:, :)
    ! Internal variables.
    integer :: nx, ny, i, j, iteration
    real(dp), allocatable :: v(:, :)
    real(dp) :: dx, l2_norm
    ! Initialize variables
    nx = size(b, 1); ny = size(b, 2); dx = 1.0_dp/(nx - 1)
    if (nx /= ny) then
        error stop "Number of points in each direction need to be equal."
    endif
    allocate (u(nx, ny), source=0.0_dp)
    allocate (v(nx, ny), source=0.0_dp)
    l2_norm = 1.0_dp
    iteration = 0
    do while ((iteration < maxiter) .and. (l2_norm > tol))
        ! Jacobi kernel (no-copy).
        call doconcurrent_kernel(nx, ny, v, u, b, dx)
        call doconcurrent_kernel(nx, ny, u, v, b, dx)
        ! Compute error norm.
        if (mod(iteration, 1000) == 0) l2_norm = error_norm(u, v)
        ! Update iteration counter.
        iteration = iteration + 2
    end do
    l2_norm = error_norm(u, v) ! Sanity check
    print *, "Do-concurrent solver :"
    print *, "    - Number of iterations :", iteration
    print *, "    - l2-norm of the error :", l2_norm
end function

pure real(dp) function error_norm(u, v) result(l2_norm)
    implicit none
    real(dp), intent(in) :: u(:, :), v(:, :)
    integer :: i, j
    l2_norm = 0.0_dp
    do concurrent(i=2:size(u, 1)-1, j=2:size(u, 2)-1) reduce(+:l2_norm)
        l2_norm = l2_norm + (u(i, j) - v(i, j))**2
    end do
    l2_norm = sqrt(l2_norm)
end function

-O3 -mtune=native -march=native

-O3 -mtune=native -march=native -ftree-parallelize-loops=n

The Hello World of iterative solvers

You’ve started University a year ago and are taking your first class on scientific computing. Maybe you already went through the hassle of Gaussian elimination and the LU factorization. During the last class, Professor X discussed about iterative solvers for linear systems. It is now the hands-on session and today’s goal is to implement the Jacobi method. Why Jacobi? Because it is simple enough to implement in an hour or so.

The exact problem you’re given is the following:

Consider the Poisson equation with homogeneous Dirichlet boundary conditions on the unit-square. Assume the Laplace operator has been discretized using a second-order accurate central finite-difference scheme. The discretized equation reads For the sake of simplicity, take . Write a function implementing the Jacobi method to solve the resulting linear system to a user-prescribed tolerance.

We can all agree this is a simple enough yet somewhat realistic example. More importantly, it is sufficient to illustrate my point. Here is what the average student might write in Python.

import numpy as np

def jacobi(b , dx, tol, maxiter):
    # Initialize variables.
    nx, ny = b.shape
    residual = 1.0
    u = np.zeros((nx, ny))
    tmp = np.zeros((nx, ny))

    # Jacobi solver.
    for iteration in range(maxiter):
        # Jacobi iteration.
        for i in range(1, nx-1):
            for j in range(1, ny-1):
                tmp[i, j] = 0.25*(b[i, j]*dx**2 - u[i+1, j] - u[i-1, j] 
                                                - u[i, j+1] - u[i, j-1])

        # Compute residual
        residual = np.linalg.norm(u-tmp)
        # Update solution.
        u = tmp
        # If converged, exit the loop.
        if residual <= tol:
            break

    return u

Yes, you shouldn’t do for loops in Python. But remember, you are not a seasoned programmer. You’re taking your first class on scientific computing and that’s how the Jacobi method is typically presented. Be forgiving.

IndentationError: expected an indented block

TabError: inconsistent use of tabs and spaces in indentation

    for iteration in range(maxiter):
    # Jacobi iteration.
    for i in range(1, nx-1):
    for j in range(1, ny-1):
    tmp[i, j] = 0.25*(b[i, j]*dx**2 - u[i+1, j] - u[i-1, j] 
                                                - u[i, j+1] - u[i, j-1])

function jacobi(b, dx, tol, maxiter) result(u)
    implicit none
    real, dimension(:, :), intent(in) :: b
    real, intent(in) :: dx, tol
    integer, intent(in) :: maxiter
    real, dimension(:, :), allocatable :: u
    ! Internal variables.
    real, dimension(:, :), allocatable :: tmp
    integer :: nx, ny, i, j, iteration

    ! Initialize variables.
    nx = size(b, 1) ; ny = size(b, 2)
    allocate(u(nx, ny), source = 0.0)
    residual = 1.0

    ! Jacobi solver.
    do iteration = 1, maxiter
        ! Jacobi iteration.
        do j = 2, ny-1
            do i = 2, nx-1
                tmp(i, j) = 0.25*(b(i, j)*dx**2 - u(i+1, j) - u(i-1, j) &
                                                - u(i, j+1) - u(i, j-1))
            enddo
        enddo

        ! Compute residual.
        residual = norm2(u - tmp)
        ! Update solution.
        u = tmp
        ! If convered, exit the loop.
        if (residual <= tol) exit
    enddo

end function

    do j = 2, ny-1
    do i = 2, nx-1
    tmp(i, j) = 0.25*(b(i, j)*dx**2 - u(i+1, j) - u(i-1, j) &
                                    - u(i, j+1) - u(i, j-1))
    enddo
    enddo

Linear least-squares, your first step into Machine Learning

Alright, let’s look at another example. Same class, later in the semester. Professor X now discusses over-determined linear systems and how it relates to least-squares, regression and basic machine learning applications. During the hands-on session, you’re given the following problem

Consider the following unconstrained quadratic program Write a least-squares solver based on the QR factorization of the matrix . You can safely assume that is a tall matrix (i.e. ).

Here is what the typical Python code written by the students might look like.

import numpy as np

def qr(A):
    # Initialize variables.
    m, n = A.shape
    Q = np.zeros((m, n))
    R = np.zeros((n, n))

    # QR factorization based on the Gram-Schmidt orthogonalization process.
    for i in range(n):
        q = A[:, i]
        # Orthogonalization w.r.t. to the previous basis vectors.
        for j in range(i):
            R[j, i] = np.vdot(q, Q[:, j])
            q = q - R[j, i]*Q[:, j]

        # Normalize and store the new vector.
        R[i, i] = np.linalg.norm(q)
        Q[:, i] = q / R[i, i]

    return Q, R

def upper_triangular_solve(R, b):
    # Initialize variables.
    n = R.shape[0]
    x = np.zeros((n))

    # Backsubstitution.
    for i in range(n-1, -1, -1):
        x[i] = b[i]
        for j in range(n-1, i, -1):
            x[i] = x[i] - R[i, j]*x[j]
        x[i] = x[i] / R[i, i]

    return x

def lstsq(A, b):
    # QR factorization.
    Q, R = qr(A)
    # Solve R @ x = Q.T @ b.
    x = upper_triangular_solve(R, Q.T @ b)
    return x

This one was adapted from an exercise I gave last year. In reality, students lumped everything into one big function unless told otherwise, but nevermind. For comparison, here is the equivalent Fortran code.

subroutine qr(A, Q, R)
    implicit none
    real, dimension(:, :), intent(in) :: A
    real, dimension(:, :), allocatable, intent(out) :: Q, R
    ! Internal variables.
    integer :: i, j, m, n
    real, dimension(:), allocatable :: q_hat

    ! Initialize variables.
    m = size(A, 1); n = size(A, 2)
    allocate(Q(m, n), source=0.0)
    allocate(R(n, n), source=0.0)
    
    ! QR factorization based on the Gram-Schmidt orthogonalization process.
    do i = 1, n
        q_hat = A(:, i)
        ! Orthogonalize w.r.t. the previous basis vectors.
        do j = 1, i-1
            R(j, i) = dot_product(q_hat, Q(:, j))
            q_hat = q_hat - R(j, i)*Q(:, j)
        end do

        ! Normalize and store the new vector.
        R(i, i) = norm2(q_hat)
        Q(:, i) = q_hat / R(i, i)
    end do
end subroutine

function upper_triangular_solve(R, b) result(x)
    implicit none
    real, dimension(:, :), intent(in) :: R
    real, dimension(:), intent(in) :: b
    real, dimension(:), allocatable :: x
    ! Internal variables.
    integer :: n, i, j

    ! Initialize variables.
    n = size(R, 1)
    allocate(x(n), source=0.0)

    ! Backsubstitution.
    do i = n, 1, -1
        x(i) = b(i)
        do j = n-1, i, -1
            x(i) = x(i) - R(i, j)*x(j)
        enddo
        x(i) = x(i) / R(i, i)
    end do
end function

function lstsq(A, b) result(x)
    implicit none
    real, dimension(:, :), intent(in) :: A
    real, dimension(:), intent(in) :: b
    real, dimension(:), allocatable :: x
    ! Internal variables.
    real, dimension(:, :), allocatable :: Q, R

    ! QR factorization.
    call qr(A, Q, R)
    ! Solve R @ x = Q.T @ b.
    x = upper_triangular_solve(R, matmul(transpose(Q), b))
end function

Just like the Jacobi example, both implementations are equally readable. At this point in the semester, the students got somewhat more comfortable with Python. The classical indentation problems were not so much of a problem anymore. The off-by-one errors due to 0-based indexing for the Gram-Schmidt orthogonalization in qr or in the backsubstitution algorithm on the other hand… That was painful. In a 90-minutes class, it took almost a whole hour simply for them to debug these errors.

But there was another thing that confused students. A lot. And that has to do with computing dot products in numpy. There’s so many different ways: np.vdot(x, y), np.dot(x.T, y), np.dot(np.transpose(x), y), or x.transpose().dot(y) to list just the ones I have seen in their codes. Again, this has nothing to do with linear algebra, but everything with the language. Not only do they need to learn the math, but they simultaneously need to learn the not-quite-necessarily-math-standard syntax used in the language (yes, I’m looking at you @). It’s just a question of habits, sure enough, but again it can be impeding the learning process.

On the other hand, the Fortran implementation is even closer to the standard mathematical description of the algorithm: 1-based indexing, intrinsic dot_product function, etc. But beside the implicit none, there is the need to use a subroutine rather than a function construct for the QR decomposition because it has two output variables. Not a big deal again, but to be fair, it does add another minor layer of abstraction due to the language semantics rather than that of the subject being studied.

Fortran may have a slight edge, but I swept some things under the rug…

In the end, when it comes to teaching the basics of numerical linear algebra, Python and Fortran are not that different. And in that regard, neither is Julia which I really like as well. The main advantages I see of using Fortran for this task however are:

• 1-based indexing : in my experience, the 0-based indexing in Python leads to so many off-by-one erros driving the students crazy. Because linear algebra textbooks naturally use 1-based indexing, having to translate everything in your head to 0-based indices is a huge cognitive burden on top of a subject already demanding enough. You might get used to it eventually, but it’s a painful process impeding the learning outcomes.
• Strong typing : combined with implicit none, having to declare the type, dimension and input or output nature of every variable you use might seem cumbersome at first. But it forces students to pause and ponder to identify which is which. Sure this is an effort, but it is worth it. Learning is not effortless and this effort forces you to have a somewhat better understanding of a numerical algorithm before even starting to implement it. Which I think is a good thing.
• Clear delineation of the constructs : at least during the first few weeks, having to rely only on visual clues to identify where does a loop ends in Python seems to be quite complicated for a non-negligible fraction of the students I have. In that respect, the do/enddo construct in Fortran is much more explicit and probably easier to grasp.

Obvisouly, I’m not expecting educators worldwide to switch back to Fortran overnight, nor is it necessarily desirable. The advantages I see are non-negligible from my perspective but certainly not enough by themselves. There are many other things that need to be taken into account. Python is a very generalist language. You can do so much more than just numerical computing so it makes complete sense to have it in the classroom. The ecosystem is incredibly vast and the interactive nature definitely has its pros. Notebooks such as Jupyter can be incredible teaching tools (although they come with their own problems in term good coding practices). So are the Pluto notebooks in Julia.

Fortran is good at one thing: enabling computational scientists and engineers to write high-performing mathematical models without all the intricacies of equally peformant but more CS-oriented languages such as C or C++. Sure enough, the modern Fortran ecosystem is orders of magnitude smaller than Python, and targetted toward numerical computing almost exclusively. And the Julia one is fairly impressive. But the community is working on it (see the fortran-lang website or the Fortran discourse if you don’t trust me). The bad rep of Fortran is unjustified, particularly for teaching purposes. Many of its detractors have hardly been exposed to anything else than FORTRAN 77. And it’s true that, by current standards, most of FORTRAN 77 codes are terrible sphagetti codes making extensive use of implicit typing and incomprehensible goto statements. Even I, as a Fortran programmer, acknowledge it. But that’s no longer what Fortran is since the 1990’s, and certainly not today!

Abstract

Automated data-driven modeling, the process of directly discovering the governing equations of a system from data, is increasingly being used across the scientific community. PySINDy is a Python package that provides tools for applying the sparse identification of nonlinear dynamics (SINDy) approach to data-driven model discovery. In this major update to PySINDy, we implement several advanced features that enable the discovery of more general differential equations from noisy and limited data. The library of candidate terms is extended for the identification of actuated systems, partial differential equations (PDEs), and implicit differential equations. Robust formulations, including the integral form of SINDy, and ensembling techniques, are also implemented to improve performance for real-word data. Finally, we provide a range of new optimization algorithms, including several sparse regression techniques and algorithms to enforce and promote inequality constraints and stability. Together, these updates enable entirely new SINDy model discovery capabilities that have not been reported in the literature, such as constrained PDE identification and ensembling with different sparse regression optimizers.

Links

Published paper

Abstract

The sparse identification of nonlinear dynamics (SINDy) is a recently proposed data-driven modelling framework that uses sparse regression techniques to identify nonlinear low-order models. With the goal of low-order models of a fluid flow, we combine this approach with dimensionality reduction techniques (e.g. proper orthogonal decomposition) and extend it to enforce physical constraints in the regression, e.g. energy-preserving quadratic nonlinearities. The resulting models, hereafter referred to as Galerkin regression models, incorporate many beneficial aspects of Galerkin projection, but without the need for a high-fidelity solver to project the Navier-Stokes equations. Instead, the most parsimonious nonlinear model is determined that is consistent with observed measurement data and satisfies necessary constraints. Galerkin regression models also readily generalize to include higher-order nonlinear terms that model the effect of truncated modes. The effectiveness of such an approach is demonstrated on two canonical flow configurations: the two-dimensional flow past a circular cylinder and the shear-driven cavity flow. For both cases, the accuracy of the identified models compare favourably against reduced-order models obtained from a standard Galerkin projection procedure. Finally, the entire code base for ou constrained sparse Galerkin regression algorithm is freely available online.

Links

Published paper

Abstract

The linear global instability and resulting transition to turbulence induced by an isolated cylindrical roughness element of height and diameter immersed within an incompressible boundary layer flow along a flat plate is investigated using the joint application of direct numerical simulations and fully three-dimensional global stability analyses. For the range of parameters investigated, base flow computations show that the roughness element induces a wake composed of a central low-speed region surrounded by a three-dimensional shear layer and a pair of low- and high-speed streaks on each of its sides. Results from the global stability analyses highlight the unstable nature of the central low-speed region and its crucial importance in the laminar-turbulent transition process. It is able to sustain two different global instabilities: a sinuous and a varicose one. Each of these globally unstable modes is related to a different physical mechanism. While the varicose mode has its root in the instability of the whole three-dimensional shear layer surrounding the central low-speed region, the sinuous instability turns out to be similar to the von Kàrmàn instability in the two-dimensional cylinder wake and has its root in the lateral shear layers of the separated zone. The aspect ratio of the roughness element plays a key role in the selection of the dominant instability: whereas the flow over thin cylindrical roughness elements transitions due to a sinuous instability of the near-wake region, for larger roughness elements the varicose instability of the central low-speed region turns out to be the dominant one. Direct numerical simulations of the flow past an aspect ratio roughness element sustaining only the sinuous instability have revealed that the bifurcation occuring in this particular case is supercritical. Finally, comparison of the transition thresholds predicted by global linear stability analyses with von Doenhoff-Braslow transition diagram provides qualitatively good agreement.

Links

Published paper