Skip to content

Convert LaplacePetsc to use PetscMatrix#3293

Open
ZedThree wants to merge 6 commits intonextfrom
petsc-laplace-indexer
Open

Convert LaplacePetsc to use PetscMatrix#3293
ZedThree wants to merge 6 commits intonextfrom
petsc-laplace-indexer

Conversation

@ZedThree
Copy link
Member

@ZedThree ZedThree commented Mar 2, 2026

As well as making this implementation a lot cleaner, this is necessary for the Z parallelisation so that we can use the GlobalIndexer, otherwise it's really horrible to work out the domain decomposition (as the grid is Z-first and the processors are X-first, and the matrix may have different number of rows on each processor due to the X boundaries).

I had to remove the TRACE macro from the PetscMatrix as this gets called in a hot loop ((nx * nz)^2 times!) and the string formatting really adds up -- yet more reason to remove MsgStack?

In debug builds, this has a bit of a weird effect on performance:

next:

Timer name Total time (s) % of top Hits Mean time/hit (s)
petscsolve 4.36366336 1.00 6 0.727277226
petscsetup 1.50196716 0.34 6 0.250327859

this PR:

Timer name Total time (s) % of top Hits Mean time/hit (s)
petscsolve 3.76327226 1.00 6 0.627212044
petscsetup 2.25778754 0.60 6 0.376297923

The solve time goes down, but the setup time goes up.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

// TODO: handle fourth order
auto set_stencil(const Mesh& localmesh, bool fourth_order) {
OperatorStencil<IndPerp> stencil;
IndexOffset<IndPerp> zero;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: variable 'zero' of type 'IndexOffset' (aka 'IndexOffset<SpecificInd<IND_TYPE::IND_PERP>>') can be declared 'const' [misc-const-correctness]

Suggested change
IndexOffset<IndPerp> zero;
IndexOffset<IndPerp> const zero;

});
}

std::vector offsetsVec(offsets.begin(), offsets.end());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: variable 'offsetsVec' of type 'std::vector<IndexOffset<SpecificInd<IND_TYPE::IND_PERP>>, std::allocator<IndexOffset<SpecificInd<IND_TYPE::IND_PERP>>>>' (aka 'vector<IndexOffset<SpecificInd<IND_TYPE::IND_PERP>>, std::allocator<IndexOffset<SpecificInd<IND_TYPE::IND_PERP>>>>') can be declared 'const' [misc-const-correctness]

Suggested change
std::vector offsetsVec(offsets.begin(), offsets.end());
std::vector const offsetsVec(offsets.begin(), offsets.end());

}
}

stencil.add([](IndPerp UNUSED(ind)) -> bool { return true; }, {zero});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: no header providing "UNUSED" is directly included [misc-include-cleaner]

src/invert/laplace/impls/petsc/petsc_laplace.cxx:27:

+ #include "bout/unused.hxx"

fourth_order(
(*opts)["fourth_order"].doc("Use fourth order stencil").withDefault(false)),
lib(opts) {
indexer(std::make_shared<GlobalIndexer<FieldPerp>>(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: no header providing "std::make_shared" is directly included [misc-include-cleaner]

src/invert/laplace/impls/petsc/petsc_laplace.cxx:27:

+ #include <memory>

}
}
// NOTE: Only A0 is the A from setCoefA ()
const BoutReal A0 = A[i];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'A0' is confusable with 'AO' [misc-confusable-identifiers]

    const BoutReal A0 = A[i];
                   ^
Additional context

/usr/lib/petscdir/petsc3.22/x86_64-linux-gnu-real/include/petscao.h:19: other declaration found here

typedef struct _p_AO *AO;
                      ^


BoutReal dx = coords->dx[i];
BoutReal dx2 = SQ(dx);
BoutReal dz = coords->dz[i];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: variable 'dz' of type 'BoutReal' (aka 'double') can be declared 'const' [misc-const-correctness]

Suggested change
BoutReal dz = coords->dz[i];
BoutReal const dz = coords->dz[i];

BoutReal dx = coords->dx[i];
BoutReal dx2 = SQ(dx);
BoutReal dz = coords->dz[i];
BoutReal dz2 = SQ(dz);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: variable 'dz2' of type 'BoutReal' (aka 'double') can be declared 'const' [misc-const-correctness]

Suggested change
BoutReal dz2 = SQ(dz);
BoutReal const dz2 = SQ(dz);

BoutReal dx2 = SQ(dx);
BoutReal dz = coords->dz[i];
BoutReal dz2 = SQ(dz);
BoutReal dxdz = dx * dz;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: variable 'dxdz' of type 'BoutReal' (aka 'double') can be declared 'const' [misc-const-correctness]

Suggested change
BoutReal dxdz = dx * dz;
BoutReal const dxdz = dx * dz;


// Need this here to ensure PETSc isn't finalised until after the global mesh,
// otherwise we get problems from `MPI_Comm_free` on the X communicator
PetscLib lib{};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: variable 'lib' of type 'PetscLib' can be declared 'const' [misc-const-correctness]

Suggested change
PetscLib lib{};
PetscLib const lib{};


BoutInitialise(argc, argv);

PetscLib lib{};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: variable 'lib' of type 'PetscLib' can be declared 'const' [misc-const-correctness]

Suggested change
PetscLib lib{};
PetscLib const lib{};

@ZedThree
Copy link
Member Author

ZedThree commented Mar 2, 2026

Hmm, the 3D build is failing with

[0]PETSC ERROR: Unable to find requested PC type mumps

even though I explicitly check that PETSc was compiled with mumps. I guess it's fine to always use sor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant