Contact simulations on a cluster

[andrisfreimanis]

Dear All,
I am working on a contact simulations with 100k-500k elements. Ideally, would run them in 3D but could start with 2D. Friction needs to be included, and since multiple simulations are needed, I plan to run them on our computing cluster. Compiled dependencies using the provided scripts not Conda.

I am testing the 3D Berkovich example in the Contact module on a single node with 8 processes and 16 threads (128 CPUs), but it is very slow. Still ongoing but the first nonlinear iteration took about 50 minutes.

I suspect that LU preconditioner is the cause. Could that be the case? The other examples and documentation show only LU preconditioner. Are there alternative preconditioners or settings I can try to speed this up?

Best regards
Andris Freimanis

edit: this is my Moose output (still running):

*** Info ***
/scratch/project_2004485/andris/4_engine/5_contact/3d_berkovich/indenter_berkovich_friction.i:43.3:
The following information occurred in the Action 'all' of type QuasiStaticSolidMechanicsPhysics.

SolidMechanics Action: selecting 'incremental finite strain' formulation.


*** Warning, This code is deprecated and will be removed in future versions:
/scratch/project_2004485/andris/4_engine/5_contact/3d_berkovich/indenter_berkovich_friction.i:221: (Outputs/print_perf_log):
The parameter 'print_perf_log' is deprecated.
Use perf_graph instead!

Currently Setting Up
  Setting Mesh
    Finished Reading Mesh                                                                [  3.72 s] [  386 MB]
  Finished Setting Mesh                                                                  [  4.08 s] [  408 MB]
  Initializing
    Initializing Equation Systems...                                                     [ 24.95 s] [ 1376 MB]
  Finished Initializing                                                                  [ 28.30 s] [ 1376 MB]
Finished Setting Up                                                                      [ 35.38 s] [ 1376 MB]
Framework Information:
MOOSE Version:           git commit 2710d28408 on 2025-01-05
LibMesh Version:         6ef7d4395794104f48dae1fd48e64077207188e8
PETSc Version:           3.22.1
SLEPc Version:           3.22.1
Current Time:            Tue Jan 14 09:59:10 2025
Executable Timestamp:    Wed Jan  8 16:23:04 2025

Input File(s):
  /scratch/project_2004485/andris/4_engine/5_contact/3d_berkovich/indenter_berkovich_friction.i

Command Line Argument(s):
  --color=off
  --n_threads=16

Checkpoint:
  Wall Time Interval:      Every 3600 s
  User Checkpoint:         Outputs/my_checkpoint
  # Checkpoints Kept:      2
  Execute On:              TIMESTEP_END 

Parallelism:
  Num Processors:          8
  Num Threads:             16

Mesh: 
  Parallel Type:           replicated
  Mesh Dimension:          3
  Spatial Dimension:       3
  Nodes:                   
    Total:                 272988
    Local:                 35640
    Min/Max/Avg:           31781/35661/34123
  Elems:                   
    Total:                 270493
    Local:                 33464
    Min/Max/Avg:           33463/36248/33811
  Num Subdomains:          2
  Num Partitions:          8
  Partitioner:             metis

Nonlinear System:
  Num DOFs:                818964
  Num Local DOFs:          106920
  Variables:               { "disp_x" "disp_y" "disp_z" } 
  Finite Element Types:    "LAGRANGE" 
  Approximation Orders:    "FIRST" 

Auxiliary System:
  Num DOFs:                2990393
  Num Local DOFs:          381160
  Variables:               { "saved_x" "saved_y" "saved_z" } { "stress_xx" "stress_xy" "stress_xz" "stress_yy" 
                             "stress_zz" } { "penetration" "contact_pressure" "nodal_area" } 
  Finite Element Types:    "LAGRANGE" "MONOMIAL" "LAGRANGE" 
  Approximation Orders:    "FIRST" "CONSTANT" "FIRST" 

Execution Information:
  Executioner:             Transient
  TimeStepper:             ConstantDT
  TimeIntegrator(s):       ImplicitEuler
  Solver Mode:             Preconditioned JFNK
  PETSc Preconditioner:    lu mat_superlu_dist_fact: SamePattern  mat_superlu_dist_replacetinypivot: true 
  MOOSE Preconditioner:    SMP

Currently Executing
  Performing Initial Setup
    Setting Up Materials
      Finished Computing Initial Material Values                                         [  0.33 s] [ 1861 MB]
    Finished Setting Up Materials                                                        [  0.33 s] [ 1861 MB]
    Reinitializing Because of Geometric Search Objects....                               [ 33.72 s] [ 2266 MB]
  Finished Performing Initial Setup                                                      [ 40.80 s] [ 2267 MB]

Time Step 0, time = 0

Postprocessor Values:
+----------------+----------------+----------------+----------------+
| time           | disp_z         | resid_z        | stress_zz      |
+----------------+----------------+----------------+----------------+
|   0.000000e+00 |   0.000000e+00 |   0.000000e+00 |   0.000000e+00 |
+----------------+----------------+----------------+----------------+


Time Step 1, time = 0.004, dt = 0.004

Performing automatic scaling calculation



*** Warning ***
Warning in PenetrationLocator. Penetration is not detected for one or more secondary nodes. This could be because those secondary nodes simply do not project to faces on the primary surface. However, this could also be because contact should be enforced on those nodes, but the faces that they project to are outside the contact patch, which will give an erroneous result. Use appropriate options for 'patch_size' and 'patch_update_strategy' in the Mesh block to avoid this issue. Setting 'patch_update_strategy=iteration' is recommended because it completely avoids this potential issue. Also note that this warning is printed only once, so a similar situation could occur multiple times during the simulation but this warning is printed only at the first occurrence.
 0 Nonlinear |R| = 2.213765e-02

    Computing Jacobian.......                                                            [ 41.27 s] [ 2457 MB]
      0 Linear |R| = 2.213765e-02
      1 Linear |R| = 4.076035e-08
   ContactSlipDamper: Damping     #Cont    #Stick     #Slip #SlipFric #StickLock  #SlipRev
                            1         0         0         0         0          0         0

 1 Nonlinear |R| = 2.422387e-05

    Computing Jacobian                                                                   [  6.36 s] [ 7484 MB]
      0 Linear |R| = 2.422387e-05
      1 Linear |R| = 4.568102e-07
      2 Linear |R| = 1.235565e-07
      3 Linear |R| = 8.821792e-08
      4 Linear |R| = 8.162893e-08
      5 Linear |R| = 7.493706e-08
      6 Linear |R| = 7.027811e-08
      7 Linear |R| = 6.644860e-08
      8 Linear |R| = 6.337994e-08
      9 Linear |R| = 6.179738e-08
     10 Linear |R| = 5.881004e-08
     11 Linear |R| = 5.818547e-08
     12 Linear |R| = 5.660035e-08
     13 Linear |R| = 5.524093e-08
     14 Linear |R| = 5.458537e-08
     15 Linear |R| = 5.343820e-08
     16 Linear |R| = 5.217066e-08
     17 Linear |R| = 5.122498e-08
     18 Linear |R| = 5.081069e-08
     19 Linear |R| = 4.950276e-08
     20 Linear |R| = 4.922093e-08
     21 Linear |R| = 4.776082e-08
     22 Linear |R| = 4.712901e-08
     23 Linear |R| = 4.573236e-08
     24 Linear |R| = 4.495617e-08
     25 Linear |R| = 4.399596e-08
     26 Linear |R| = 4.311643e-08
     27 Linear |R| = 4.109232e-08
     28 Linear |R| = 4.051019e-08
     29 Linear |R| = 3.968345e-08
     30 Linear |R| = 1.032994e-07
     31 Linear |R| = 5.342622e-08
     32 Linear |R| = 4.113642e-08
     33 Linear |R| = 3.906332e-08
     34 Linear |R| = 3.716212e-08
     35 Linear |R| = 3.682659e-08
     36 Linear |R| = 3.380259e-08
     37 Linear |R| = 3.371072e-08
     38 Linear |R| = 3.131150e-08
     39 Linear |R| = 3.123866e-08
     40 Linear |R| = 3.122988e-08
     41 Linear |R| = 3.038183e-08
     42 Linear |R| = 3.037545e-08
     43 Linear |R| = 3.030275e-08
     44 Linear |R| = 2.987998e-08
     45 Linear |R| = 2.971970e-08
     46 Linear |R| = 2.969715e-08
     47 Linear |R| = 2.956390e-08
     48 Linear |R| = 2.922606e-08
     49 Linear |R| = 2.853192e-08
     50 Linear |R| = 2.831385e-08
     51 Linear |R| = 2.791087e-08
     52 Linear |R| = 2.788347e-08
     53 Linear |R| = 2.779429e-08
     54 Linear |R| = 2.779338e-08
     55 Linear |R| = 2.776008e-08
     56 Linear |R| = 2.775804e-08
     57 Linear |R| = 2.772530e-08
     58 Linear |R| = 2.772509e-08
     59 Linear |R| = 2.772431e-08
     60 Linear |R| = 2.771192e-08
    Linear solve did not converge due to DIVERGED_ITS iterations 60
   ContactSlipDamper: Damping     #Cont    #Stick     #Slip #SlipFric #StickLock  #SlipRev
                            1         1         0         0         1          0         0

 2 Nonlinear |R| = 1.461026e-05

    Computing Jacobian                                                                   [  7.04 s] [ 7646 MB]
      0 Linear |R| = 1.461026e-05
      1 Linear |R| = 2.453459e-06
      2 Linear |R| = 4.612174e-07
      3 Linear |R| = 9.562367e-08
      4 Linear |R| = 1.980773e-08
      5 Linear |R| = 4.050844e-09
   ContactSlipDamper: Damping     #Cont    #Stick     #Slip #SlipFric #StickLock  #SlipRev
                            1         1         0         0         1          0         0

 3 Nonlinear |R| = 1.307640e-07
 Solve Converged!
  Finished Solving                                                                       [8311.69 s] [ 5610 MB]

Postprocessor Values:
+----------------+----------------+----------------+----------------+
| time           | disp_z         | resid_z        | stress_zz      |
+----------------+----------------+----------------+----------------+
|   0.000000e+00 |   0.000000e+00 |   0.000000e+00 |   0.000000e+00 |
|   4.000000e-03 |  -4.000000e-03 |  -1.861519e+01 |  -3.647783e-03 |
+----------------+----------------+----------------+----------------+

  Currently Outputting
    Finished Outputting Step                                                             [  6.95 s] [ 5615 MB]
  Finished Outputting                                                                    [ 10.30 s] [ 5615 MB]

Time Step 2, time = 0.008, dt = 0.004
 0 Nonlinear |R| = 2.213765e-02

    Computing Jacobian                                                                   [  6.47 s] [ 5616 MB]

[GiudGiud]

Hello

LU is good for medium sized problems when the Jacobian is exact. This can be challenging for contact.

You ll want to avoid running very many linear iterations like it did when this happened

    Linear solve did not converge due to DIVERGED_ITS iterations 60

One way would be to set a loose linear solve tolerance. LU tends to solve very tightly so it should not be an issue. The preferred way would be to set a max number of linear iteraitons to trigger a fail faster

[andrisfreimanis]

l_tol is already set to 1e-3, but guess could try it with l_tol = 1e-2 or even 1e-1. Also, the next nl iterations are shorter. 2-5 linear iteration per nonlinear, so I'll try with l_max_its = 5.

I worry that the time required to compute LU is the cause. Are there other preconditioners that could be used with contact, or is it only LU?

Thanks
Andris

[andrisfreimanis]

Tried with l_tol=1e-2 and l_max_its=5. It cut down about 50 minutes from the first time step.

Is there anything else to try to speed this up?

Time Step 1, time = 0.004, dt = 0.004

Performing automatic scaling calculation



*** Warning ***
Warning in PenetrationLocator. Penetration is not detected for one or more secondary nodes. This could be because those secondary nodes simply do not project to faces on the primary surface. However, this could also be because contact should be enforced on those nodes, but the faces that they project to are outside the contact patch, which will give an erroneous result. Use appropriate options for 'patch_size' and 'patch_update_strategy' in the Mesh block to avoid this issue. Setting 'patch_update_strategy=iteration' is recommended because it completely avoids this potential issue. Also note that this warning is printed only once, so a similar situation could occur multiple times during the simulation but this warning is printed only at the first occurrence.
 0 Nonlinear |R| = 2.213765e-02

    Computing Jacobian.......                                                            [ 41.42 s] [ 2661 MB]
      0 Linear |R| = 2.213765e-02
      1 Linear |R| = 4.076027e-08
   ContactSlipDamper: Damping     #Cont    #Stick     #Slip #SlipFric #StickLock  #SlipRev
                            1         0         0         0         0          0         0

 1 Nonlinear |R| = 2.422387e-05

    Computing Jacobian                                                                   [  6.42 s] [ 7445 MB]
      0 Linear |R| = 2.422387e-05
      1 Linear |R| = 4.568116e-07
      2 Linear |R| = 1.235567e-07
   ContactSlipDamper: Damping     #Cont    #Stick     #Slip #SlipFric #StickLock  #SlipRev
                            1         1         0         0         1          0         0

 2 Nonlinear |R| = 8.223119e-07
 Solve Converged!
  Finished Solving                                                                       [5415.95 s] [ 5642 MB]

[GiudGiud]

you could use more procs

  Num Local DOFs:          106920

maybe up to 5x more might be beneficial

you could try the line searches. Maybe the contact one helps here?

You could profile and see what steps are actually slowing this down and we can take a look at speeding them up
https://mooseframework.inl.gov/application_development/profiling.html

[andrisfreimanis]

Great! I can throw more cores at this and will try profiling.

Thanks
Andris