git-svn-id: svn://svn.icms.temple.edu/lammps-ro/trunk@16053 f3b2605a-c512-4ea7-a41b...

[lammps.git] / doc / src / accelerate_intel.txt

"Previous Section"_Section_packages.html - "LAMMPS WWW Site"_lws -
"LAMMPS Documentation"_ld - "LAMMPS Commands"_lc :c

:link(lws,http://lammps.sandia.gov)
:link(ld,Manual.html)
:link(lc,Section_commands.html#comm)

:line

"Return to Section accelerate overview"_Section_accelerate.html

5.3.2 USER-INTEL package :h5

The USER-INTEL package is maintained by Mike Brown at Intel
Corporation.  It provides two methods for accelerating simulations,
depending on the hardware you have.  The first is acceleration on
Intel CPUs by running in single, mixed, or double precision with
vectorization.  The second is acceleration on Intel Xeon Phi
coprocessors via offloading neighbor list and non-bonded force
calculations to the Phi.  The same C++ code is used in both cases.
When offloading to a coprocessor from a CPU, the same routine is run
twice, once on the CPU and once with an offload flag. This allows
LAMMPS to run on the CPU cores and coprocessor cores simulataneously.

[Currently Available USER-INTEL Styles:]

Angle Styles: charmm, harmonic :ulb,l
Bond Styles: fene, harmonic :l
Dihedral Styles: charmm, harmonic, opls :l
Fixes: nve, npt, nvt, nvt/sllod :l
Improper Styles: cvff, harmonic :l
Pair Styles: buck/coul/cut, buck/coul/long, buck, eam, gayberne,
charmm/coul/long, lj/cut, lj/cut/coul/long, sw, tersoff :l
K-Space Styles: pppm :l
:ule

[Speed-ups to expect:]

The speedups will depend on your simulation, the hardware, which
styles are used, the number of atoms, and the floating-point
precision mode. Performance improvements are shown compared to
LAMMPS {without using other acceleration packages} as these are
under active development (and subject to performance changes). The
measurements were performed using the input files available in
the src/USER-INTEL/TEST directory. These are scalable in size; the
results given are with 512K particles (524K for Liquid Crystal).
Most of the simulations are standard LAMMPS benchmarks (indicated
by the filename extension in parenthesis) with modifications to the
run length and to add a warmup run (for use with offload
benchmarks).

:c,image(JPG/user_intel.png)

Results are speedups obtained on Intel Xeon E5-2697v4 processors
(code-named Broadwell) and Intel Xeon Phi 7250 processors
(code-named Knights Landing) with "18 Jun 2016" LAMMPS built with
Intel Parallel Studio 2016 update 3. Results are with 1 MPI task
per physical core. See {src/USER-INTEL/TEST/README} for the raw
simulation rates and instructions to reproduce.

:line

[Quick Start for Experienced Users:]

LAMMPS should be built with the USER-INTEL package installed.
Simulations should be run with 1 MPI task per physical {core},
not {hardware thread}.

For Intel Xeon CPUs:

Edit src/MAKE/OPTIONS/Makefile.intel_cpu_intelmpi as necessary. :ulb,l
If using {kspace_style pppm} in the input script, add "neigh_modify binsize 3" and "kspace_modify diff ad" to the input script for better
performance. :l
"-pk intel 0 omp 2 -sf intel" added to LAMMPS command-line :l
:ule

For Intel Xeon Phi CPUs for simulations without {kspace_style
pppm} in the input script :

Edit src/MAKE/OPTIONS/Makefile.knl as necessary. :ulb,l
Runs should be performed using MCDRAM. :l
"-pk intel 0 omp 2 -sf intel" {or} "-pk intel 0 omp 4 -sf intel"
should be added to the LAMMPS command-line. Choice for best
performance will depend on the simulation. :l
:ule

For Intel Xeon Phi CPUs for simulations with {kspace_style
pppm} in the input script:

Edit src/MAKE/OPTIONS/Makefile.knl as necessary. :ulb,l
Runs should be performed using MCDRAM. :l
Add "neigh_modify binsize 3" to the input script for better
performance. :l
Add "kspace_modify diff ad" to the input script for better
performance. :l
export KMP_AFFINITY=none :l
"-pk intel 0 omp 3 lrt yes -sf intel" or "-pk intel 0 omp 1 lrt yes
-sf intel" added to LAMMPS command-line. Choice for best performance
will depend on the simulation. :l
:ule

For Intel Xeon Phi coprocessors (Offload):

Edit src/MAKE/OPTIONS/Makefile.intel_coprocessor as necessary :ulb,l
"-pk intel N omp 1" added to command-line where N is the number of
coprocessors per node. :l
:ule

:line

[Required hardware/software:]

In order to use offload to coprocessors, an Intel Xeon Phi
coprocessor and an Intel compiler are required. For this, the
recommended version of the Intel compiler is 14.0.1.106 or
versions 15.0.2.044 and higher.

Although any compiler can be used with the USER-INTEL pacakge,
currently, vectorization directives are disabled by default when
not using Intel compilers due to lack of standard support and
observations of decreased performance. The OpenMP standard now
supports directives for vectorization and we plan to transition the
code to this standard once it is available in most compilers. We
expect this to allow improved performance and support with other
compilers.

For Intel Xeon Phi x200 series processors (code-named Knights
Landing), there are multiple configuration options for the hardware.
For best performance, we recommend that the MCDRAM is configured in
"Flat" mode and with the cluster mode set to "Quadrant" or "SNC4".
"Cache" mode can also be used, although the performance might be
slightly lower.

[Notes about Simultaneous Multithreading:]

Modern CPUs often support Simultaneous Multithreading (SMT). On
Intel processors, this is called Hyper-Threading (HT) technology.
SMT is hardware support for running multiple threads efficiently on
a single core. {Hardware threads} or {logical cores} are often used
to refer to the number of threads that are supported in hardware.
For example, the Intel Xeon E5-2697v4 processor is described
as having 36 cores and 72 threads. This means that 36 MPI processes
or OpenMP threads can run simultaneously on separate cores, but that
up to 72 MPI processes or OpenMP threads can be running on the CPU
without costly operating system context switches.

Molecular dynamics simulations will often run faster when making use
of SMT. If a thread becomes stalled, for example because it is
waiting on data that has not yet arrived from memory, another thread
can start running so that the CPU pipeline is still being used
efficiently. Although benefits can be seen by launching a MPI task
for every hardware thread, for multinode simulations, we recommend
that OpenMP threads are used for SMT instead, either with the
USER-INTEL package, "USER-OMP package"_accelerate_omp.html, or
"KOKKOS package"_accelerate_kokkos.html. In the example above, up
to 36X speedups can be observed by using all 36 physical cores with
LAMMPS. By using all 72 hardware threads, an additional 10-30%
performance gain can be achieved.

The BIOS on many platforms allows SMT to be disabled, however, we do
not recommend this on modern processors as there is little to no
benefit for any software package in most cases. The operating system
will report every hardware thread as a separate core allowing one to
determine the number of hardware threads available. On Linux systems,
this information can normally be obtained with:

cat /proc/cpuinfo :pre

[Building LAMMPS with the USER-INTEL package:]

The USER-INTEL package must be installed into the source directory:

make yes-user-intel :pre

Several example Makefiles for building with the Intel compiler are
included with LAMMPS in the src/MAKE/OPTIONS/ directory:

Makefile.intel_cpu_intelmpi # Intel Compiler, Intel MPI, No Offload
Makefile.knl                # Intel Compiler, Intel MPI, No Offload
Makefile.intel_cpu_mpich    # Intel Compiler, MPICH, No Offload
Makefile.intel_cpu_openpmi  # Intel Compiler, OpenMPI, No Offload
Makefile.intel_coprocessor  # Intel Compiler, Intel MPI, Offload :pre

Makefile.knl is identical to Makefile.intel_cpu_intelmpi except that
it explicitly specifies that vectorization should be for Intel
Xeon Phi x200 processors making it easier to cross-compile. For
users with recent installations of Intel Parallel Studio, the
process can be as simple as:

make yes-user-intel
source /opt/intel/parallel_studio_xe_2016.3.067/psxevars.sh
# or psxevars.csh for C-shell
make intel_cpu_intelmpi :pre

Alternatively, the build can be accomplished with the src/Make.py
script, described in "Section 2.4"_Section_start.html#start_4 of the
manual. Type "Make.py -h" for help. For an example:

Make.py -v -p intel omp -intel cpu -a file intel_cpu_intelmpi :pre

Note that if you build with support for a Phi coprocessor, the same
binary can be used on nodes with or without coprocessors installed.
However, if you do not have coprocessors on your system, building
without offload support will produce a smaller binary.

The general requirements for Makefiles with the USER-INTEL package
are as follows. "-DLAMMPS_MEMALIGN=64" is required for CCFLAGS. When
using Intel compilers, "-restrict" is required and "-qopenmp" is
highly recommended for CCFLAGS and LINKFLAGS. LIB should include
"-ltbbmalloc". For builds supporting offload, "-DLMP_INTEL_OFFLOAD"
is required for CCFLAGS and "-qoffload" is required for LINKFLAGS.
Other recommended CCFLAG options for best performance are
"-O2 -fno-alias -ansi-alias -qoverride-limits fp-model fast=2
-no-prec-div". The Make.py command will add all of these
automatically.

NOTE: The vectorization and math capabilities can differ depending on
the CPU. For Intel compilers, the "-x" flag specifies the type of
processor for which to optimize. "-xHost" specifies that the compiler
should build for the processor used for compiling. For Intel Xeon Phi
x200 series processors, this option is "-xMIC-AVX512". For fourth
generation Intel Xeon (v4/Broadwell) processors, "-xCORE-AVX2" should
be used. For older Intel Xeon processors, "-xAVX" will perform best
in general for the different simulations in LAMMPS. The default
in most of the example Makefiles is to use "-xHost", however this
should not be used when cross-compiling.

[Running LAMMPS with the USER-INTEL package:]

Running LAMMPS with the USER-INTEL package is similar to normal use
with the exceptions that one should 1) specify that LAMMPS should use
the USER-INTEL package, 2) specify the number of OpenMP threads, and
3) optionally specify the specific LAMMPS styles that should use the
USER-INTEL package. 1) and 2) can be performed from the command-line
or by editing the input script. 3) requires editing the input script.
Advanced performance tuning options are also described below to get
the best performance.

When running on a single node (including runs using offload to a
coprocessor), best performance is normally obtained by using 1 MPI
task per physical core and additional OpenMP threads with SMT. For
Intel Xeon processors, 2 OpenMP threads should be used for SMT.
For Intel Xeon Phi CPUs, 2 or 4 OpenMP threads should be used
(best choice depends on the simulation). In cases where the user
specifies that LRT mode is used (described below), 1 or 3 OpenMP
threads should be used. For multi-node runs, using 1 MPI task per
physical core will often perform best, however, depending on the
machine and scale, users might get better performance by decreasing
the number of MPI tasks and using more OpenMP threads. For
performance, the product of the number of MPI tasks and OpenMP
threads should not exceed the number of available hardware threads in
almost all cases.

NOTE: Setting core affinity is often used to pin MPI tasks and OpenMP
threads to a core or group of cores so that memory access can be
uniform. Unless disabled at build time, affinity for MPI tasks and
OpenMP threads on the host (CPU) will be set by default on the host
{when using offload to a coprocessor}. In this case, it is unnecessary
to use other methods to control affinity (e.g. taskset, numactl,
I_MPI_PIN_DOMAIN, etc.). This can be disabled with the {no_affinity}
option to the "package intel"_package.html command or by disabling the
option at build time (by adding -DINTEL_OFFLOAD_NOAFFINITY to the
CCFLAGS line of your Makefile). Disabling this option is not
recommended, especially when running on a machine with Intel
Hyper-Threading technology disabled.

[Run with the USER-INTEL package from the command line:]

To enable USER-INTEL optimizations for all available styles used in
the input script, the "-sf intel"
"command-line switch"_Section_start.html#start_7 can be used without
any requirement for editing the input script. This switch will
automatically append "intel" to styles that support it. It also
invokes a default command: "package intel 1"_package.html. This
package command is used to set options for the USER-INTEL package.
The default package command will specify that USER-INTEL calculations
are performed in mixed precision, that the number of OpenMP threads
is specified by the OMP_NUM_THREADS environment variable, and that
if coprocessors are present and the binary was built with offload
support, that 1 coprocessor per node will be used with automatic
balancing of work between the CPU and the coprocessor.

You can specify different options for the USER-INTEL package by using
the "-pk intel Nphi" "command-line switch"_Section_start.html#start_7
with keyword/value pairs as specified in the documentation. Here,
Nphi = # of Xeon Phi coprocessors/node (ignored without offload
support). Common options to the USER-INTEL package include {omp} to
override any OMP_NUM_THREADS setting and specify the number of OpenMP
threads, {mode} to set the floating-point precision mode, and
{lrt} to enable Long-Range Thread mode as described below. See the
"package intel"_package.html command for details, including the
default values used for all its options if not specified, and how to
set the number of OpenMP threads via the OMP_NUM_THREADS environment
variable if desired.

Examples (see documentation for your MPI/Machine for differences in
launching MPI applications):

mpirun -np 72 -ppn 36 lmp_machine -sf intel -in in.script                                 # 2 nodes, 36 MPI tasks/node, $OMP_NUM_THREADS OpenMP Threads
mpirun -np 72 -ppn 36 lmp_machine -sf intel -in in.script -pk intel 0 omp 2 mode double   # Don't use any coprocessors that might be available, use 2 OpenMP threads for each task, use double precision :pre

[Or run with the USER-INTEL package by editing an input script:]

As an alternative to adding command-line arguments, the input script
can be edited to enable the USER-INTEL package. This requires adding
the "package intel"_package.html command to the top of the input
script. For the second example above, this would be:

package intel 0 omp 2 mode double :pre

To enable the USER-INTEL package only for individual styles, you can
add an "intel" suffix to the individual style, e.g.:

pair_style lj/cut/intel 2.5 :pre

Alternatively, the "suffix intel"_suffix.html command can be added to
the input script to enable USER-INTEL styles for the commands that
follow in the input script.

[Tuning for Performance:]

NOTE: The USER-INTEL package will perform better with modifications
to the input script when "PPPM"_kspace_style.html is used:
"kspace_modify diff ad"_kspace_modify.html and "neigh_modify binsize
3"_neigh_modify.html should be added to the input script.

Long-Range Thread (LRT) mode is an option to the "package
intel"_package.html command that can improve performance when using
"PPPM"_kspace_style.html for long-range electrostatics on processors
with SMT. It generates an extra pthread for each MPI task. The thread
is dedicated to performing some of the PPPM calculations and MPI
communications. On Intel Xeon Phi x200 series CPUs, this will likely
always improve performance, even on a single node. On Intel Xeon
processors, using this mode might result in better performance when
using multiple nodes, depending on the machine. To use this mode,
specify that the number of OpenMP threads is one less than would
normally be used for the run and add the "lrt yes" option to the "-pk"
command-line suffix or "package intel" command. For example, if a run
would normally perform best with "-pk intel 0 omp 4", instead use
"-pk intel 0 omp 3 lrt yes". When using LRT, you should set the
environment variable "KMP_AFFINITY=none". LRT mode is not supported
when using offload.

Not all styles are supported in the USER-INTEL package. You can mix
the USER-INTEL package with styles from the "OPT"_accelerate_opt.html
package or the "USER-OMP package"_accelerate_omp.html. Of course,
this requires that these packages were installed at build time. This
can performed automatically by using "-sf hybrid intel opt" or
"-sf hybrid intel omp" command-line options. Alternatively, the "opt"
and "omp" suffixes can be appended manually in the input script. For
the latter, the "package omp"_package.html command must be in the
input script or the "-pk omp Nt" "command-line
switch"_Section_start.html#start_7 must be used where Nt is the
number of OpenMP threads. The number of OpenMP threads should not be
set differently for the different packages. Note that the "suffix
hybrid intel omp"_suffix.html command can also be used within the
input script to automatically append the "omp" suffix to styles when
USER-INTEL styles are not available.

When running on many nodes, performance might be better when using
fewer OpenMP threads and more MPI tasks. This will depend on the
simulation and the machine. Using the "verlet/split"_run_style.html
run style might also give better performance for simulations with
"PPPM"_kspace_style.html electrostatics. Note that this is an
alternative to LRT mode and the two cannot be used together.

Currently, when using Intel MPI with Intel Xeon Phi x200 series
CPUs, better performance might be obtained by setting the
environment variable "I_MPI_SHM_LMT=shm" for Linux kernels that do
not yet have full support for AVX-512. Runs on Intel Xeon Phi x200
series processors will always perform better using MCDRAM. Please
consult your system documentation for the best approach to specify
that MPI runs are performed in MCDRAM.

[Tuning for Offload Performance:]

The default settings for offload should give good performance.

When using LAMMPS with offload to Intel coprocessors, best performance
will typically be achieved with concurrent calculations performed on
both the CPU and the coprocessor. This is achieved by offloading only
a fraction of the neighbor and pair computations to the coprocessor or
using "hybrid"_pair_hybrid.html pair styles where only one style uses
the "intel" suffix. For simulations with long-range electrostatics or
bond, angle, dihedral, improper calculations, computation and data
transfer to the coprocessor will run concurrently with computations
and MPI communications for these calculations on the host CPU. This
is illustrated in the figure below for the rhodopsin protein benchmark
running on E5-2697v2 processors with a Intel Xeon Phi 7120p
coprocessor. In this plot, the vertical access is time and routines
running at the same time are running concurrently on both the host and
the coprocessor.

:c,image(JPG/offload_knc.png)

The fraction of the offloaded work is controlled by the {balance}
keyword in the "package intel"_package.html command. A balance of 0
runs all calculations on the CPU.  A balance of 1 runs all
supported calculations on the coprocessor.  A balance of 0.5 runs half
of the calculations on the coprocessor.  Setting the balance to -1
(the default) will enable dynamic load balancing that continously
adjusts the fraction of offloaded work throughout the simulation.
Because data transfer cannot be timed, this option typically produces
results within 5 to 10 percent of the optimal fixed balance.

If running short benchmark runs with dynamic load balancing, adding a
short warm-up run (10-20 steps) will allow the load-balancer to find a
near-optimal setting that will carry over to additional runs.

The default for the "package intel"_package.html command is to have
all the MPI tasks on a given compute node use a single Xeon Phi
coprocessor.  In general, running with a large number of MPI tasks on
each node will perform best with offload.  Each MPI task will
automatically get affinity to a subset of the hardware threads
available on the coprocessor.  For example, if your card has 61 cores,
with 60 cores available for offload and 4 hardware threads per core
(240 total threads), running with 24 MPI tasks per node will cause
each MPI task to use a subset of 10 threads on the coprocessor.  Fine
tuning of the number of threads to use per MPI task or the number of
threads to use per core can be accomplished with keyword settings of
the "package intel"_package.html command.

The USER-INTEL package has two modes for deciding which atoms will be
handled by the coprocessor.  This choice is controlled with the {ghost}
keyword of the "package intel"_package.html command.  When set to 0,
ghost atoms (atoms at the borders between MPI tasks) are not offloaded
to the card.  This allows for overlap of MPI communication of forces
with computation on the coprocessor when the "newton"_newton.html
setting is "on".  The default is dependent on the style being used,
however, better performance may be achieved by setting this option
explictly.

When using offload with CPU Hyper-Threading disabled, it may help
performance to use fewer MPI tasks and OpenMP threads than available
cores.  This is due to the fact that additional threads are generated
internally to handle the asynchronous offload tasks.

If pair computations are being offloaded to an Intel Xeon Phi
coprocessor, a diagnostic line is printed to the screen (not to the
log file), during the setup phase of a run, indicating that offload
mode is being used and indicating the number of coprocessor threads
per MPI task.  Additionally, an offload timing summary is printed at
the end of each run.  When offloading, the frequency for "atom
sorting"_atom_modify.html is changed to 1 so that the per-atom data is
effectively sorted at every rebuild of the neighbor lists. All the
available coprocessor threads on each Phi will be divided among MPI
tasks, unless the {tptask} option of the "-pk intel" "command-line
switch"_Section_start.html#start_7 is used to limit the coprocessor
threads per MPI task.

[Restrictions:]

When offloading to a coprocessor, "hybrid"_pair_hybrid.html styles
that require skip lists for neighbor builds cannot be offloaded.
Using "hybrid/overlay"_pair_hybrid.html is allowed.  Only one intel
accelerated style may be used with hybrid styles.
"Special_bonds"_special_bonds.html exclusion lists are not currently
supported with offload, however, the same effect can often be
accomplished by setting cutoffs for excluded atom types to 0.  None of
the pair styles in the USER-INTEL package currently support the
"inner", "middle", "outer" options for rRESPA integration via the
"run_style respa"_run_style.html command; only the "pair" option is
supported.

[References:]

Brown, W.M., Carrillo, J.-M.Y., Mishra, B., Gavhane, N., Thakker, F.M., De Kraker, A.R., Yamada, M., Ang, J.A., Plimpton, S.J., “Optimizing Classical Molecular Dynamics in LAMMPS,” in Intel Xeon Phi Processor High Performance Programming: Knights Landing Edition, J. Jeffers, J. Reinders, A. Sodani, Eds. Morgan Kaufmann. :ulb,l

Brown, W. M., Semin, A., Hebenstreit, M., Khvostov, S., Raman, K., Plimpton, S.J. Increasing Molecular Dynamics Simulation Rates with an 8-Fold Increase in Electrical Power Efficiency. 2016 International Conference for High Performance Computing. In press. :l

Brown, W.M., Carrillo, J.-M.Y., Gavhane, N., Thakkar, F.M., Plimpton, S.J. Optimizing Legacy Molecular Dynamics Software with Directive-Based Offload. Computer Physics Communications. 2015. 195: p. 95-101. :l
:ule