Vectorization by jnvance · Pull Request #294 · espressopp/espressopp

jnvance · 2020-01-08T16:03:34Z

This PR adds a preliminary version of the vectorization submodule which speeds up the calculation of short-range pair interactions targeting Intel CPUs that support AVX2 and AVX-512.

When the espressopp.vectorization.Vectorization class is instantiated, it connects some of its methods to the integrator and storage objects (currently, only VelocityVerlet and DomainDecomposition are supported).

During every time step of integrator.run(), the Vectorization class copies the necessary data from the Particle lists in the cells (currently only position) into ParticleArray, performs the force calculation, and then copies back the forces to the Particle lists. The data in ParticleArray are stored in a structure of arrays (SOA) layout. This enables the vectorization of the neighbor-list construction and force calculation using only #pragma vector directives (no intrinsics), similar to how it is done in the LAMMPS-INTEL package.

Most of the additional source code is contained in the directory src/vectorization, except for two new Boost signals:

onCellAdjust - called when the cell structure of the storage class is modified
aftCalcFLocal - called from integrator.run after calcForces and just before exchanging ghost forces with neighboring subdomains

Vectorized versions of the VerletList and LennardJones classes are implemented in the vectorization submodule. So to use these on current scripts, one only needs to add the vectorization keyword and pass the correct arguments, e.g.

vec         = espressopp.vectorization.Vectorization(system, integrator)
verletlist  = espressopp.vectorization.VerletList(system, vec, r_cutoff)
potential   = espressopp.vectorization.interaction.LennardJones(...)
interaction = espressopp.vectorization.interaction.VerletListLennardJones(verletlist)

In this version, the energy and virial calculations still rely on the previous VerletList layout, so the Verlet list rebuild has to be manually triggered before calling any analysis functions, e.g.

verletlist.rebuildPairs()
espressopp.tools.analyse.info(system, integrator)

An example script is provided in examples/vectorization/lennard_jones.py which shows the minimal modifications needed for the examples/lennard_jones/lennard_jones.py script to use vectorized versions of the routines (currently only for the equilibration phase since LennardJonesCapped is not yet implemented).

A test was also added in testsuite/vectorization to verify that particle trajectories resulting from the MD integration are almost equal whether the vectorized or the non-vectorized version is used.

Development of the submodule is optimized for the Intel 2018 compiler with -O3 -xHost -restrict flags for Broadwell, and -O3 -xSKYLAKE-AVX512 -qopt-zmm-usage=high for Skylake. Improvements can also be observed with GCC as long as the appropriate optimization flags are used.

…orized analysis

junghans · 2020-01-08T16:48:47Z

How does this approach compares to https://github.com/ECP-copa/Cabana?

jnvance · 2020-01-09T10:48:56Z

How does this approach compares to https://github.com/ECP-copa/Cabana?

Cabana seems to provide a more generalized way of representing Struct-of-Arrays and Array-of-Struct-of-Arrays (where attributes are template arguments) similar to how the data are stored in ParticleArray (where the attributes are explicitly listed). Our approach, on the other hand, still uses the original std::vector<Particle> (an extended AOSOA) to store particle information everywhere else but just copies necessary information, like position, to the more compact and aligned SOA or AOSOA layout in ParticleArray.

…n warnings

…claration warnings" This reverts commit 374b824.

junghans · 2020-01-09T19:25:48Z

How does this approach compares to https://github.com/ECP-copa/Cabana?

Cabana seems to provide a more generalized way of representing Struct-of-Arrays and Array-of-Struct-of-Arrays (where attributes are template arguments) similar to how the data are stored in ParticleArray (where the attributes are explicitly listed). Our approach, on the other hand, still uses the original std::vector<Particle> (an extended AOSOA) to store particle information everywhere else but just copies necessary information, like position, to the more compact and aligned SOA or AOSOA layout in ParticleArray.

Cabana has support for GPU data layouts, too.

jkrajniak

LGTM, however please adjust the copyright years in the file headers. Also, I've pointed out some places where there is room for small adjustments.

jnvance · 2020-01-28T16:28:18Z

It seems the CI failures all result from The job exceeded the maximum log length, and has been terminated. during the make install phase of the test script. Most of the log is from deprecation warnings about std::auto_ptr from the internal Boost library on Ubuntu with GCC. The limit is around 4MB, and even the corresponding logs for the master branch are close to exceeding this limit.

What do you guys think would be the best way to remove or suppress the warnings?

jkrajniak · 2020-01-28T16:33:02Z

Let's try with -Wdeprecated-declarations

jnvance · 2020-01-28T16:42:59Z

Let's try with -Wdeprecated-declarations

I placed that flag in the Dockerfile before but reverted it on advice of @junghans

Can we also place that flag directly in .travis.yml just for env: DISTRO=ubuntu EXTERNAL=OFF, DISTRO=ubuntu_rolling EXTERNAL=OFF and DISTRO=ubuntu_mpich EXTERNAL=OFF with compiler: gcc?

jkrajniak · 2020-01-29T17:23:22Z

Let's try with -Wdeprecated-declarations

I placed that flag in the Dockerfile before but reverted it on advice of @junghans

Can we also place that flag directly in .travis.yml just for env: DISTRO=ubuntu EXTERNAL=OFF, DISTRO=ubuntu_rolling EXTERNAL=OFF and DISTRO=ubuntu_mpich EXTERNAL=OFF with compiler: gcc?

You can try; These warnings are related to boost library, I wouldn't care too much about switching this off.

junghans · 2020-01-29T17:28:03Z

@jnvance see https://github.com/votca/tools/blob/master/include/votca/tools/eigen.h#L34-L55 for an example on how to disable warning from an included header.

…eed limit

jkrajniak

LGTM

jkrajniak · 2020-02-01T13:58:33Z

@@ -1,4 +1,6 @@
 /*
+  Copyright (C) 2018-2019


@junghans do you think this copyright of VOTCA team should be placed here?

James Vance added 12 commits January 7, 2020 16:08

Vectorized verlet list rebuild and force calculation

e3c6b94

Implemented rebuildPairs which must be called before running non-vect…

6aae44f

…orized analysis

Added some lines for documentation

14de1bb

Added some lines for documentation

e2985d3

Removed some restrict and auto keywords not compatible with gcc

60b91ad

Added test for vectorization

176a726

Added/fixed documentation

0f3fea8

Added cellAdjust signal

0402b23

Added sample script showing vectorized LJ simulation

6b32bf2

Reduced number of steps for vectorization test

66d1f9f

Fixed errors with restrict on clang

e281ff1

Fixed errors with restrict on clang

1ce2568

jnvance added the performance label Jan 8, 2020

jkrajniak self-requested a review January 8, 2020 16:39

James Vance added 2 commits January 9, 2020 12:00

Reduce Travis CI log file size by switching off deprecated declaratio…

374b824

…n warnings

Reduce verlet_list_buffer test data

2c58007

junghans reviewed Jan 9, 2020

View reviewed changes

Comment thread docker/Dockerfile Outdated

Revert "Reduce Travis CI log file size by switching off deprecated de…

d0bba59

…claration warnings" This reverts commit 374b824.

jkrajniak previously approved these changes Jan 24, 2020

View reviewed changes

James Vance added 6 commits January 27, 2020 10:49

Use camel case for C++ methods

f132f56

Added labels for AOS and SOA, and simplified if block

5239e33

Added separate test for AOS and SOA

9f8a7de

Removed slower and obsolete parts of vectorized verlet list

b1d0d4c

Simplified if block in addition of particles to neighborList

f10ec29

Updated copyright year for vectorization source files

81c8c68

jnvance dismissed jkrajniak’s stale review via 81c8c68 January 27, 2020 14:23

jkrajniak previously approved these changes Jan 27, 2020

View reviewed changes

Comment thread src/vectorization/Vectorization.py Outdated

Minor cleanup

40a8854

jnvance dismissed jkrajniak’s stale review via 40a8854 January 28, 2020 08:54

jkrajniak self-requested a review January 28, 2020 16:04

jkrajniak previously approved these changes Jan 28, 2020

View reviewed changes

Reduce number of warnings emitted by gcc causing travis-ci log to exc…

21cf2cc

…eed limit

jnvance dismissed jkrajniak’s stale review via 21cf2cc January 30, 2020 13:52

jkrajniak previously approved these changes Feb 1, 2020

View reviewed changes

Updated copyright

6406e51

jnvance dismissed jkrajniak’s stale review via 6406e51 February 1, 2020 08:23

jkrajniak reviewed Feb 1, 2020

View reviewed changes

junghans approved these changes Feb 1, 2020

View reviewed changes

jkrajniak self-requested a review February 1, 2020 14:14

jkrajniak approved these changes Feb 1, 2020

View reviewed changes

jkrajniak merged commit 6f20e3e into espressopp:master Feb 1, 2020

jnvance deleted the vectorization branch February 4, 2020 16:44

jnvance restored the vectorization branch February 4, 2020 16:44

jnvance deleted the vectorization branch February 4, 2020 16:45

Conversation

jnvance commented Jan 8, 2020

Uh oh!

junghans commented Jan 8, 2020

Uh oh!

jnvance commented Jan 9, 2020

Uh oh!

Uh oh!

junghans commented Jan 9, 2020

Uh oh!

jkrajniak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jnvance commented Jan 28, 2020

Uh oh!

jkrajniak commented Jan 28, 2020

Uh oh!

jnvance commented Jan 28, 2020

Uh oh!

jkrajniak commented Jan 29, 2020

Uh oh!

junghans commented Jan 29, 2020

Uh oh!

jkrajniak left a comment

Choose a reason for hiding this comment

Uh oh!

jkrajniak Feb 1, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants