Feature #61646: vectorize MD5 calculations for ETags - rgw - Ceph

Actions

Copy link

Feature #61646

open

vectorize MD5 calculations for ETags

Added by Casey Bodley almost 3 years ago. Updated over 2 years ago.

Status:

In Progress

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Backport:

Reviewed:

Affected Versions:

Pull request ID:

Tags (freeform):

Merge Commit:

Fixed In:

Released In:

Upkeep Timestamp:

Description

take advantage of SIMD to speed up object etag calculations

there's a c++ library that uses AVX2 instructions that's worth considering: https://github.com/ashbob999/md5-simd

similar work in golang:
https://dzone.com/articles/parallelizing-md5-checksum-computation-to-speed-up
https://github.com/minio/md5-simd

an earlier contribution in https://github.com/ceph/ceph/pull/42435 tried offloading MD5 calculations to background threads, but did not show a consistent improvement

Actions

Copy link

Updated by Casey Bodley over 2 years ago

ceph already uses the ISA-L crypto library for crypto acceleration, which includes multi-buffer md5 support from https://github.com/intel/isa-l_crypto/blob/master/include/md5_mb.h

The MD5 CTX interface functions are available for 4 architectures: SSE, AVX, AVX2 and AVX512. In addition, a multibinary interface is provided, which selects the appropriate architecture-specific function at runtime.

Actions

Copy link