Inverse for Hermitian matrix by anaruse · Pull Request #2495 · cupy/cupy

anaruse · 2019-09-25T10:31:53Z

This PR adds a function cupyx.linalg.invh to compute a inverse of Hermetian matrix.

This is related to #2313.

hvy · 2019-09-26T05:32:52Z

Thanks for the PR. Do you have any benchmarks or an idea how much of a performance gain this gives over inv?

anaruse · 2019-09-26T09:23:12Z

The performance measure results on GV100 are as follows. invh is about twice as fast as inv.

[<class 'numpy.float32'>]
	size, cupy_inv (ms), cupyx_invh (ms)
	250, 1.2500740587711334, 0.6322164088487625
	500, 3.1124955974519253, 1.141417771577835
	1000, 7.6446679420769215, 3.0313841067254543
	2000, 16.841696109622717, 6.631651613861322
	4000, 50.19212290644646, 25.125201791524887
[<class 'numpy.float64'>]
	size, cupy_inv (ms), cupyx_invh (ms)
	250, 1.2039597146213055, 0.6404704414308071
	500, 3.2406019046902657, 1.2113604694604874
	1000, 8.462569955736399, 3.5064077004790306
	2000, 20.811030082404613, 9.409281425178051
	4000, 72.34507771208882, 42.73106111213565

import cupy
import cupyx
from cupy.cuda import stream
import timeit

loop = 10
st = stream.Stream()

def cupy_inv(a):
    with st:
        b = cupy.linalg.inv(a)
    st.synchronize()
    return b
                                                                                                                    
def cupyx_invh(a):
    with st:
        b = cupyx.linalg.invh(a)
    st.synchronize()
    return b

def create_symmetric_matrix(n, dtype):
    a = cupy.random.random((n, n)).astype(dtype)
    a = a + a.T + cupy.eye(n, dtype=dtype) * n
    return a

# dummy run
a = create_symmetric_matrix(1, cupy.float32)
cupy_inv(a)
cupyx_invh(a)

# measurement run
for dtype in (cupy.float32, cupy.float64):
    print('[{}]'.format(dtype))
    print('\t{}, {}, {}'.format('size', 'cupy_inv (ms)', 'cupyx_invh (ms)'))
    for n in (250, 500, 1000, 2000, 4000):
        a = create_symmetric_matrix(n, dtype)
        time_cupy_inv = timeit.timeit('cupy_inv(a)', globals=globals(),
                                      number=loop)
        time_cupyx_invh = timeit.timeit('cupyx_invh(a)', globals=globals(),
                                        number=loop)
        print('\t{}, {}, {}'.format(n,
                                    time_cupy_inv / loop * 1000,
                                    time_cupyx_invh / loop * 1000))

leofang · 2019-09-26T17:25:27Z

It's also about 2x speedup on P100 and V100.

P100:

leofang@p100:~/test$ python test_new_invh.py 
[<class 'numpy.float32'>]
	size, cupy_inv (ms), cupyx_invh (ms)
	250, 3.173336200416088, 1.4493644936010242
	500, 6.208867812529206, 2.329370193183422
	1000, 12.1913154842332, 5.0745195941999555
	2000, 27.29409351013601, 12.957816990092397
	4000, 78.97032969631255, 46.617854200303555
[<class 'numpy.float64'>]
	size, cupy_inv (ms), cupyx_invh (ms)
	250, 3.098821802996099, 1.3088918989524245
	500, 6.121482094749808, 2.3283300921320915
	1000, 13.830788992345333, 5.601490498520434
	2000, 32.329136319458485, 15.229962696321309
	4000, 104.9062664853409, 65.85105890408158
leofang@p100:~/test$ python test_new_invh.py 
[<class 'numpy.float32'>]
	size, cupy_inv (ms), cupyx_invh (ms)
	250, 3.156185895204544, 1.1962832184508443
	500, 6.207655998878181, 2.3230151971802115
	1000, 12.610913487151265, 5.125051084905863
	2000, 27.278455207124352, 12.960979994386435
	4000, 78.91009959857911, 46.63353420328349
[<class 'numpy.float64'>]
	size, cupy_inv (ms), cupyx_invh (ms)
	250, 2.9437166871502995, 1.116643799468875
	500, 6.063109403476119, 2.2480450104922056
	1000, 13.806240912526846, 5.5285342037677765
	2000, 32.42374949622899, 15.16240001656115
	4000, 105.13669641222805, 65.93173949513584

V100:

leofang@v100:~/test$ python test_new_invh.py 
[<class 'numpy.float32'>]
	size, cupy_inv (ms), cupyx_invh (ms)
	250, 3.5224422812461853, 0.9978475980460644
	500, 5.643645208328962, 2.0491267554461956
	1000, 11.20378440245986, 4.316417686641216
	2000, 22.649669833481312, 9.237194713205099
	4000, 60.89561861008406, 29.915789421647787
[<class 'numpy.float64'>]
	size, cupy_inv (ms), cupyx_invh (ms)
	250, 2.586251962929964, 0.8944944478571415
	500, 5.31118530780077, 1.9136645831167698
	1000, 12.001861166208982, 4.664854798465967
	2000, 26.72860072925687, 11.140924878418446
	4000, 80.86111126467586, 46.179613936692476
leofang@v100:~/test$ python test_new_invh.py 
[<class 'numpy.float32'>]
	size, cupy_inv (ms), cupyx_invh (ms)
	250, 2.8356215916574, 1.0268189013004303
	500, 5.62763623893261, 2.0748370327055454
	1000, 11.101678851991892, 4.331118427217007
	2000, 22.502884082496166, 9.287280309945345
	4000, 60.76891012489796, 29.883697628974915
[<class 'numpy.float64'>]
	size, cupy_inv (ms), cupyx_invh (ms)
	250, 2.5642848573625088, 0.9037657640874386
	500, 5.302123725414276, 1.9433654844760895
	1000, 11.950754560530186, 4.686544090509415
	2000, 26.559684239327908, 11.151218507438898
	4000, 80.61537211760879, 46.08936421573162

hvy · 2019-09-27T05:49:28Z

Thanks @anaruse and @leofang. I could also verify the speedups on P100, V100. Could you proceed with writing unit tests? We could e.g. compare the results with numpy.linalg.inv?

cupyx/linalg/linalg.py

anaruse · 2019-10-09T05:10:51Z

Thanks for your comment. I added unit tests for cupyx.linalg.invh. Could you review it again?

hvy · 2019-10-17T09:18:22Z

A detail but how about placing the function in solve.py (and the test to test_solve.py) instead of linalg.py, more similar to how inv is located?

hvy · 2019-10-18T10:40:33Z

Oh, this seems to apply for this PR as well #2529 Could you remove the availability checks?

anaruse · 2019-10-21T02:37:42Z

Thanks for your comment, I changed file names and removed cuSolver availability checking. Could you check this again?

hvy · 2019-10-21T10:03:17Z

Thanks, LGTM!

hvy · 2019-10-21T10:03:31Z

Jenkins, test this please.

pfn-ci-bot · 2019-10-21T10:03:35Z

Successfully created a job for commit 219f64f:

Dashboard for commit 219f64f

chainer-ci · 2019-10-21T10:38:00Z

Jenkins CI test (for commit 219f64f, target branch master) failed with status FAILURE.

hvy · 2019-10-23T10:37:12Z

Sorry it seems like tests are failing on Jenkins (test collection) due to the fact that __init__.py is missing in the test directory. Could you possibly add it in this PR?

anaruse · 2019-10-24T02:26:52Z

I added a empty __init__.py file, though I'm not sure if this will solve the issue..

hvy · 2019-10-24T02:32:35Z

Thanks @anaruse , I'll try running it to check.

Jenkins, test this please.

pfn-ci-bot · 2019-10-24T02:32:38Z

Successfully created a job for commit 13918b7:

Dashboard for commit 13918b7

chainer-ci · 2019-10-24T03:06:15Z

Jenkins CI test (for commit 13918b7, target branch master) succeeded!

leofang · 2019-10-24T03:21:49Z

Wow it really worked. What a bizarre bug...I think @hvy was referring to this pytest-dev/pytest#774 (comment)?

hvy · 2019-10-28T01:27:20Z

Jenkins, test this please.

pfn-ci-bot · 2019-10-28T01:27:23Z

Successfully created a job for commit 13918b7:

Dashboard for commit 13918b7

chainer-ci · 2019-10-28T02:42:14Z

Jenkins CI test (for commit 13918b7, target branch master) succeeded!

hvy · 2019-10-28T02:45:31Z

LGTM!

anaruse · 2019-10-28T04:47:46Z

Thank you for merging the PR!

Add invh to compute a inverse of Hermetian matrix

d8b06b9

anaruse changed the title ~~Inverse for Hermitian matrix~~ [WIP] Inverse for Hermitian matrix Sep 25, 2019

asi1024 assigned hvy Sep 26, 2019

hvy added the cat:feature New features/APIs label Sep 27, 2019

hvy suggested changes Sep 27, 2019

View reviewed changes

cupyx/linalg/linalg.py Outdated Show resolved Hide resolved

cupyx/linalg/linalg.py Outdated Show resolved Hide resolved

cupyx/linalg/linalg.py Outdated Show resolved Hide resolved

hvy mentioned this pull request Sep 27, 2019

Introduce errstate configuration to control cuSOLVER devInfo and cuBLAS infoArray checks #2437

Merged

3 tasks

anaruse added 2 commits October 9, 2019 13:07

Fix docs and error message

dce93fc

Add tests for cupyx.invh

e8f1213

Fix typos

3cd1db3

anaruse changed the title ~~[WIP] Inverse for Hermitian matrix~~ Inverse for Hermitian matrix Oct 11, 2019

Remove cuSolver availability checking and change file names

219f64f

hvy approved these changes Oct 21, 2019

View reviewed changes

hvy added this to the v7.0.0rc1 milestone Oct 21, 2019

hvy added the st:test-and-merge (deprecated) Ready to merge after test pass. label Oct 21, 2019

Add empty __init__.py to satisfy Jenkins?

13918b7

hvy removed the st:test-and-merge (deprecated) Ready to merge after test pass. label Oct 24, 2019

beam2d removed this from the v7.0.0rc1 milestone Oct 25, 2019

hvy merged commit bde961a into cupy:master Oct 28, 2019

hvy added this to the v7.0.0 milestone Oct 28, 2019

Uh oh!

Conversation

anaruse commented Sep 25, 2019

Uh oh!

hvy commented Sep 26, 2019

Uh oh!

anaruse commented Sep 26, 2019

Uh oh!

leofang commented Sep 26, 2019

Uh oh!

hvy commented Sep 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anaruse commented Oct 9, 2019

Uh oh!

hvy commented Oct 17, 2019

Uh oh!

hvy commented Oct 18, 2019

Uh oh!

anaruse commented Oct 21, 2019

Uh oh!

hvy commented Oct 21, 2019

Uh oh!

hvy commented Oct 21, 2019

Uh oh!

pfn-ci-bot commented Oct 21, 2019

Uh oh!

chainer-ci commented Oct 21, 2019

Uh oh!

hvy commented Oct 23, 2019

Uh oh!

anaruse commented Oct 24, 2019

Uh oh!

hvy commented Oct 24, 2019

Uh oh!

pfn-ci-bot commented Oct 24, 2019

Uh oh!

chainer-ci commented Oct 24, 2019

Uh oh!

leofang commented Oct 24, 2019

Uh oh!

hvy commented Oct 28, 2019

Uh oh!

pfn-ci-bot commented Oct 28, 2019

Uh oh!

chainer-ci commented Oct 28, 2019

Uh oh!

hvy commented Oct 28, 2019

Uh oh!

anaruse commented Oct 28, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

hvy commented Sep 27, 2019 •

edited

Loading