[qa] blockstore: Switch to dumb dbm #8834

maflcko · 2016-09-29T10:24:10Z

I could not see any performance degradation by this commit:

$ time qa/pull-tester/rpc-tests.py p2p-fullblocktest

Before:

TEST                 | PASSED | DURATION

p2p-fullblocktest.py | True   | 297 s

ALL                  | True   | 297 s (accumulated)

Runtime: 297 s

real    4m57.633s
user    4m1.588s
sys 0m11.419s

After:

p2p-fullblocktest.py:
Pass: True, Duration: 288 s

TEST                 | PASSED | DURATION

p2p-fullblocktest.py | True   | 288 s

ALL                  | True   | 288 s (accumulated)

Runtime: 288 s

real    4m48.573s
user    3m56.126s
sys 0m11.823s

laanwj · 2016-09-29T11:02:23Z

Concept ACK, this is by far the most straightforward solution, if there is no need for platform-specific exceptions then let's avoid them.

laanwj · 2016-09-29T11:53:33Z

Interesting, apparently this causes the NULLDUMMY test to fail during close, on some platforms. I don't understand why this file would suddenly be missing during close:

stderr:
Exception ignored in: <bound method _Database.close of <dbm.dumb._Database object at 0x7f8f097f1be0>>
Traceback (most recent call last):
  File "/usr/lib/python3.4/dbm/dumb.py", line 250, in close
    self._commit()
  File "/usr/lib/python3.4/dbm/dumb.py", line 112, in _commit
    with self._io.open(self._dirfile, 'w', encoding="Latin-1") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/testkz7n1z0q/1/blocks.dir'
Exception ignored in: <bound method _Database.close of <dbm.dumb._Database object at 0x7f8f097f1d68>>
Traceback (most recent call last):
  File "/usr/lib/python3.4/dbm/dumb.py", line 250, in close
    self._commit()
  File "/usr/lib/python3.4/dbm/dumb.py", line 112, in _commit
    with self._io.open(self._dirfile, 'w', encoding="Latin-1") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/testkz7n1z0q/1/transactions.dir'

I can reproduce this locally on FreeBSD.
Are perhaps the database files wiped before closing it?
This only affects nulldummy.py not the other tests involving block stores.

jonasschnelli · 2016-09-29T12:05:05Z

Failed on my OSX machine:

time qa/pull-tester/rpc-tests.py p2p-fullblocktest
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
p2p-fullblocktest.py:
Initializing test directory /var/folders/hp/kb9p9q8x4k3_z_ccy588hxrc0000gn/T/test7n1apn12/0
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11000
Test 1: PASS [1]
Test 2: PASS [100]
Test 3: PASS [101]
Test 4: PASS [102]
Test 5: PASS [102]
Test 6: PASS [103]
Test 7: PASS [103]
Test 8: PASS [104]
Test 9: PASS [104]
Test 10: PASS [104]
Test 11: PASS [104]
Test 12: PASS [104]
Test 13: PASS [104]
Test 14: PASS [104]
Test 15: PASS [104]
Test 16: PASS [105]
Test 17: PASS [106]
Test 18: PASS [106]
Test 19: PASS [106]
Test 20: PASS [106]
Test 21: PASS [106]
Test 22: PASS [106]
Test 23: PASS [106]
Test 24: PASS [106]
Test 25: PASS [107]
Test 26: PASS [107]
Test 27: PASS [107]
Test 28: PASS [107]
Test 29: PASS [107]
Test 30: PASS [107]
Test 31: PASS [107]
Test 32: PASS [108]
Test 33: PASS [109]
Test 34: PASS [109]
Test 35: PASS [110]
Test 36: PASS [110]
Test 37: PASS [111]
Test 38: PASS [111]
Test 39: PASS [111]
Test 40: PASS [111]
Test 41: PASS [112]
Test 42: PASS [112]
Test 43: PASS [113]
Test 44: PASS [113]
Test 45: PASS [114]
Test 46: PASS [115]
Test 47: PASS [115]
Test 48: PASS [115]
Test 49: PASS [115]
Test 50: PASS [115]
Test 51: PASS [115]
Test 52: PASS [115]
Test 53: PASS [115]
Test 54: PASS [115]
Test 55: PASS [115]
Test 56: PASS [115]
Test 57: PASS [116]
Test 58: PASS [116]
Test 59: PASS [116]
Test 60: PASS [117]
Test 61: PASS [117]
Test 62: PASS [117]
Test 63: PASS [117]
Test 64: PASS [118]
Test 65: PASS [118]
Test 66: PASS [118]
Test 67: PASS [118]
Test 68: PASS [118]
Test 69: PASS [119]
Test 70: PASS [120]
Test 71: PASS [120]
Test 72: PASS [120]
Test 73: PASS [120]
Test 74: PASS [121]
Test 75: PASS [121]
Test 76: PASS [121]
Test 77: PASS [122]
Test 78: PASS [122]
Test 79: PASS [122]
Test 80: PASS [123]
Test 81: PASS [124]
Test 82: PASS [125]
Test 83: PASS [126]
Test 84: PASS [127]
Test 85: PASS [127]
Test 86: PASS [127]
Test 87: PASS [128]
Test 88: PASS [129]
Test 89: PASS [130]
Test 90: PASS [130]
Test 91: PASS [131]
Test 92: PASS [131]
Test 93: PASS [132]
Test 94: PASS [132]
Test 95: PASS [1220]
Test 96: PASS [1220]
Test 97: PASS [1221]
Test 98: PASS [1221]
Unexpected exception caught during testing: ConnectionResetError(54, 'Connection reset by peer')
Stopping nodes
Not cleaning up dir /var/folders/hp/kb9p9q8x4k3_z_ccy588hxrc0000gn/T/test7n1apn12/0
Failed

stderr:
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/test_framework.py", line 151, in main
    self.run_test()
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/p2p-fullblocktest.py", line 73, in run_test
    self.test.run()
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/comptool.py", line 395, in run
    print("Test %d: PASS" % test_number, [ c.rpc.getblockcount() for c in self.connections ])
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/comptool.py", line 395, in <listcomp>
    print("Test %d: PASS" % test_number, [ c.rpc.getblockcount() for c in self.connections ])
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/coverage.py", line 49, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 148, in __call__
    response = self._request('POST', self.__url.path, postdata.encode('utf-8'))
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 125, in _request
    return self._get_response()
  File "/Users/jonasschnelli/Documents/bitcoin/bitcoin/qa/rpc-tests/test_framework/authproxy.py", line 163, in _get_response
    http_response = self.__conn.getresponse()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1197, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)

Pass: False, Duration: 928 s

TEST                 | PASSED | DURATION

p2p-fullblocktest.py | False  | 928 s

ALL                  | False  | 928 s (accumulated)

Runtime: 928 s

real    15m28.427s
user    14m47.171s
sys 0m31.176s

Trying again....

laanwj · 2016-09-29T12:19:12Z

"Connection reset by peer" is a different issue, unrelated to the database. I have the same on FreeBSD. @jonasschnelli can you try:

diff --git a/qa/rpc-tests/test_framework/authproxy.py b/qa/rpc-tests/test_framework/authproxy.py
index f5e0be2..fd7f32b 100644
--- a/qa/rpc-tests/test_framework/authproxy.py
+++ b/qa/rpc-tests/test_framework/authproxy.py
@@ -130,8 +130,9 @@ class AuthServiceProxy(object):
                 return self._get_response()
             else:
                 raise
-        except BrokenPipeError:
-            # Python 3.5+ raises this instead of BadStatusLine when the connection was reset
+        except (BrokenPipeError,ConnectionResetError):
+            # Python 3.5+ raises BrokenPipeError instead of BadStatusLine when the connection was reset
+            # ConnectionResetError happens on FreeBSD with Python 3.4
             self.__conn.close()
             self.__conn.request(method, path, postdata, headers)
             return self._get_response()

jonasschnelli · 2016-09-29T12:53:33Z

Passed after adding ConnectionResetError.

time qa/pull-tester/rpc-tests.py p2p-fullblocktest
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
p2p-fullblocktest.py:
Pass: True, Duration: 969 s

TEST                 | PASSED | DURATION

p2p-fullblocktest.py | True   | 969 s

ALL                  | True   | 969 s (accumulated)

Runtime: 969 s

real    16m9.820s
user    15m22.178s
sys 0m34.593s

maflcko · 2016-09-29T13:37:42Z

Interesting, apparently this causes the NULLDUMMY test to fail during close, on some platforms. I don't understand why this file would suddenly be missing during close:

Unrelated to this pull, but needs travis retriggered after #8835

laanwj · 2016-09-29T14:46:56Z

Unrelated to this pull, but needs travis retriggered after #8835

Done

This is necessary on FreeBSD and MacOSX, at least. See bitcoin#8834 (comment)

laanwj · 2016-09-29T15:22:51Z

Tested ACK fa9cd25

fa9cd25 [qa] blockstore: Switch to dumb dbm (MarcoFalke)

Github-Pull: bitcoin#8834 Rebased-From: fa9cd25

This is necessary on FreeBSD and MacOSX, at least. See bitcoin#8834 (comment) Github-Pull: bitcoin#8839 Rebased-From: 1d28faf

This is necessary on FreeBSD and MacOSX, at least. See bitcoin/bitcoin#8834 (comment)

This is necessary on FreeBSD and MacOSX, at least. See bitcoin#8834 (comment)

fa9cd25 [qa] blockstore: Switch to dumb dbm (MarcoFalke)

[qa] blockstore: Switch to dumb dbm

fa9cd25

maflcko added Tests Needs backport labels Sep 29, 2016

maflcko added this to the 0.13.1 milestone Sep 29, 2016

laanwj mentioned this pull request Sep 29, 2016

[qa] nulldummy.py: Don't run unused code #8835

Merged

laanwj added a commit to laanwj/bitcoin that referenced this pull request Sep 29, 2016

test: Avoid ConnectionResetErrors during RPC tests

1d28faf

This is necessary on FreeBSD and MacOSX, at least. See bitcoin#8834 (comment)

laanwj mentioned this pull request Sep 29, 2016

test: Avoid ConnectionResetErrors during RPC tests #8839

Merged

laanwj merged commit fa9cd25 into bitcoin:master Sep 29, 2016

laanwj added a commit that referenced this pull request Sep 29, 2016

Merge #8834: [qa] blockstore: Switch to dumb dbm

c9d7b0d

fa9cd25 [qa] blockstore: Switch to dumb dbm (MarcoFalke)

maflcko deleted the Mf1610-qaBlockstoreDumb branch September 29, 2016 15:25

maflcko pushed a commit to maflcko/bitcoin-core that referenced this pull request Oct 3, 2016

[qa] blockstore: Switch to dumb dbm

1dd1783

Github-Pull: bitcoin#8834 Rebased-From: fa9cd25

maflcko pushed a commit to maflcko/bitcoin-core that referenced this pull request Oct 3, 2016

test: Avoid ConnectionResetErrors during RPC tests

31ab2f8

This is necessary on FreeBSD and MacOSX, at least. See bitcoin#8834 (comment) Github-Pull: bitcoin#8839 Rebased-From: 1d28faf

maflcko removed the Needs backport label Oct 13, 2016

str4d pushed a commit to str4d/zcash that referenced this pull request Mar 24, 2017

test: Avoid ConnectionResetErrors during RPC tests

e93a3e1

This is necessary on FreeBSD and MacOSX, at least. See bitcoin/bitcoin#8834 (comment)

ftrader mentioned this pull request Apr 5, 2017

Fix walletbackup.py failure in Python 3.5+ BitcoinUnlimited/BitcoinUnlimited#385

Merged

dagurval pushed a commit to dagurval/bitcoinxt that referenced this pull request Aug 28, 2017

test: Avoid ConnectionResetErrors during RPC tests

621894d

This is necessary on FreeBSD and MacOSX, at least. See bitcoin/bitcoin#8834 (comment)

dagurval pushed a commit to dagurval/bitcoinxt that referenced this pull request Aug 28, 2017

test: Avoid ConnectionResetErrors during RPC tests

7e28438

This is necessary on FreeBSD and MacOSX, at least. See bitcoin/bitcoin#8834 (comment)

dagurval pushed a commit to dagurval/bitcoinxt that referenced this pull request Aug 29, 2017

test: Avoid ConnectionResetErrors during RPC tests

4e3c29e

This is necessary on FreeBSD and MacOSX, at least. See bitcoin/bitcoin#8834 (comment)

dagurval pushed a commit to dagurval/bitcoinxt that referenced this pull request Sep 18, 2017

test: Avoid ConnectionResetErrors during RPC tests

9b7555e

This is necessary on FreeBSD and MacOSX, at least. See bitcoin/bitcoin#8834 (comment)

OlegGirko pushed a commit to OlegGirko/dash that referenced this pull request Nov 7, 2017

test: Avoid ConnectionResetErrors during RPC tests

c50ed3f

This is necessary on FreeBSD and MacOSX, at least. See bitcoin#8834 (comment)

codablock pushed a commit to codablock/dash that referenced this pull request Jan 12, 2018

Merge bitcoin#8834: [qa] blockstore: Switch to dumb dbm

e6af218

fa9cd25 [qa] blockstore: Switch to dumb dbm (MarcoFalke)

andvgal pushed a commit to energicryptocurrency/gen2-energi that referenced this pull request Jan 6, 2019

Merge bitcoin#8834: [qa] blockstore: Switch to dumb dbm

c91e199

fa9cd25 [qa] blockstore: Switch to dumb dbm (MarcoFalke)

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[qa] blockstore: Switch to dumb dbm #8834

[qa] blockstore: Switch to dumb dbm #8834

maflcko commented Sep 29, 2016

Uh oh!

laanwj commented Sep 29, 2016

Uh oh!

laanwj commented Sep 29, 2016 •

edited

Loading

Uh oh!

jonasschnelli commented Sep 29, 2016

Uh oh!

laanwj commented Sep 29, 2016 •

edited

Loading

Uh oh!

jonasschnelli commented Sep 29, 2016

Uh oh!

maflcko commented Sep 29, 2016

Uh oh!

laanwj commented Sep 29, 2016

Uh oh!

laanwj commented Sep 29, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[qa] blockstore: Switch to dumb dbm #8834

[qa] blockstore: Switch to dumb dbm #8834

Conversation

maflcko commented Sep 29, 2016

Uh oh!

laanwj commented Sep 29, 2016

Uh oh!

laanwj commented Sep 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonasschnelli commented Sep 29, 2016

Uh oh!

laanwj commented Sep 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonasschnelli commented Sep 29, 2016

Uh oh!

maflcko commented Sep 29, 2016

Uh oh!

laanwj commented Sep 29, 2016

Uh oh!

laanwj commented Sep 29, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

laanwj commented Sep 29, 2016 •

edited

Loading

laanwj commented Sep 29, 2016 •

edited

Loading