Flat buffers Master Merge #3

AlvinMooreSr · 2019-04-30T21:08:26Z

Merged Flatbuffer into master in order to ensure that the build system can built the branch
Made Windows cmake compilation updates
Resolved merge conflicts with Master files

Set the threshold of dump to 2GB.

… rateLimit is set to a low value and it would take very long to read the entire database. Fix this by setting the rateLimit to the maximum allowed value if reading the entire database is taking a long time.

…t packet

For small clusters that are growing quickly, it could happen that the…

…ase-6.1

do not log a SevError trace event if we cannot deserialize the connect packet

update installer WIX GUID following release

CC apple#1255, apple#1173

… with a less than 100% probability.

This hopefully fixes a weird msbuild issue where files are generated more than once. This issue is described here: https://gitlab.kitware.com/cmake/cmake/issues/16767 This workaround makes coverage_tool not to use the generated options files, which hopefully will result in msbuild not trying to generate these files more than once.

…rd-estimate-validation Fix consistency check shard estimate validation

* Added support for testing cpack and defined very long directory name for source because of problem with building debug rpms * Added support for building rpms by adding file package * Added preinstall target to prevent cpack from having to do so much compilation as single thread

# Conflicts: # documentation/sphinx/source/release-notes.rst

Don't pass options file to coverage tool

Fix OPEN_FOR_IDE -Wunused-variable warnings

Capitalize the first letter.

Fix unit test failure due to trace event names

Fix unused-variable warning in linux-only code

…able scl tools Removed conditionals from docker-compose

Installed boto3 via pip

Merge Release 6.1 Into Master

When cloning repository without specifying the destination, building documentation works fine. But it fail when the top level directory is not 'foundatinodb'. To reproduce: ``` git clone https://github.com/apple/foundationdb.git /tmp/apple-fdb cd /tmp/apple-fdb/documentation/sphinx make ``` will result in ``` Exception occurred: File "/Users/ssudakovich/.pyenv/versions/2.7.13/lib/python2.7/xml/etree/ElementTree.py", line 647, in parse source = open(source, "rb") IOError: [Errno 2] No such file or directory: u'/private/tmp/apple_fdb/documentation/sphinx/../../../foundationdb/versions.target' ```

# Conflicts: # build/Dockerfile # build/docker-compose.yaml

Build Docker Update

Dump heap profiler when memory usage is high

…foundationdb into flat-buffers # Conflicts: # cmake/ConfigureCompiler.cmake # fdbserver/fdbserver.actor.cpp # fdbserver/workloads/JavaWorkload.actor.cpp

/home/anoyes/workspace/foundationdb/fdbserver/Ratekeeper.actor.cpp:86:8: runtime error: load of value 1231493777, which is not a valid value for type 'limitReason_t' #0 0x310e961 in StorageQueueInfo::StorageQueueInfo(StorageQueueInfo&&) /home/anoyes/workspace/foundationdb/fdbserver/Ratekeeper.actor.cpp:86 #1 0x310eacd in MapPair<UID, StorageQueueInfo>::MapPair<UID, StorageQueueInfo>(UID&&, StorageQueueInfo&&) /home/anoyes/workspace/foundationdb/flow/IndexedSet.h:242 #2 0x310b35e in MapPair<std::decay<UID>::type, std::decay<StorageQueueInfo>::type> mapPair<UID, StorageQueueInfo>(UID&&, StorageQueueInfo&&) /home/anoyes/workspace/foundationdb/flow/IndexedSet.h:258 #3 0x30a8b79 in a_body1 /home/anoyes/workspace/foundationdb/fdbserver/Ratekeeper.actor.cpp:195 #4 0x309b529 in TrackStorageServerQueueInfoActor /home/anoyes/build/foundationdb/fdbserver/Ratekeeper.actor.g.cpp:495 #5 0x309b9be in trackStorageServerQueueInfo(RatekeeperData* const&, StorageServerInterface const&) /home/anoyes/workspace/foundationdb/fdbserver/Ratekeeper.actor.cpp:194 #6 0x30cff63 in a_body1loopBody1when1cont1 /home/anoyes/workspace/foundationdb/fdbserver/Ratekeeper.actor.cpp:303 #7 0x30cd9da in a_body1loopBody1when1when1 /home/anoyes/build/foundationdb/fdbserver/Ratekeeper.actor.g.cpp:1170 #8 0x30ed4dd in a_callback_fire /home/anoyes/build/foundationdb/fdbserver/Ratekeeper.actor.g.cpp:1185 #9 0x30e6d81 in fire /home/anoyes/workspace/foundationdb/flow/flow.h:998 #10 0x4df0dc in void SAV<Void>::send<Void>(Void&&) /home/anoyes/workspace/foundationdb/flow/flow.h:447 apple#11 0x959891 in void Promise<Void>::send<Void>(Void&&) const /home/anoyes/workspace/foundationdb/flow/flow.h:778 apple#12 0x7b4b018 in Sim2::execTask(Sim2::Task&) (/home/anoyes/build/foundationdb/bin/fdbserver+0x7b4b018) apple#13 0x7bf9168 in Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_body1loopBody1cont1(Void const&, int) /home/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:979 apple#14 0x7be7b68 in Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_body1loopBody1when1(Void const&, int) /home/anoyes/build/foundationdb/fdbrpc/sim2.actor.g.cpp:5391 apple#15 0x7c329ff in Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_callback_fire(ActorCallback<Sim2::RunLoopActor, 0, Void>*, Void) /home/anoyes/build/foundationdb/fdbrpc/sim2.actor.g.cpp:5406 apple#16 0x7c1fc73 in ActorCallback<Sim2::RunLoopActor, 0, Void>::fire(Void const&) /home/anoyes/workspace/foundationdb/flow/flow.h:998 apple#17 0x4df0dc in void SAV<Void>::send<Void>(Void&&) /home/anoyes/workspace/foundationdb/flow/flow.h:447 apple#18 0x959891 in void Promise<Void>::send<Void>(Void&&) const /home/anoyes/workspace/foundationdb/flow/flow.h:778 apple#19 0x7fe74a4 in N2::PromiseTask::operator()() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:481 apple#20 0x7fb6ff7 in N2::Net2::run() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:657 apple#21 0x7b71bd3 in Sim2::_runActorState<Sim2::_runActor>::a_body1(int) /home/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:989 apple#22 0x7b2ee51 in Sim2::_runActor::_runActor(Sim2* const&) /home/anoyes/build/foundationdb/fdbrpc/sim2.actor.g.cpp:5608 apple#23 0x7b2f268 in Sim2::_run(Sim2* const&) /home/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:987 apple#24 0x7b2f2c8 in Sim2::run() /home/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:996 apple#25 0x21040a6 in main /home/anoyes/workspace/foundationdb/fdbserver/fdbserver.actor.cpp:1793 apple#26 0x7f03492ba504 in __libc_start_main (/lib64/libc.so.6+0x22504) apple#27 0x464914 (/home/anoyes/build/foundationdb/bin/fdbserver+0x464914)

/Users/anoyes/workspace/foundationdb/fdbserver/TLogInterface.h:149:8: runtime error: load of value 232, which is not a valid value for type 'bool' #0 0xc608fb in TLogPeekReply::TLogPeekReply(TLogPeekReply const&) /Users/anoyes/workspace/foundationdb/fdbserver/TLogInterface.h:149 #1 0x242bf87 in ILogSystem::ServerPeekCursor::ServerPeekCursor(TLogPeekReply const&, LogMessageVersion const&, LogMessageVersion const&, int, int, bool, long, Tag) /Users/anoyes/workspace/foundationdb/fdbserver/LogSystemPeekCursor.actor.cpp:35 #2 0x242da77 in ILogSystem::ServerPeekCursor::cloneNoMore() /Users/anoyes/workspace/foundationdb/fdbserver/LogSystemPeekCursor.actor.cpp:47 #3 0x24362d5 in ILogSystem::MergedPeekCursor::cloneNoMore() /Users/anoyes/workspace/foundationdb/fdbserver/LogSystemPeekCursor.actor.cpp:325 #4 0x244bf45 in ILogSystem::MultiCursor::cloneNoMore() /Users/anoyes/workspace/foundationdb/fdbserver/LogSystemPeekCursor.actor.cpp:838 #5 0x36b5a36 in a_body1cont5loopBody1 /Users/anoyes/workspace/foundationdb/fdbserver/storageserver.actor.cpp:2621 #6 0x36b3110 in a_body1cont5loopHead1 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8664 #7 0x36b07fe in a_body1cont5 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8576 #8 0x36abda8 in a_body1cont4when1 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8582 #9 0x36a8dc2 in a_body1cont4 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8454 #10 0x36a4bf6 in a_body1cont3break1 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8489 apple#11 0x36a2c01 in a_body1cont3loopBody1cont1 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8505 apple#12 0x369fd36 in a_body1cont3loopBody1when1 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8513 apple#13 0x3700dcb in a_callback_fire /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8528 apple#14 0x36e5210 in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998 apple#15 0x4dfb2a in SAV<Void>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479 apple#16 0x2484b07 in a_body1loopBody1cont1 /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:1526 apple#17 0x24822cf in a_body1loopBody1cont2 /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:1535 apple#18 0x247e228 in a_body1loopBody1when1 /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:1541 apple#19 0x249be87 in a_callback_fire /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:1556 apple#20 0x249668f in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998 apple#21 0x4dfb2a in SAV<Void>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479 apple#22 0x80557e in a_body1when1 /Users/anoyes/build/foundationdb/flow/genericactors.actor.g.h:11591 apple#23 0x8916ef in a_callback_fire /Users/anoyes/build/foundationdb/flow/genericactors.actor.g.h:11620 apple#24 0x8735f5 in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998 apple#25 0x4dfb2a in SAV<Void>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479 apple#26 0x24820f8 in a_body1cont1loopBody1when1 /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:860 apple#27 0x249c852 in a_callback_fire /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:886 apple#28 0x249786c in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998 apple#29 0xc9d2dc in SAV<TLogPeekReply>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479 apple#30 0x248b39f in a_body1cont2 /Users/anoyes/build/foundationdb/flow/genericactors.actor.g.h:11858 apple#31 0x2489d02 in a_body1when1 /Users/anoyes/build/foundationdb/flow/genericactors.actor.g.h:11865 apple#32 0x249a150 in a_callback_fire /Users/anoyes/build/foundationdb/flow/genericactors.actor.g.h:11880 apple#33 0x2492a4f in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998 apple#34 0xc9d2dc in SAV<TLogPeekReply>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479 apple#35 0x248df9b in a_body1cont2 /Users/anoyes/build/foundationdb/fdbrpc/genericactors.actor.g.h:2762 apple#36 0x248b7da in a_body1when1 /Users/anoyes/build/foundationdb/fdbrpc/genericactors.actor.g.h:2769 apple#37 0x2499c88 in a_callback_fire /Users/anoyes/build/foundationdb/fdbrpc/genericactors.actor.g.h:2784 apple#38 0x2492371 in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998 apple#39 0xc9d2dc in SAV<TLogPeekReply>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479 apple#40 0xc60fb3 in void SAV<TLogPeekReply>::sendAndDelPromiseRef<TLogPeekReply&>(TLogPeekReply&) /Users/anoyes/workspace/foundationdb/flow/flow.h:472 apple#41 0xc1137a in NetSAV<TLogPeekReply>::receive(ArenaObjectReader&) /Users/anoyes/workspace/foundationdb/fdbrpc/fdbrpc.h:111 apple#42 0x78eda75 in a_body1cont1 /Users/anoyes/workspace/foundationdb/fdbrpc/FlowTransport.actor.cpp:652 apple#43 0x78f7967 in a_body1cont2 /Users/anoyes/build/foundationdb/fdbrpc/FlowTransport.actor.g.cpp:2369 apple#44 0x78ed4d8 in a_body1when1 /Users/anoyes/build/foundationdb/fdbrpc/FlowTransport.actor.g.cpp:2375 apple#45 0x791af45 in a_callback_fire /Users/anoyes/build/foundationdb/fdbrpc/FlowTransport.actor.g.cpp:2390 apple#46 0x7914670 in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998 apple#47 0x4df0dc in void SAV<Void>::send<Void>(Void&&) /Users/anoyes/workspace/foundationdb/flow/flow.h:447 apple#48 0x959891 in void Promise<Void>::send<Void>(Void&&) const /Users/anoyes/workspace/foundationdb/flow/flow.h:778 apple#49 0x7b4b022 in Sim2::execTask(Sim2::Task&) (/Users/anoyes/build/foundationdb/bin/fdbserver+0x7b4b022) apple#50 0x7bf9172 in Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_body1loopBody1cont1(Void const&, int) /Users/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:979 apple#51 0x7be7b72 in Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_body1loopBody1when1(Void const&, int) /Users/anoyes/build/foundationdb/fdbrpc/sim2.actor.g.cpp:5391 apple#52 0x7c32a09 in Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_callback_fire(ActorCallback<Sim2::RunLoopActor, 0, Void>*, Void) /Users/anoyes/build/foundationdb/fdbrpc/sim2.actor.g.cpp:5406 apple#53 0x7c1fc7d in ActorCallback<Sim2::RunLoopActor, 0, Void>::fire(Void const&) /Users/anoyes/workspace/foundationdb/flow/flow.h:998 apple#54 0x4df0dc in void SAV<Void>::send<Void>(Void&&) /Users/anoyes/workspace/foundationdb/flow/flow.h:447 apple#55 0x959891 in void Promise<Void>::send<Void>(Void&&) const /Users/anoyes/workspace/foundationdb/flow/flow.h:778 apple#56 0x7fe74ae in N2::PromiseTask::operator()() /Users/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:481 apple#57 0x7fb7001 in N2::Net2::run() /Users/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:657 apple#58 0x7b71bdd in Sim2::_runActorState<Sim2::_runActor>::a_body1(int) /Users/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:989 apple#59 0x7b2ee5b in Sim2::_runActor::_runActor(Sim2* const&) /Users/anoyes/build/foundationdb/fdbrpc/sim2.actor.g.cpp:5608 apple#60 0x7b2f272 in Sim2::_run(Sim2* const&) /Users/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:987 apple#61 0x7b2f2d2 in Sim2::run() /Users/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:996 apple#62 0x2104064 in main /Users/anoyes/workspace/foundationdb/fdbserver/fdbserver.actor.cpp:1793 apple#63 0x7fb7c6561504 in __libc_start_main (/lib64/libc.so.6+0x22504) apple#64 0x464914 (/Users/anoyes/build/foundationdb/bin/fdbserver+0x464914)

Merge upstream

Some documentation for multi-threaded client

Merge Master

jzhou77 and others added 30 commits April 5, 2019 16:12

Dump heap profiler when memory usage is high

09b2c35

Set the threshold of dump to 2GB.

Change memory profiling threshold to a flag

4b08042

Move profiling option help message to devhelp

ab834c4

For small clusters that are growing quickly, it could happen that the…

04e9aa6

… rateLimit is set to a low value and it would take very long to read the entire database. Fix this by setting the rateLimit to the maximum allowed value if reading the entire database is taking a long time.

do not log a SevError trace event if we cannot deserialize the connec…

8e05713

…t packet

Merge pull request apple#1452 from bnamasivayam/release-6.1

1a5da85

For small clusters that are growing quickly, it could happen that the…

Merge branch 'release-6.1' of github.com:apple/foundationdb into rele…

86c0a11

…ase-6.1

updated documentation for 6.1.2

e446272

Merge pull request apple#1453 from etschannen/release-6.1

0159e07

do not log a SevError trace event if we cannot deserialize the connect packet

update installer WIX GUID following release

575f317

Merge pull request apple#1455 from etschannen/prepare-release-6.1.2

cb05403

update installer WIX GUID following release

Fix OPEN_FOR_IDE -Wunused-variable warnings

781b6ec

CC apple#1255, apple#1173

Prefer success to cast-to-void

08e66d1

Fix compile error

6888baf

Don't validate the shard size estimate unless enough keys are sampled…

43533b3

… with a less than 100% probability.

Update release notes.

bb012d2

Merge pull request apple#1468 from ajbeamon/fix-consistency-check-sha…

b8e6338

…rd-estimate-validation Fix consistency check shard estimate validation

Fix more unused variable warnings

13ba915

Merge branch 'release-6.1' of copy of master

2bea995

# Conflicts: # documentation/sphinx/source/release-notes.rst

Fix more unused-variable warnings

ef04471

Add /MP option to compiler to allow for more parallelism

33b6e4e

Fix JavaWorkload unused variable

5af8208

Address review comments

d1e8677

Merge pull request apple#1470 from mpilman/fixes/windows-parallel-build

40d632b

Don't pass options file to coverage tool

Merge branch 'master' into pprof

6870e13

Merge pull request apple#1465 from atn34/fix-open-for-ide-build

65c297b

Fix OPEN_FOR_IDE -Wunused-variable warnings

Fix unit test failure due to trace event names

e7ee616

Capitalize the first letter.

Fix unused-variable warning in linux-only code

ddd6e26

jzhou77 and others added 15 commits April 21, 2019 10:37

Merge pull request apple#1474 from jzhou77/fix-traceevent

05c1924

Fix unit test failure due to trace event names

Merge pull request apple#1475 from atn34/fix-open-for-ide-linux-error

aaa53b9

Fix unused-variable warning in linux-only code

Updated the build docker file to be Centos-based

45868e9

Updated the services defined within the docker-compose file to use en…

7f5b89d

…able scl tools Removed conditionals from docker-compose

Changed working directory to fix problem with debug rpms

ad74872

Removed the setting of permissions and ownership for the /opt directory

c2f8799

Installed boto3 via pip

Updated old release 6.1 release notes

20b995d

Merge pull request apple#1471 from AlvinMooreSr/release-6.1-merge

e0f76ed

Merge Release 6.1 Into Master

Merge branch 'master' of github.com:apple/foundationdb into new-docker

0654afa

# Conflicts: # build/Dockerfile # build/docker-compose.yaml

Fix comments to use transaction_too_old instead of past_version

9fea606

Merge pull request apple#1476 from AlvinMooreSr/new-docker

0f4fd9c

Build Docker Update

Merge pull request apple#1429 from jzhou77/pprof

1a4c175

Dump heap profiler when memory usage is high

Merge branch 'features/protocol-version-flags' of github.com:mpilman/…

535b3f9

…foundationdb into flat-buffers # Conflicts: # cmake/ConfigureCompiler.cmake # fdbserver/fdbserver.actor.cpp # fdbserver/workloads/JavaWorkload.actor.cpp

Enabled c++17 for compilation

8b8b516

mpilman merged commit ca2ecb5 into mpilman:features/protocol-version-flags May 1, 2019

AlvinMooreSr deleted the flat-buffers branch May 11, 2019 02:34

mpilman pushed a commit that referenced this pull request Feb 24, 2020

Merge pull request #3 from apple/master

e715e61

Merge upstream

mpilman pushed a commit that referenced this pull request Mar 15, 2021

Merge pull request #3 from vishesh/sears-multithreaded_client

5be98b8

Some documentation for multi-threaded client

mpilman pushed a commit that referenced this pull request Mar 15, 2021

Merge pull request #3 from apple/master

f5f2629

Merge Master

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Flat buffers Master Merge #3

Flat buffers Master Merge #3

Uh oh!

AlvinMooreSr commented Apr 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

Flat buffers Master Merge #3

Flat buffers Master Merge #3

Uh oh!

Conversation

AlvinMooreSr commented Apr 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants