Skip to content

Implementation of batching on write for speeding up Redis Mass Ingest#1464

Merged
rfecher merged 10 commits intolocationtech:masterfrom
rfecher:redis-batch
Nov 19, 2018
Merged

Implementation of batching on write for speeding up Redis Mass Ingest#1464
rfecher merged 10 commits intolocationtech:masterfrom
rfecher:redis-batch

Conversation

@rfecher
Copy link
Copy Markdown
Contributor

@rfecher rfecher commented Nov 12, 2018 •
edited
Loading

  • clearly writes needed to be batched for performance
  • compression wasn't actually wired in as anticipated (and redisson's snappycodecv2 has a major bug where it only works for 32K bytes so I changed to v1 where it chunks the data appropriately)
  • and some sort of pagination from redisson results was required to not have the potential to kill Java memory
  • "recalcstats" command deletes the previous stats now (which I think is implied by "recalc")

@rfecher rfecher merged commit c31cbdd into locationtech:master Nov 19, 2018
@rfecher rfecher deleted the redis-batch branch November 19, 2018 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@rfecher