Skip to content

Added parallel wordcount example.#475

Merged
StefanKarpinski merged 1 commit intoJuliaLang:masterfrom
ajdecon:wordcount-example
Feb 27, 2012
Merged

Added parallel wordcount example.#475
StefanKarpinski merged 1 commit intoJuliaLang:masterfrom
ajdecon:wordcount-example

Conversation

@ajdecon
Copy link
Copy Markdown
Contributor

@ajdecon ajdecon commented Feb 27, 2012

Now that the map-reduce wordcount is working, I thought it might make a good example to include as a simple illustration of spawn-fetch parallelization. I've tried to add enough comments to make it a good educational tool, explaining what each part does.

Currently the major limitation of this implementation is that the "reduce" is single-threaded. I haven't been able to find a way to perform a key-sort-and-separate step on the HashTables returned by the map step, to allow parallelizing reduce, that isn't just as slow as doing the reduce on a single node anyway.

@StefanKarpinski
Copy link
Copy Markdown
Member

This is a great example. Love it.

StefanKarpinski added a commit that referenced this pull request Feb 27, 2012
Added parallel wordcount example.
@StefanKarpinski StefanKarpinski merged commit 91364df into JuliaLang:master Feb 27, 2012
KristofferC added a commit that referenced this pull request Aug 4, 2018
KristofferC added a commit that referenced this pull request Aug 4, 2018
KristofferC added a commit that referenced this pull request Aug 4, 2018
KristofferC added a commit that referenced this pull request Aug 4, 2018
KristofferC added a commit that referenced this pull request Feb 11, 2019
Keno pushed a commit that referenced this pull request Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants