Skip to content

[SPARK-8999][MLlib] PrefixSpan non-temporal sequences #7818

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
feynmanliang wants to merge 26 commits into from
Closed

[SPARK-8999][MLlib] PrefixSpan non-temporal sequences #7818

feynmanliang wants to merge 26 commits into from

Conversation

@feynmanliang
Copy link
Contributor

@feynmanliang feynmanliang commented Jul 31, 2015

@mengxr Extends PrefixSpan to non-temporal itemsets. Continues work by @zhangjiajin

  • Internal API uses List[Set[Int]] which is likely not efficient; will need to refactor during QA

Closes #7646

zhangjiajin added 14 commits July 22, 2015 16:16
@zhangjiajin
Add new algorithm PrefixSpan and test file.
c6ceb63
@zhangjiajin
Delete Prefixspan.scala
d9d8137 
Use PrefixSpan.scala instead of Prefixspan.scala. Delete Prefixspan.scala
@zhangjiajin
Delete PrefixspanSuite.scala
c1d13d0 
Use PrefixSpanSuite.scala instead of PrefixspanSuite.scala, Delete PrefixspanSuite.scala.
@zhangjiajin
Add feature: Collect enough frequent prefixes before projection in Pr…
a7e50d4 
…efixSpan.
@zhangjiajin
fix a scala style error.
f06772f
@zhangjiajin
initialize file before rebase.
b572f54
@zhangjiajin
Support non-temporal sequence in PrefixSpan
216ab0c
@zhangjiajin
Fixed some Scala style errors.
ae8c02d
@zhangjiajin
Merge branch 'master' of https://github.com/apache/spark into multiIt…
09dc409 
…ems_2
@zhangjiajin
restore original version
a5d649d
@zhangjiajin
support non-temporal sequence
5785cb8
@zhangjiajin
fixed a scala style error.
740c203
@zhangjiajin
fixed a scala style error.
60a0b76
@zhangjiajin
change maxLocalProjDBSize and fix a bug (remove -3 from frequent items).
f486dcd
@feynmanliang feynmanliang force-pushed the SPARK-8999-nonTemporal branch from 8b74bbf to 7d41217 Compare July 31, 2015 06:52
@SparkQA
Copy link

SparkQA commented Jul 31, 2015

Test build #39177 has finished for PR 7818 at commit 7d41217.

  • This patch fails Python style tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@feynmanliang feynmanliang force-pushed the SPARK-8999-nonTemporal branch from 7d41217 to 41ba908 Compare July 31, 2015 06:59
@SparkQA
Copy link

SparkQA commented Jul 31, 2015

Test build #39181 has finished for PR 7818 at commit 41ba908.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@feynmanliang feynmanliang changed the title Spark 8999 non temporal [SPARK-8999][MLlib] PrefixSpan non-temporal sequences Jul 31, 2015
@feynmanliang
Copy link
Contributor Author

feynmanliang commented Jul 31, 2015

Jenkins retest this please

@SparkQA
Copy link

SparkQA commented Jul 31, 2015

Test build #39184 has finished for PR 7818 at commit 41ba908.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 31, 2015

Test build #175 has finished for PR 7818 at commit 41ba908.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@mengxr
Copy link
Contributor

mengxr commented Jul 31, 2015

@feynmanliang Could you fix code style so Jenkins can run tests?

@SparkQA
Copy link

SparkQA commented Jul 31, 2015

Test build #39231 has finished for PR 7818 at commit f5cdff4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@feynmanliang
Copy link
Contributor Author

feynmanliang commented Jul 31, 2015

@mengxr Please wait until I integrate @zhangjiajin 's updated #7646 tests and rebase so his commits are in the history.

@feynmanliang feynmanliang force-pushed the SPARK-8999-nonTemporal branch from f5cdff4 to 03156ca Compare July 31, 2015 18:23
@feynmanliang
Copy link
Contributor Author

feynmanliang commented Jul 31, 2015

@mengxr Ready for review

@SparkQA
Copy link

SparkQA commented Jul 31, 2015

Test build #39249 has finished for PR 7818 at commit 03156ca.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@feynmanliang feynmanliang mentioned this pull request Jul 31, 2015
Closed
mengxr
mengxr reviewed Jul 31, 2015
View reviewed changes
mllib/src/main/scala/org/apache/spark/mllib/fpm/LocalPrefixSpan.scala Outdated
Copy link
Contributor

@mengxr mengxr Jul 31, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove blank line

Copy link
Contributor Author

@feynmanliang feynmanliang Jul 31, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

@mengxr
Copy link
Contributor

mengxr commented Jul 31, 2015

Except inline comments, my major comment is that we should process one item at a time rather than generating all subsets for each itemset. But we can also merge this first and update the implementation later because we also have API change to merge before 1.5.

@feynmanliang
Copy link
Contributor Author

feynmanliang commented Jul 31, 2015

OK, lets merge this and #7837 to get API down before feature freeze

@SparkQA
Copy link

SparkQA commented Aug 1, 2015

Test build #39299 has finished for PR 7818 at commit 350e67e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 1, 2015

Test build #39309 has finished for PR 7818 at commit 4ded81d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@feynmanliang
Copy link
Contributor Author

feynmanliang commented Aug 1, 2015

@mengxr do we plan to merge this before 1.5?

@asfgit asfgit closed this in d2a9b66 Aug 1, 2015
@mengxr
Copy link
Contributor

mengxr commented Aug 1, 2015

Merged into master. Thanks!

asfgit pushed a commit that referenced this pull request Aug 2, 2015
@mengxr
[SPARK-9000] [MLLIB] Support generic item types in PrefixSpan
28d944e 
mengxr Please review after #7818 merges and master is rebased.

Continues work by rikima

Closes #7400

Author: Feynman Liang <fliang@databricks.com>
Author: masaki rikitoku <rikima3132@gmail.com>

Closes #7837 from feynmanliang/SPARK-7400-genericItems and squashes the following commits:

8b2c756 [Feynman Liang] Remove orig
92443c8 [Feynman Liang] Style fixes
42c6349 [Feynman Liang] Style fix
14e67fc [Feynman Liang] Generic prefixSpan itemtypes
b3b21e0 [Feynman Liang] Initial support for generic itemtype in public api
b86e0d5 [masaki rikitoku] modify to support generic item type
@feynmanliang feynmanliang deleted the SPARK-8999-nonTemporal branch August 3, 2015 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@feynmanliang @SparkQA @mengxr @zhangjiajin