Skip to content

Commit 5697911

Browse files
xuanyuankingxuanyuanking
committed
sync with master, resolve conflict
2 parents 92ed369 + c329a56 commit 5697911

1,066 files changed

Lines changed: 37811 additions & 13854 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/PULL_REQUEST_TEMPLATE‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,9 @@
22

33
(Please fill in changes proposed in this fix)
44

5-
65
## How was this patch tested?
76

87
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
9-
10-
118
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
129

10+
Please review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark before opening a pull request.

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
R-unit-tests.log
2525
R/unit-tests.out
2626
R/cran-check.out
27+
R/pkg/vignettes/sparkr-vignettes.html
2728
build/*.jar
2829
build/apache-maven*
2930
build/scala*

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ It lists steps that are required before creating a PR. In particular, consider:
66

77
- Is the change important and ready enough to ask the community to spend time reviewing?
88
- Have you searched for existing, related JIRAs and pull requests?
9-
- Is this a new feature that can stand alone as a package on http://spark-packages.org ?
9+
- Is this a new feature that can stand alone as a [third party project](https://cwiki.apache.org/confluence/display/SPARK/Third+Party+Projects) ?
1010
- Is the change being proposed clearly explained and motivated?
1111

1212
When you contribute code, you affirm that the contribution is your original work and that you

‎LICENSE‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -263,7 +263,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
263263
(New BSD license) Protocol Buffer Java API (org.spark-project.protobuf:protobuf-java:2.4.1-shaded - http://code.google.com/p/protobuf)
264264
(The BSD License) Fortran to Java ARPACK (net.sourceforge.f2j:arpack_combined_all:0.1 - http://f2j.sourceforge.net)
265265
(The BSD License) xmlenc Library (xmlenc:xmlenc:0.52 - http://xmlenc.sourceforge.net)
266-
(The New BSD License) Py4J (net.sf.py4j:py4j:0.10.3 - http://py4j.sourceforge.net/)
266+
(The New BSD License) Py4J (net.sf.py4j:py4j:0.10.4 - http://py4j.sourceforge.net/)
267267
(Two-clause BSD-style license) JUnit-Interface (com.novocode:junit-interface:0.10 - http://github.com/szeiger/junit-interface/)
268268
(BSD licence) sbt and sbt-launch-lib.bash
269269
(BSD 3 Clause) d3.min.js (https://github.com/mbostock/d3/blob/master/LICENSE)

‎R/create-docs.sh‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,17 +17,26 @@
1717
# limitations under the License.
1818
#
1919

20-
# Script to create API docs for SparkR
21-
# This requires `devtools` and `knitr` to be installed on the machine.
20+
# Script to create API docs and vignettes for SparkR
21+
# This requires `devtools`, `knitr` and `rmarkdown` to be installed on the machine.
2222

2323
# After running this script the html docs can be found in
2424
# $SPARK_HOME/R/pkg/html
25+
# The vignettes can be found in
26+
# $SPARK_HOME/R/pkg/vignettes/sparkr_vignettes.html
2527

2628
set -o pipefail
2729
set -e
2830

2931
# Figure out where the script is
3032
export FWDIR="$(cd "`dirname "$0"`"; pwd)"
33+
export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
34+
35+
# Required for setting SPARK_SCALA_VERSION
36+
. "${SPARK_HOME}"/bin/load-spark-env.sh
37+
38+
echo "Using Scala $SPARK_SCALA_VERSION"
39+
3140
pushd $FWDIR
3241

3342
# Install the package (this will also generate the Rd files)
@@ -43,4 +52,21 @@ Rscript -e 'libDir <- "../../lib"; library(SparkR, lib.loc=libDir); library(knit
4352

4453
popd
4554

55+
# Find Spark jars.
56+
if [ -f "${SPARK_HOME}/RELEASE" ]; then
57+
SPARK_JARS_DIR="${SPARK_HOME}/jars"
58+
else
59+
SPARK_JARS_DIR="${SPARK_HOME}/assembly/target/scala-$SPARK_SCALA_VERSION/jars"
60+
fi
61+
62+
# Only create vignettes if Spark JARs exist
63+
if [ -d "$SPARK_JARS_DIR" ]; then
64+
# render creates SparkR vignettes
65+
Rscript -e 'library(rmarkdown); paths <- .libPaths(); .libPaths(c("lib", paths)); Sys.setenv(SPARK_HOME=tools::file_path_as_absolute("..")); render("pkg/vignettes/sparkr-vignettes.Rmd"); .libPaths(paths)'
66+
67+
find pkg/vignettes/. -not -name '.' -not -name '*.Rmd' -not -name '*.md' -not -name '*.pdf' -not -name '*.html' -delete
68+
else
69+
echo "Skipping R vignettes as Spark JARs not found in $SPARK_HOME"
70+
fi
71+
4672
popd

‎R/pkg/NAMESPACE‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
importFrom("methods", "setGeneric", "setMethod", "setOldClass")
44
importFrom("methods", "is", "new", "signature", "show")
55
importFrom("stats", "gaussian", "setNames")
6-
importFrom("utils", "download.file", "packageVersion", "untar")
6+
importFrom("utils", "download.file", "object.size", "packageVersion", "untar")
77

88
# Disable native libraries till we figure out how to package it
99
# See SPARKR-7839
@@ -71,6 +71,7 @@ exportMethods("arrange",
7171
"covar_samp",
7272
"covar_pop",
7373
"createOrReplaceTempView",
74+
"crossJoin",
7475
"crosstab",
7576
"dapply",
7677
"dapplyCollect",
@@ -336,6 +337,9 @@ export("as.DataFrame",
336337
"read.parquet",
337338
"read.text",
338339
"spark.lapply",
340+
"spark.addFile",
341+
"spark.getSparkFilesRootDirectory",
342+
"spark.getSparkFiles",
339343
"sql",
340344
"str",
341345
"tableToDF",

0 commit comments

Comments
 (0)