Add Spark example using ask-and-tell interface#328
Conversation
Summary
This ensures the example follows code style guidelines and includes required dependencies ( Let me know if anything else should be added or refined! |
nabenabe0928
left a comment
There was a problem hiding this comment.
I cannot make your code work.
Could you please modify your code so that your code works without any modifications?
For example, an obvious bug is optuna.seed, which does not exist.
|
Thanks for pointing that out! I've removed the invalid |
|
Hi @nabenabe0928, just following up on this PR — I made the changes you suggested a few days ago (removed the invalid Appreciate your time and feedback! 🙇♂️ |
|
This pull request has not seen any recent activity. |
Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com>
Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com>
|
Thank you! I've applied your suggested changes |
nabenabe0928
left a comment
There was a problem hiding this comment.
I missed one thing, could you please reduce the indents?
Please see my comment for more details.
Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com>
|
Thank you for catching that! I've committed the suggested indentation fix. |
|
This pull request has not seen any recent activity. |
|
Thank you for catching that! I've committed the suggested indentation fix. |
|
This pull request has not seen any recent activity. |
nabenabe0928
left a comment
There was a problem hiding this comment.
Sorry for the late response, LGTM!
Overview
This PR adds an example that demonstrates how to use Optuna's ask-and-tell interface with Apache Spark to distribute trial evaluations in parallel using PySpark's RDD API.
The script defines a simple quadratic objective function and evaluates it across Spark worker nodes. It shows how ask-and-tell can be used with a distributed engine without requiring integration with external orchestrators.
Motivation
The example is inspired by multiple user questions regarding Spark integration, as discussed in optuna/optuna#6077. While the functionality already exists via
ask()andtell(), this example provides a hands-on script for users working with Spark.Notes
spark/directory.Please let me know if you'd like the example adjusted further.