This PR mainly closes #488. It also closes #524 and closes #584.

New VotingLearner class

Relevant files: learner/voting.py and learner/utils.py.

• The main contribution of this PR is to allow SKLL to use VotingClassifier and VotingRegressor learners from scikit-learn. This was not as straightforward since this is the first true meta-learner class we have added to SKLL. By meta-learner, I mean a class that builds on top of SKLL's Learner class. This meta-learner class is called VotingLearner.

• The implementation of VotingLearner provides the same 7 methods as in the original Learner class: train(), cross_validate(), evaluate(), predict(), learning_curve(), from_file() and save(). There is some minor code duplication between these methods and the corresponding methods in the Learner class but most of it was avoided by the refactoring of the common code into utility functions last year. There are some changes to those functions in order to accommodate the new meta-learner. In addition, there is some new refactoring primarily for the from_file() and save() methods for the two classes which now use the refactored functions _save_learner_to_disk() and _load_learner_from_disk().

• The implementation supports passing in keyword arguments to the underlying learners, as well as samplers and sampler arguments.

• The train() implementation nicely incorporates grid search such that the learners underlying the voting meta-learner are automatically tuned with grid search (assuming the user requests it) before their predictions are used for voting. Grid search also works with cross_validate() although, as expected, doing this is much slower since the per-fold-grid-search will now be done for each underlying learner. Both these methods also accept a list of parameter grids for tuning the underlying learners.

• The evaluate() and predict() methods support returning (and writing out) not only the meta-learner's predictions but also the predictions from the underlying learners that were used in the voting process.

Integration with run_experiment

Relevant files: config/__init__.py, experiments/__init__.py, and experiments/utils.py.

• The VotingLearner class can be used via the SKLL API as described in the previous section. This PR also includes hooks that allow users to specify VotingClassifier and VotingRegressor as their chosen learners in an experiment configuration file as input to run_experiment. The hooks are set up to require users to specify the underlying estimators as fixed parameters using the estimator_names key. The following additional fixed parameters can also be specified: voting_type, estimator_fixed_parameters, estimator_samplers, estimator_sampler_parameters, and estimator_param_grids. These parameters are fully documented and example configuration files are also included (see documentation section below for details).

• A new configuration field called save_votes is added to allow the user to save the predictions from the underlying learners in addition to the predictions from the VotingClassifier or VotingRegressor. The default value for this field is False.

• A new JSON encoder called PipelineTypeEncoder() was added so as to support being able to serialize VotingLearner instances to JSON. This was necessary since these instances in turn contain Pipeline instances.

New tests

Relevant files: tests/test_voting_learners_api_*.py, tests/test_voting_learners_expts_*.py, and tests/utils.py.

• The artificially constructed datasets we use for the existing tests are not very useful for VotingLearner tests since they are either too small or too toy-ish. Therefore, we use the digits and housing datasets (included in scikit-learn) for classification and regression tests respectively. To make it easy to use these datasets, two new utility functions are added: make_digits_data() and make_california_housing_data().

• Note that we were already using a version of the digits dataset in the learning curve tests for the Learner class. This use was refactored to use the new make_digits_data() utility function.

• All of the API tests test the methods of the VotingLearner class by first calling that method on an instantiated VotingLearner in SKLL space, then using only scikit-learn functions to perform (nearly) identical
  operations, and then comparing the two results to make sure they are as close as possible. Most of the classification tests only those results up to 2 decimal places because there are some inherent differences between scikit-learn and SKLL that make it difficult to replicate SKLL operations in scikit-learn space. Most of the regression tests are able to compare to more decimal places since there are no probabilities involved.

• Since the API tests are so comprehensive, there is no real reason for the experiment tests to run real experiments since the same API methods are called from within run_configuration() anyway. For this reason, we focus all of the experiment tests on making sure that the right methods are called based on the "task" value and that they are called with the right arguments derived from the fields specified in the configuration file. To do so, we mock the appropriate API methods, call run_configuration() on different configuration files, and check that the mocked methods were called the expected number of times and with the expected arguments.

• To make experiment testing easier, we add a new utility function called fill_in_config_options_for_voting_learners() that takes an empty configuration file template (tests/configs/test_voting_learner.template.cfg ) and populates it with the right values depending on the arguments with which the function was called. In addition, a new class called BoolDictis added that returns ``False`` as the default value for key lookups rather than ``None``. This simplifies thefill_in_config_options_for_voting_learners()` function significantly.

• New tests were added to test_input.py for the save_votes field. In addition, existing tests in the same file were updated to accommodate this new field.

• Note that there is still some code duplication between the tests but this adding them all into the same file will add a lot more complexity (if statements and the like). This way they are fully self-contained and can be run fully independently.

Documentation

Relevant files: doc/run_experiment.rst, doc/api/learner.rst, examples/iris/voting.cfg, examples/boston/voting.cfg, and others (see below).

• Updated run_experiment documentation to include detailed description of VotingClassifier and VotingRegressor and added an entry for the save_votes configuration field. Note that some of the links will only work after the PR is merged.
• Updated doc/api/learner.rst to include the VotingLearner class and improved sub-headings.
• Added new configuration files illustrating the use of VotingClassifier and VotingRegressor for the Iris and Boston examples respectively.
• Updated doc/contributing.rst page for readability and fixed links to existing methods. Note that some of the links will only work after the PR is merged.
• Since top-level imports have been removed, deleted api/skll.rst and updated api/quickstart.rst and the Tutorial notebook.
• Updated doc/conf.py to fix imports and changed year from 2019 to 2021.

Other changes

• There was a major bug when using samplers. We were calling fit_transform() on the test set rather than transform(). This was fixed and tests/test_classification.py:test_sampler:test_sparse_predict_sampler() was updated.

• A new parallel build job was added to both Travis and Azure CI builds to accommodate the new tests. Test files were redistributed across all 6 jobs to make sure that the overall build time is still optimized.

• The Travis CI configuration now creates a new conda environment rather than using the default miniconda one. In addition, it does not activate said environment when running the tests. Finally, it also configures nosetests via environment variables for simplicity.

• Both Azure CI and Travis CI now set logging level for tests to WARNING and do not use --nologcapture which significantly reduces the size of the logs produced.

• The warning and error messages in Learner.learning_curve() have been tweaked to be more concise.