Update ens builder (#1434)

* Move ensemble_bulder test data to named folder

* Update backend to take a temlate to copy from

* Update tests to use new cases system

* Update tests to be documented and cleaned up

* Switch to using cached automl backends

* Readd missing file which failed test for `case_3_models`

* Seperate out tests that rely on old toy data and those that don't

* Setup test framework for ensemble builder on real situations

* Formatting

* Remove `unit_test` arg

* Remove SAVE2DISC

* Split builder and manager into seperate files

* Tidy up init of EnsembleBuilder

* Moved to cached properties

* Change List to list

* Move to solely using cached properties

* Add disk util file with `sizeof`

* Update tests to use cached mechanism

* Switch `sizeof` for disk consumption

* Remove disk consumption

* Remove unneeded function

* Add type hints and documenation

* Simplyify _read_np_fn

* Update get_valid_test_preds to use Pathlib

* Add intersection to functional

* Make functional take *args

* Further simplifications

* Add a dataclass to represent run information for builder

* Rename to Run

* Change to Run objects

* Formatting

* Reduce side effects of `compute_loss_per_model`

To make testing easier and changes easier, the targets are now passed to
the method. This also reduces it's complexity by removing the checking
from the method as we can assume the parameters coming in are correct.

* Change Tuple to tuple

* Forcibly add data files for tests

* Fix: Can now load pickled numpy arrays w/ test

* Add test for checking ensemble builder output

* Fix bug with using list instead of set

* Making deubgging message a little clearer

* Fix typing and case name

* Rename test file to reflect what it tests

* Make pynisher context optional

* Fix loaded models test

* Updates to Run dataclass

* Add method to `Run` to allow recording of last modified

* Change Run mtimes to dictionary

* Change `compute_loss_per_model` to use new Run dataclass

* Factor out run loss into main loop

* Simplyify get_nbest and compute_losses

* Major rewrite of ensemble builder main loop

* Change to simpler hashing

* Start value split

* Add `value_split`

* Reworked Builder

* Add some docstring

* Formatting

* Fix type signature

* Fix typing for `loss`

* Removed Literal

* Mypy fixes for ensemble builder

* Mypy fixes

* Tests for `Runs`

* Move `make_run` to fixtures

* Fix run deletion

* Test candidates

* Made delete it's own function

* Further simplifications

* Fixup test with simplification

* Test: `max_models` for `requires_deletion`

* Test: `memory_limit` for `requires_deletion`

* Test: Loss of runs

* Test: Delete runs

* Test: `fit_ensemble` of ensemble builder

* Add test for run time parameter

* Remove parameter `return_predictions`

* Add note about pickled arrays should not be supported

* Make cached automl instances copy backend

* Add valid static method to run

* Remove old test data

* Add filter for bad run dirs

* Made `main` args optional

* Fix check for updated runs

* Make `main` raise errors

* Fix default value for ensemble builder `main`

* Test valid ensemble with real runs

* Rename parameter for manager

* Add defaults and reorder parameters for EnsembleBuilderManager

* Fixup parameters in `fit_and_return_ensemble`

* Typing fixes

* Make `fit_and_return_ensemble` a staticmethod

* Add: `make_ensemble_builder_manager`

* Add: Test files for manager

* Add atomic rmtree

* Add: atomic rmtree now accepts where mv should go

* Make builder use atomic rmtree

* Fix import bugs, remove valid preds in builder

* Remove `np.inf` as valid arg for `read_at_most`

* Possible reproducible num_run, no predictions error

* Make automl caching robust to `pytest-xdist`

* Test fixes

* Extend interval for test on run caching

* Use pickle for reseting cache

* Fix test for caching mechanism to not rely on `stat`

* Move run deletion to the end of the builder `main`

* Remove `getattr` version of tae.client

* Remove `normalize`

* Extend not for `Run`

* Fix `__init__` of `Run`

* Parameter and comment fixes from feedback

* Change to `min(...)` instead of `sorted(...)[0]`

* Make default time `np.inf`

* Add test for safe deletion in builder

* Update docstring of `loss` for a run

* Remove stray print

* Minor feedback fixes

* Fix `_metric` to `_metrics`

* Fix `make_ensemble_builder`

* One more fix for multiple metrics

Author: eddiebergman

autosklearn/automl.py

@@ -62,7 +62,7 @@
     convert_if_sparse,
 )
 from autosklearn.data.xy_data_manager import XYDataManager
-from autosklearn.ensemble_builder import EnsembleBuilderManager
+from autosklearn.ensemble_building import EnsembleBuilderManager
 from autosklearn.ensembles.singlebest_ensemble import SingleBest
 from autosklearn.evaluation import ExecuteTaFuncWithQueue, get_cost_of_crash
 from autosklearn.evaluation.abstract_evaluator import _fit_and_suppress_warnings
@@ -303,6 +303,8 @@ def __init__(
         self._label_num = None
         self._parser = None
         self._can_predict = False
+        self._read_at_most = None
+        self._max_ensemble_build_iterations = None
         self.models_: Optional[dict] = None
         self.cv_models_: Optional[dict] = None
         self.ensemble_ = None
@@ -808,9 +810,9 @@ def fit(
                     max_models_on_disc=self._max_models_on_disc,
                     seed=self._seed,
                     precision=self.precision,
-                    max_iterations=None,
-                    read_at_most=np.inf,
-                    ensemble_memory_limit=self._memory_limit,
+                    max_iterations=self._max_ensemble_build_iterations,
+                    read_at_most=self._read_at_most,
+                    memory_limit=self._memory_limit,
                     random_state=self._seed,
                     logger_port=self._logger_port,
                     pynisher_context=self._multiprocessing_context,
@@ -923,7 +925,7 @@ def fit(
                 )
                 result = proc_ensemble.futures.pop().result()
                 if result:
-                    ensemble_history, _, _, _, _ = result
+                    ensemble_history, _ = result
                     self.ensemble_performance_history.extend(ensemble_history)
                 self._logger.info("Ensemble script finished, continue shutdown.")
 
@@ -1524,8 +1526,8 @@ def fit_ensemble(
             seed=self._seed,
             precision=precision if precision else self.precision,
             max_iterations=1,
-            read_at_most=np.inf,
-            ensemble_memory_limit=self._memory_limit,
+            read_at_most=None,
+            memory_limit=self._memory_limit,
             random_state=self._seed,
             logger_port=self._logger_port,
             pynisher_context=self._multiprocessing_context,
@@ -1538,7 +1540,7 @@ def fit_ensemble(
                 "Error building the ensemble - please check the log file and command "
                 "line output for error messages."
             )
-        self.ensemble_performance_history, _, _, _, _ = result
+        self.ensemble_performance_history, _ = result
         self._ensemble_size = ensemble_size
 
         self._load_models()
@@ -2096,6 +2098,15 @@ def has_key(rv, key):
 
         return ensemble_dict
 
+    def has_ensemble(self) -> bool:
+        """
+        Returns
+        -------
+        bool
+            Whether this AutoML instance has an ensemble
+        """
+        return self.ensemble_ is not None
+
     def _create_search_space(
         self,
         tmp_dir: str,
@@ -2154,7 +2165,7 @@ def fit(
         y: SUPPORTED_TARGET_TYPES | spmatrix,
         X_test: Optional[SUPPORTED_FEAT_TYPES] = None,
         y_test: Optional[SUPPORTED_TARGET_TYPES | spmatrix] = None,
-        feat_type: Optional[list[bool]] = None,
+        feat_type: Optional[list[str]] = None,
         dataset_name: Optional[str] = None,
         only_return_configuration_space: bool = False,
         load_models: bool = True,
@@ -2244,7 +2255,7 @@ def fit(
         y: SUPPORTED_TARGET_TYPES | spmatrix,
         X_test: Optional[SUPPORTED_FEAT_TYPES] = None,
         y_test: Optional[SUPPORTED_TARGET_TYPES | spmatrix] = None,
-        feat_type: Optional[list[bool]] = None,
+        feat_type: Optional[list[str]] = None,
         dataset_name: Optional[str] = None,
         only_return_configuration_space: bool = False,
         load_models: bool = True,