Module topicnet.cooking_machine.models.frozen_score

Classes

class FrozenScore (value: List[Union[float, NoneType]], original_score: BaseScore = None)

Custom scores can have anything inside. So there is a probability that pickle will not be able to dump them. Frozen score helps to store the value of the original score without its internal logic, so as it can be saved.

Parameters

name

Name of the score

should_compute

Function which decides whether the score should be computed on the current fit iteration or not. If should_compute is None, then score is going to be computed on every iteration. At the same time, whatever function one defines, score is always computed on the last fit iteration. This is done for two reasons. Firstly, so that the score is always computed at least once during model._fit(). Secondly, so that experiment.select() works correctly.

The parameter should_compute might be helpful if the score is slow but one still needs to get the dependence of the score on iteration (for the described case, one may compute the score on every even iteration or somehow else). However, be aware that if should_compute is used for some model's scores, then the scores may have different number of values in model.scores! Number of score values is the number of times the scores was calculated; first value corresponds to the first fit iteration which passed should_compute etc.

There are a couple of things also worth noting. Fit iteration numbering starts from zero. And every new model._fit() call is a new range of fit iterations.

Examples

Scores created below are unworkable (as BaseScore has no call method inplemented). These are just the examples of how one can create a score and set some of its parameters.

Scores to be computed on every iteration:

>>> score = BaseScore()
>>> score = BaseScore(should_compute=BaseScore.compute_always)
>>> score = BaseScore(should_compute=lambda i: True)
>>> score = BaseScore(should_compute=True)

Scores to be computed only on the last iteration:

>>> score = BaseScore(should_compute=BaseScore.compute_on_last)
>>> score = BaseScore(should_compute=lambda i: False)
>>> score = BaseScore(should_compute=False)

Score to be computed only on even iterations:

>>> score = BaseScore(should_compute=lambda i: i % 2 == 0)

Expand source code

class FrozenScore(BaseScore):
    """
    Custom scores can have anything inside.
    So there is a probability that pickle will not be able to dump them.
    Frozen score helps to store the value of the original score without its internal logic,
    so as it can be saved.
    """
    def __init__(self, value: List[Optional[float]], original_score: BaseScore = None):
        super().__init__()

        self.value = value
        self._original_score: BaseScore = None

        if original_score is not None:
            self._save_original(original_score)

    def __repr__(self):
        return f'{self.__class__.__name__}(original_score={self._original_score!r})'

    def __getattr__(self, attribute_name):
        if attribute_name.startswith('__'):
            raise AttributeError()

        if attribute_name == '_original_score':  # some dill-loading stuff?
            raise AttributeError()

        if self._original_score is not None and hasattr(self._original_score, attribute_name):
            return getattr(self._original_score, attribute_name)

        raise AttributeError(
            f'Frozen score doesn\'t have such attribute: "{attribute_name}"'
        )

    def update(self, score_value: float) -> None:
        """
        Update is not supposed to be applied to Frozen score.
        It is not supposed to be changed.
        Still, the situation with an endeavour to update can generally happen if one tries
        to train the model further after loading.
        """
        warnings.warn(
            f'Trying to update Frozen score! Update value "{score_value}". '
            f'Frozen score is not supposed to be updated, '
            f'as there is no computation logic inside'
        )

        if score_value is not None:
            # TODO: it shouldn't be possible to pass such score_value value to update()
            #  other than the one returned by self.call()
            warnings.warn(
                f'Can\'t update Frozen score with value other than None: "{score_value}"!'
                f' Saving None score'
            )

        self.value.append(None)

    def call(self, model, *args, **kwargs) -> Optional[float]:
        return None

    def _save_original(self, original_score: BaseScore) -> None:
        field_types_for_saving = (Number, str, bool, Enum)
        self._original_score = BaseScore()

        for field_name in dir(original_score):
            field_value = getattr(original_score, field_name)

            if field_value is not None and not isinstance(field_value, field_types_for_saving):
                continue

            try:
                setattr(self._original_score, field_name, field_value)
            except AttributeError:
                # TODO: log?
                pass

        self._name = self._original_score._name

Ancestors

• BaseScore

Methods

def update(self, score_value: float) ‑> NoneType

Update is not supposed to be applied to Frozen score. It is not supposed to be changed. Still, the situation with an endeavour to update can generally happen if one tries to train the model further after loading.

Expand source code

def update(self, score_value: float) -> None:
    """
    Update is not supposed to be applied to Frozen score.
    It is not supposed to be changed.
    Still, the situation with an endeavour to update can generally happen if one tries
    to train the model further after loading.
    """
    warnings.warn(
        f'Trying to update Frozen score! Update value "{score_value}". '
        f'Frozen score is not supposed to be updated, '
        f'as there is no computation logic inside'
    )

    if score_value is not None:
        # TODO: it shouldn't be possible to pass such score_value value to update()
        #  other than the one returned by self.call()
        warnings.warn(
            f'Can\'t update Frozen score with value other than None: "{score_value}"!'
            f' Saving None score'
        )

    self.value.append(None)

Inherited members

• BaseScore:
  • call