bex.loggers package

Module contents

class bex.loggers.BasicLogger(attributes, path, n_batches=5)

Bases: object

Basic logger class

Variables:

metrics – dictionary containing the metrics logged by Bex

This logger logs the metrics computed by the benchmark for a given explainer along with the run configuration and some images of successful counterfactuals. It mostly serves as a base class for custom loggers but it is fully functional.

Parameters:
  • attributes (Dict) – dictionary containing the run config

  • path – (string): output path for the logger (see run())

  • n_batches – (int, optional): max number of image batches to log

If you wish to test your own explainer on our benchmark use this as a base class and override the accumulate() and log() methods

Example

import wandb

class WandbLogger(BasicLogger):


    def __init__(self, attributes, path, n_batches=10):

        super().__init__(attributes, path, n_batches)

        wandb.init(project="Bex", dir=self.path, config=self.attributes, reinit=True)


    def accumulate(self, data, images):

        super().accumulate(data, images)

        wandb.log({f"{k}" :v for k, v in data.items()}, commit=True)


    def log(self):

        self.metrics = {f"{k}_avg": np.mean(v) for k, v in self.metrics.items()}
        wandb.log(self.metrics)

        # create matplotlib figure with the counterfactuals generated
        fig = self.create_cf_figure()

        wandb.log({"Counterfactuals": fig})
        plt.close()

bn = Benchmark()
bn.run("dive", logger=WandbLogger)
accumulate(data, images)

Updates the metric and image history

Parameters:
  • data (Dict) – dictionary containing the metrics compute by Bex

  • images (Dict) – dictionary containing tensor images from a given batch and their orthogonal counterfactuals

create_cf_figure()

Helper function to build a figure with the accumulated images. Must be called before log()

log()

Log the average for each of the metrics computed by Bex along with a figure with some of the orthogonal counterfactuals produced.

The images will the be logged to output_path (see run()) as a .png file while the metrics along with the run config will be logged as a .csv file