Metrics

Metrics are components that track and store a given quantity over the duration of the simulation. All metrics in this module are based on the Measurement abstract class, which inherits from the BaseObservable abstract class (refer to the Observer design pattern for more details).

This module comes with a number of pre-loaded metrics, as well as a simple programmer interface to develop new metrics.

Interaction Measurement

class metrics.measurement.InteractionMeasurement(name='interaction_histogram', verbose=False)[source]

Keeps track of the interactions between users and items.

Specifically, at each timestep, it stores a histogram of length \(|I|\), where element \(i\) is the number of interactions received by item \(i\).

Parameters: verbose (bool, default False) – If True, enables verbose mode. Disabled by default.

Inherited by Measurement

Type: Measurement

name

Name of the measurement component.

Type: str, default "interaction_histogram"

measure(recommender)[source]

Measures and stores a histogram of the number of interactions per item at the given timestep.

Parameters: recommender (BaseRecommender) – Model that inherits from BaseRecommender.

Interaction Spread

class metrics.measurement.InteractionSpread(verbose=False)[source]

Measures the diversity of the interactions between users and items.

Specifically, at each timestep, it measures whether interactions are spread among many items or only a few items.

This class inherits from InteractionMeasurement.

Parameters: verbose (bool, default False) – If True, enables verbose mode. Disabled by default.

Inherited by InteractionMeasurement

Type: InteractionMeasurement

name

Name of the measurement component.

Type: str, default "interaction_spread"

_old_histogram

A copy of the histogram at the previous timestep.

Type: None, list, array_like

measure(recommender)[source]

Measures the diversity of user interactions – that is, whether interactions are spread among many items or only a few items.

Parameters: recommender (BaseRecommender) – Model that inherits from BaseRecommender.

RecSimilarity

class metrics.measurement.RecSimilarity(pairs, name='rec_similarity', verbose=False)[source]

Keeps track of the average Jaccard similarity between items seen by pairs of users at each timestep. The pairs of users must be passed in by the user.

Parameters

pairs (iterable of tuples) – Contains tuples representing each pair of users. Each user should be represented as an index into the user profiles matrix.
verbose (bool, default False) – If True, enables verbose mode. Disabled by default.

Inherited by Measurement

Type: Measurement

name

Name of the measurement component.

Type: str, default "rec_similarity"

measure(recommender)[source]

Measures the average Jaccard index of items shown to pairs of users in the system. Intuitively, a higher average Jaccard index corresponds to increasing “homogenization” in that the recommender system is starting to treat each user the same way (i.e., show them the same items).

Parameters: recommender (BaseRecommender) – Model that inherits from BaseRecommender.

InteractionSimilarity

class metrics.measurement.InteractionSimilarity(pairs, name='interaction_similarity', verbose=False, diagnostics=False, **kwargs)[source]

Keeps track of the average Jaccard similarity between interactions with items between pairs of users at each timestep. The pairs of users must be passed in by the user.

Parameters

pairs (iterable of tuples) – Contains tuples representing each pair of users. Each user should be represented as an index into the user profiles matrix.
verbose (bool, default False) – If True, enables verbose mode. Disabled by default.

Inherited by Measurement

Type: Measurement

name

Name of the measurement component.

Type: str, default "interaction_similarity"

measure(recommender)[source]

Measures the average Jaccard index of items that pairs of users have interacted with in the system. Intuitively, a higher average Jaccard index corresponds to increasing “homogenization” in that user behavior is becoming more and more similar (i.e., users have all interacted with the same items).

Parameters: recommender (BaseRecommender) – Model that inherits from BaseRecommender.

MSE Measurement

class metrics.measurement.MSEMeasurement(verbose=False, diagnostics=False, **kwargs)[source]

Measures the mean squared error (MSE) between real and predicted user scores.

It can be used to evaluate how accurate the model predictions are.

This class inherits from Measurement.

Parameters: verbose (bool, default False) – If True, enables verbose mode. Disabled by default.

Inherited by Measurement

Type: Measurement

name

Name of the measurement component.

Type: str (optional, default: “mse”)

measure(recommender)[source]

Measures and records the mean squared error between the user preferences predicted by the system and the users’ actual preferences.

Parameters: recommender (BaseRecommender) – Model that inherits from BaseRecommender.

Diffusion Tree

class metrics.measurement.DiffusionTreeMeasurement(verbose=False)[source]

Class that implements an information diffusion tree. The current implementation assumes that agents using this class (i.e., a model) implement an infection_state matrix that denotes the initial state of information.

In this implementation, the nodes represent users and are labeled with the user indices. A branch between nodes u and v indicates that user u passed information onto user v – that is, u “infected” v.

Trees are implemented using the Networkx library. Please refer to Networkx’s documentation for more details.

Parameters

infection_state (InfectionState) – The initial “infection state” of all users
verbose (bool, default False) – If True, enables verbose mode. Disabled by default.

Inherited by Measurement

Type: Measurement

name

Name of the metric that is recorded at each time step. Note that, in this case, the metric stored in measurement_history is actually the number of infected users. The diffusion tree itself is kept in the diffusion_tree data structure.

Type: str, default "num_infected"

diffusion_tree

Diffusion tree.

Type: networkx.Graph

_old_infection_state

Infection state at the previous timestep.

Type: array_like

draw_tree()[source]: Plots the tree using the Networkx library API.

measure(recommender)[source]

Updates tree with new infections and stores information about new infections. In measurement_history, it stores the total number of infected users in the system – that is, the number of nodes in the tree.

Parameters: recommender (BaseRecommender) – Model that inherits from BaseRecommender.

Measurement of Structural Virality

class metrics.measurement.StructuralVirality(verbose=False)[source]

This class extends DiffusionTreeMeasurement with the concept of structural virality developed by Goel, Anderson, Hofman, and Watts in The Structural Virality of Online Diffusion. It is used in BassModel.

get_structural_virality()[source]

Returns a measure of structural virality.

Returns: Structural virality
Return type: float

Average Feature Score Range

class metrics.measurement.AverageFeatureScoreRange(name='afsr', verbose=False)[source]

Measures the average range (across users) of item attributes for items users were recommended at a time step.

This metric is based on the item diversity measure used in :

Willemsen, M. C., Graus, M. P., & Knijnenburg, B. P. (2016). Understanding the role of latent feature diversification on choice difficulty and satisfaction. User Modeling and User-Adapted Interaction, 26(4), 347-389.

This class inherits from Measurement.

Parameters: verbose (bool, default False) – If True, enables verbose mode. Disabled by default.

Inherited by Measurement

Type: Measurement

name

Name of the measurement component.

Type: str, default "afsr"

measure(recommender)[source]

Measures the average range (across users) of item attributes for items users were recommended at a time step. Used as a measure of within list recommendation diversity

This metric is based on the item diversity measure used in : Willemsen, M. C., Graus, M. P., & Knijnenburg, B. P. (2016). Understanding the role of latent feature diversification on choice difficulty and satisfaction. User Modeling and User-Adapted Interaction, 26(4), 347-389.

Parameters: recommender (BaseRecommender) – Model that inherits from BaseRecommender.

Measurement: base class

class metrics.measurement.Measurement(name, verbose=False)[source]

Abstract observable class to store measurements.

Parameters: verbose (bool, default False) – If True, enables verbose mode. Disabled by default.

measurement_history

List of measurements. A new element is added at each timestep.

Type: list

name

Name of the measurement quantity.

Type: str

get_measurement()[source]

Returns measurements. See get_observable() for more details.

Returns: Measurements
Return type: dict

get_timesteps()[source]

Returns the number of measurements stored (which is equivalent to the number of timesteps that the system has been measuring).

Returns: Length of measurement_history
Return type: int

abstract measure(recommender)[source]: Function that should calculate some outcome of interest of the system at the current timestep

observe(observation, copy=True)[source]

Stores measurements. It can be called by implementations to ensure consistency when storing different measurements.

Parameters

observation (array_like or int or float or None) – Element that will be stored
copy (bool, default True) – If True, the function stores a copy of observation. Useful for numpy.ndarray.