Reward models and arm priors

Reward models describe observation noise conditional on true arm means. Arm priors generate those means. Keeping the concepts separate avoids interpreting one generic scale as a Gaussian standard deviation in one place and a Beta concentration in another.

Available reward models are GaussianReward, BernoulliReward, and UniformReward. Available priors are GaussianArmPrior, BetaArmPrior, and UniformArmPrior.

Reward models and explicit priors for simulated arm means.

class ArmPrior[source]

Bases: ABC

Distribution used only to generate initial true arm means.

abstractmethod sample(*, n_arms, rng)[source]

Generate one true mean per arm.

Parameters:

n_arms (int)
rng (Generator)

Return type:

ndarray[tuple[Any, …], dtype[float64]]

class BernoulliReward[source]

Bases: RewardModel

Binary observations whose means are success probabilities.

domain: RewardDomain = 'binary'

validate_means(means)[source]

Validate true arm means against this model’s support.

Parameters:: means (ndarray[tuple[Any, ...], dtype[float64]])
Return type:: None

sample(means, rng)[source]

Sample one potential reward for every supplied arm mean.

Parameters:

means (ndarray[tuple[Any, ...], dtype[float64]])
rng (Generator)

Return type:

ndarray[tuple[Any, …], dtype[float64]]

__init__()

Return type:: None

clone()

Return an independent reward-model instance for one replicate.

Return type:: RewardModel

sample_one(mean, *, rng)

Sample a scalar reward for one arm using an explicit random stream.

Parameters:

mean (float)
rng (Generator)

Return type:

float

class BetaArmPrior[source]

Bases: ArmPrior

BetaArmPrior(alpha: ‘float’ = 1.0, beta: ‘float’ = 1.0)

alpha: float = 1.0

beta: float = 1.0

sample(*, n_arms, rng)[source]

Generate one true mean per arm.

Parameters:

n_arms (int)
rng (Generator)

Return type:

ndarray[tuple[Any, …], dtype[float64]]

__init__(alpha=1.0, beta=1.0)

Parameters:

alpha (float)
beta (float)

Return type:

None

class GaussianArmPrior[source]

Bases: ArmPrior

GaussianArmPrior(mean: ‘float’ = 0.0, std: ‘float’ = 1.0)

mean: float = 0.0

std: float = 1.0

sample(*, n_arms, rng)[source]

Generate one true mean per arm.

Parameters:

n_arms (int)
rng (Generator)

Return type:

ndarray[tuple[Any, …], dtype[float64]]

__init__(mean=0.0, std=1.0)

Parameters:

mean (float)
std (float)

Return type:

None

class GaussianReward[source]

Bases: RewardModel

Gaussian observations with known standard deviation.

std: float = 1.0

domain: RewardDomain = 'real'

sample(means, rng)[source]

Sample one potential reward for every supplied arm mean.

Parameters:

means (ndarray[tuple[Any, ...], dtype[float64]])
rng (Generator)

Return type:

ndarray[tuple[Any, …], dtype[float64]]

__init__(std=1.0)

Parameters:: std (float)
Return type:: None

clone()

Return an independent reward-model instance for one replicate.

Return type:: RewardModel

sample_one(mean, *, rng)

Sample a scalar reward for one arm using an explicit random stream.

Parameters:

mean (float)
rng (Generator)

Return type:

float

validate_means(means)

Validate true arm means against this model’s support.

Parameters:: means (ndarray[tuple[Any, ...], dtype[float64]])
Return type:: None

class RewardModel[source]

Bases: ABC

Distribution of observed rewards conditional on their true means.

domain: RewardDomain

abstractmethod sample(means, rng)[source]

Sample one potential reward for every supplied arm mean.

Parameters:

means (ndarray[tuple[Any, ...], dtype[float64]])
rng (Generator)

Return type:

ndarray[tuple[Any, …], dtype[float64]]

sample_one(mean, *, rng)[source]

Sample a scalar reward for one arm using an explicit random stream.

Parameters:

mean (float)
rng (Generator)

Return type:

float

clone()[source]

Return an independent reward-model instance for one replicate.

Return type:: RewardModel

validate_means(means)[source]

Validate true arm means against this model’s support.

Parameters:: means (ndarray[tuple[Any, ...], dtype[float64]])
Return type:: None

class UniformArmPrior[source]

Bases: ArmPrior

UniformArmPrior(low: ‘float’ = 0.0, high: ‘float’ = 1.0)

low: float = 0.0

high: float = 1.0

sample(*, n_arms, rng)[source]

Generate one true mean per arm.

Parameters:

n_arms (int)
rng (Generator)

Return type:

ndarray[tuple[Any, …], dtype[float64]]

__init__(low=0.0, high=1.0)

Parameters:

low (float)
high (float)

Return type:

None

class UniformReward[source]

Bases: RewardModel

Uniform observations centered on each true arm mean.

half_width: float = 1.0

domain: RewardDomain = 'real'

sample(means, rng)[source]

Sample one potential reward for every supplied arm mean.

Parameters:

means (ndarray[tuple[Any, ...], dtype[float64]])
rng (Generator)

Return type:

ndarray[tuple[Any, …], dtype[float64]]

__init__(half_width=1.0)

Parameters:: half_width (float)
Return type:: None

clone()

Return an independent reward-model instance for one replicate.

Return type:: RewardModel

sample_one(mean, *, rng)

Sample a scalar reward for one arm using an explicit random stream.

Parameters:

mean (float)
rng (Generator)

Return type:

float

validate_means(means)

Validate true arm means against this model’s support.

Parameters:: means (ndarray[tuple[Any, ...], dtype[float64]])
Return type:: None

resolve_reward_model(model, *, observation_scale=1.0)[source]

Resolve a short name, class, or instance into a reward model.

Parameters:

model (str | RewardModel | type[RewardModel])
observation_scale (float)

Return type:

RewardModel