Benchmarking and inference

compare runs all policies inside one paired experiment. Its independent unit is the replicate, not an individual time step.

import numpy as np

from pymab import BanditEnvironment, ExperimentConfig, compare
from pymab.benchmarking import BenchmarkConfig
from pymab.policies import RandomPolicy, UCBPolicy
from pymab.statistics import BootstrapConfig

benchmark = compare(
    {"random": RandomPolicy(n_arms=3), "ucb": UCBPolicy(n_arms=3)},
    environment=BanditEnvironment(means=np.array([0.1, 0.3, 0.8])),
    config=ExperimentConfig(horizon=20, n_replicates=4, seed=7),
    analysis=BenchmarkConfig(
        bootstrap=BootstrapConfig(n_resamples=200, seed=91),
        baseline="random",
    ),
)

assert benchmark.lowest_mean_regret_policy in {"random", "ucb"}
assert len(benchmark.summary()) == 2
assert len(benchmark.compare_to_baseline()) == 1

summary returns immutable PolicySummary records containing typed IntervalEstimate values for cumulative expected regret, realized total reward, optimal-action rate, and final simple regret. compare_to_baseline returns PolicyComparison records and bootstraps paired replicate differences, preserving the common-random-number design. to_dict is the flat JSON-compatible boundary.

lowest_mean_regret_policy is only a point-estimate ranking. Use paired intervals, effect size, domain relevance, and adequate independent replication before making a substantive claim.

Persistence and tables

SimulationResult.save_npz stores arrays plus versioned JSON metadata. Both NPZ and JSON formats have matching load methods. to_pandas includes policy IDs, replicate numbers, and replicate seeds on every tidy row.

Paired, replicate-aware policy benchmarking.

class BenchmarkConfig[source]

Bases: object

Configuration for replicate-level benchmark analysis.

bootstrap: BootstrapConfig
baseline: str | None = None
to_dict()[source]

Return a JSON-compatible analysis configuration.

Return type:

dict[str, object]

__init__(bootstrap=<factory>, baseline=None)
Parameters:
Return type:

None

class BenchmarkResult[source]

Bases: object

A simulation plus deterministic replicate-level uncertainty analysis.

result: SimulationResult
config: BenchmarkConfig
property policy_ids: tuple[str, ...]
property lowest_mean_regret_policy: str

Policy with the lowest final cumulative-regret point estimate.

summary()[source]

Return typed point estimates and bootstrap intervals by policy.

Return type:

tuple[PolicySummary, …]

compare_to_baseline(baseline=None)[source]

Return paired replicate differences relative to a baseline policy.

Parameters:

baseline (str | None)

Return type:

tuple[PolicyComparison, …]

to_pandas()[source]

Return summary rows as a pandas DataFrame.

Return type:

Any

to_dict()[source]

Return the analysis and simulation as a JSON-compatible record.

Return type:

dict[str, object]

plot_average_reward(*, output_path=None, show=False)[source]

Plot replicate-aware average reward curves.

Parameters:
Return type:

Any

plot_cumulative_regret(*, output_path=None, show=False)[source]

Plot replicate-aware cumulative regret curves.

Parameters:
Return type:

Any

plot_optimal_action_rate(*, output_path=None, show=False)[source]

Plot replicate-aware optimal-action rates.

Parameters:
Return type:

Any

__init__(result, config=<factory>)
Parameters:
Return type:

None

class PolicyComparison[source]

Bases: object

Typed paired metric differences relative to a baseline policy.

policy_id: str
baseline_id: str
cumulative_regret: IntervalEstimate
total_reward: IntervalEstimate
optimal_action_rate: IntervalEstimate
final_simple_regret: IntervalEstimate
property n_replicates: int

Number of paired simulation replicates in each estimate.

to_dict()[source]

Return a flat JSON-compatible comparison row.

Return type:

dict[str, str | int | float | None]

__init__(policy_id, baseline_id, cumulative_regret, total_reward, optimal_action_rate, final_simple_regret)
Parameters:
Return type:

None

class PolicySummary[source]

Bases: object

Typed uncertainty estimates for one policy’s benchmark metrics.

policy_index: int
policy_id: str
cumulative_regret: IntervalEstimate
total_reward: IntervalEstimate
optimal_action_rate: IntervalEstimate
final_simple_regret: IntervalEstimate
property n_replicates: int

Number of independent simulation replicates in each estimate.

to_dict()[source]

Return a flat JSON-compatible summary row.

Return type:

dict[str, str | int | float | None]

__init__(policy_index, policy_id, cumulative_regret, total_reward, optimal_action_rate, final_simple_regret)
Parameters:
Return type:

None

compare(policies, *, environment, config, analysis=None)[source]

Run one paired experiment and summarize independent replicates.

Parameters:
Return type:

BenchmarkResult

Plot uncertainty bands use BootstrapConfig and a bounded-memory, replicate-level bootstrap. Benchmark plot wrappers inherit the benchmark’s analysis configuration.

Optional plotting helpers for simulation results.

plot_average_reward(result, *, output_path=None, show=False, band_config=None)[source]

Plot average reward by step with Plotly when installed.

Parameters:
Return type:

Any

plot_cumulative_regret(result, *, output_path=None, show=False, band_config=None)[source]

Plot cumulative expected regret by step with Plotly when installed.

Parameters:
Return type:

Any

plot_optimal_action_rate(result, *, output_path=None, show=False, band_config=None)[source]

Plot the optimal-action selection rate by step with Plotly.

Parameters:
Return type:

Any