User Experience Optimization Experiment Simulations

<p># The `uxo_sim` Package</p> <p>A package for simulations of data matching industry UX optimization experiments, as discussed in:</p> <p>```<br> @article{van_adelsberg_modeling_2019,<br> &nbsp;&nbsp; &nbsp;title = {Modeling {A}/{B} {Test} {Data} is {Hard}: {Effects} of {Overdispersion}, {RandomWalks}, and {Cointegration}},<br> &nbsp;&nbsp; &nbsp;language = {en},<br> &nbsp;&nbsp; &nbsp;journal = {NeurIPS 2019 Workshop on Robust &nbsp;AI in Financial Services: Data, Fairness, Explainability, Trustworthiness, and Privacy},<br> &nbsp;&nbsp; &nbsp;author = {van Adelsberg, Matthew and Sweeney, Mackenzie},<br> &nbsp;&nbsp; &nbsp;month = dec,<br> &nbsp;&nbsp; &nbsp;year = {2019}<br> }<br> ```</p> <p>The code for running the simulations is included, along with figures and CSV files for each of three specific simulation runs that are used in a publication currently under review for ICML 2020.</p> <p>## Packages:</p> <p>1. `data`: code for running the simulations to produce datasets<br> 2. `viz`: code for visualizing the simulation outputs</p> <p>## Scripts:</p> <p>1. `save_datasets`: save CSV for each simulated dataset in the `inventory`<br> 2. `save_figs`: save PNG figure for each simulated dataset in `plots`</p> <p>## Simulation Datasets:</p> <p>### `fixed_effects_od_20_21_seed27`</p> <p>Data is simulated from a beta-binomial distribution with overdispersion parameter `gamma=0.01` for each of the two treatments and rates `theta=0.20` and `theta=0.21`. This corresponds to beta distribution parameters `alpha, beta = 19.8, 79.2` and `alpha, beta = 20.79, 78.21`.</p> <p>### `drift_down_then_up`</p> <p>Data is simulated from a beta-binomial distribution with overdispersion parameter `gamma=0.01` for each of the two treatments. The rates start at `theta=0.20` and `theta=0.21` and then:</p> <p>1. decrease by 0.005 each day for 20 days<br> 2. increase by 0.005 each day for 30 days<br> 3. stay constant for 10 days</p> <p>The corresponding beta distribution parameters on each day are a function of `theta, gamma` and can be obtained via this function (implemented in `uxo_sims.data.simulations`:<br> ```python<br> def alpha_beta_from_gamma_theta(gamma, theta):<br> &nbsp; &nbsp; virtual_sample_size = 1 / gamma - 1<br> &nbsp; &nbsp; alpha = theta * virtual_sample_size<br> &nbsp; &nbsp; beta = virtual_sample_size - alpha<br> &nbsp; &nbsp; return alpha, beta<br> ```</p> <p><br> ### `arm_addition`</p> <p>Data is simulated from a beta-binomial distribution with overdispersion parameter `gamma=0.01` for each of the two treatments. The rates start at `theta=0.10` and `theta=0.11` and increase by 0.005 each day for 40 days. The corresponding beta distribution parameters can be obtained with the same function as indicated in `drift_down_then_up`.</p>