Synthetic Data + Model

[jack89roberts]

Implement experiments with a synthetic dataset + model for a baseline/quick way to vary a lot of parameters.

E.g. something like

Data

from random import random

n = 1000
imbalance = 0.1
labels = [1 if random() < imbalance else 0 for _ in range(n)]

Model

pos_err_rate = 0.1
neg_err_rate = 0.2

err_rate = pos_err_rate if label == 1 else neg_err_rate
pred = label if random() > err_rate else 1 - label

Need to figure out:

• Predicting model scores not only labels.
• Systematic bias in errors, e.g. model is correct only when a certain feature is present/absent.