Feature: Implementing Continuous NN Policy Learner

[usaito]

new features

• implement BaseContinuousOfflinePolicyLearner in policy/base.py
• implement ContinuousNNPolicyLearner in policy/offline_continuous.py, which trains a decision making policy modeled by a neural network using logged bandit data with continuous actions. This class works as follows.

# define a neural network policy
nn_policy = ContinuousNNPolicyLearner(
   dim_context=10,
   pg_method="dpg", # pg_method: policy gradient method
   hidden_layer_size=(10,10,10),
   activation="relu",
   solver="adam"
)

# train the policy on logged bandit feedback data
nn_policy.fit(
    context=bandit_feedback_train["context"],
    action=bandit_feedback_train["action"],
    reward=bandit_feedback_train["reward"],
    pscore=bandit_feedback_train["pscore"],
)

# predict new action values for the test data
predicted_actions = nn_policy.fit(
    context=bandit_feedback_test["context"],
)

where bandit_feedback is assumed to be generated by SyntheticContinuousBanditFeedback.

reference
Nathan Kallus and Masatoshi Uehara.
"Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies", NeurIPS2020.

tests

• test_offline_continuous.py checks the inputs of the functions of ContinuousNNPolicyLearner.
• test_offline_learner_performance.py checks whether ContinuousNNPolicyLearner can outperform the uniform random policy (which randomly samples continuous actions from the action space) on a simple synthetic setting.

refactor

• add some descriptions to test_offline.py and test_offline_learner_performance.py (just for clarification)
• fix typos in offline.py

[nomuramasahir0]

After confirming the above minor comments and resolve the conflict, LGTM!

[usaito]

@nmasahiro Thanks!