spell_error_correction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine tuning a Pretrained BERT for spell correction tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data download and setup\n",
    "\n",
    "We will be using a general spell correction dataset available [here](https://github.com/mhagiwara/xfspell/blob/master/data/fce/fce-split.us.tsv). This dataset is an induced spell errors using general english sentences. Download the file and store it in the local notebook instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "ScheckDF = pd.read_csv(\"xfspell/data/fce/fce-split.us.tsv\",sep=\"\\t\",header=None,names=[\"input_text\",\"target_text\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['I WANT TO THAK YOU FOR PREPARING SUCH A GOOD PROGRAMME FOR US AND ESPECIALLY FOR TAKING US ON THE RIVER TRIP TO GREENWICH.',\n",
       "        'I WANT TO THANK YOU FOR PREPARING SUCH A GOOD PROGRAMME FOR US AND ESPECIALLY FOR TAKING US ON THE RIVER TRIP TO GREENWICH.'],\n",
       "       ['I WOULD LIKE TO KNOW IF THERE IS ANY CHANCE OF CHANGING THE PROGRAMME BECAUSE WE HAVE FOUND A VERY INTERESTING ACTIVITY TO DO ON TUESDAY 14 MARCH.',\n",
       "        'I WOULD LIKE TO KNOW IF THERE IS ANY CHANCE OF CHANGING THE PROGRAMME BECAUSE WE HAVE FOUND A VERY INTERESTING ACTIVITY TO DO ON TUESDAY 14 MARCH.'],\n",
       "       ['IT INVOLVES VISITING THE LONDON FASHION AND LEISURE SHOW AT THE CENTRAL EXHIBITION HALL.',\n",
       "        'IT INVOLVES VISITING THE LONDON FASHION AND LEISURE SHOW AT THE CENTRAL EXHIBITION HALL.'],\n",
       "       [\"I THINK IT'S A GREAT OPPORTUNITY TO MAKE GREATER USE OF OUR KNOWLEDGE OF THE ENGLISH LANGUAGE.\",\n",
       "        \"I THINK IT'S A GREAT OPPORTUNITY TO MAKE GREATER USE OF OUR KNOWLEDGE OF THE ENGLISH LANGUAGE.\"],\n",
       "       ['ALSO, WE COULD LEARN THE DIFFERENT WAYS TO GET TO THE CENTRAL EXHIBITION HALL.',\n",
       "        'ALSO, WE COULD LEARN THE DIFFERENT WAYS TO GET TO THE CENTRAL EXHIBITION HALL.']],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ScheckDF.head().values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 8498 entries, 0 to 8499\n",
      "Data columns (total 2 columns):\n",
      " #   Column       Non-Null Count  Dtype \n",
      "---  ------       --------------  ----- \n",
      " 0   input_text   8498 non-null   object\n",
      " 1   target_text  8498 non-null   object\n",
      "dtypes: object(2)\n",
      "memory usage: 199.2+ KB\n"
     ]
    }
   ],
   "source": [
    "trainDF.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Divide the data into train and eval set and remove null values if any."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainDF = ScheckDF[:8500]\n",
    "evalDF = ScheckDF[8500:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainDF = trainDF.dropna(how='any',axis=0) \n",
    "evalDF = evalDF.dropna(how='any',axis=0) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"size of training data {}\".format(len(trainDF.index)))\n",
    "print(\"size of eval data {}\".format(len(evalDF.index)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation details\n",
    "\n",
    "The Idea is to use a Seq2Seq Architecture with both inputs and targets as text sequences. For our dataset we will use the incorrect sentences as inputs and the correct sentences as outputs. We will be further using a encoder-decoder architecture with fine tuning with Bert/Roberta for both encoder and decoder networks.\n",
    "\n",
    "We will be using a python library called \"simpletransformers\" which uses the \"Transformers\" library from hugging faces to build more high level usable API's for certain language tasks. More information can be found [here](https://simpletransformers.ai/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install the simple transformers library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting simpletransformers\n",
      "  Downloading simpletransformers-0.48.14-py3-none-any.whl (214 kB)\n",
      "\u001b[K     |████████████████████████████████| 214 kB 3.1 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting regex\n",
      "  Downloading regex-2020.10.15-cp36-cp36m-manylinux2010_x86_64.whl (662 kB)\n",
      "\u001b[K     |████████████████████████████████| 662 kB 7.7 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: requests in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from simpletransformers) (2.22.0)\n",
      "Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from simpletransformers) (1.0.1)\n",
      "Collecting tokenizers\n",
      "  Downloading tokenizers-0.9.2-cp36-cp36m-manylinux1_x86_64.whl (2.9 MB)\n",
      "\u001b[K     |████████████████████████████████| 2.9 MB 16.1 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting seqeval\n",
      "  Downloading seqeval-1.2.1.tar.gz (43 kB)\n",
      "\u001b[K     |████████████████████████████████| 43 kB 4.0 MB/s  eta 0:00:01\n",
      "\u001b[?25hCollecting wandb\n",
      "  Downloading wandb-0.10.7-py2.py3-none-any.whl (1.7 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.7 MB 32.5 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: numpy in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from simpletransformers) (1.18.1)\n",
      "Collecting transformers>=3.0.2\n",
      "  Downloading transformers-3.4.0-py3-none-any.whl (1.3 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.3 MB 43.5 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting streamlit\n",
      "  Downloading streamlit-0.69.2-py2.py3-none-any.whl (7.4 MB)\n",
      "\u001b[K     |████████████████████████████████| 7.4 MB 68.6 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: scikit-learn in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from simpletransformers) (0.22.1)\n",
      "Requirement already satisfied: scipy in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from simpletransformers) (1.4.1)\n",
      "Collecting tqdm>=4.47.0\n",
      "  Downloading tqdm-4.50.2-py2.py3-none-any.whl (70 kB)\n",
      "\u001b[K     |████████████████████████████████| 70 kB 16.4 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting tensorboardx\n",
      "  Downloading tensorboardX-2.1-py2.py3-none-any.whl (308 kB)\n",
      "\u001b[K     |████████████████████████████████| 308 kB 64.3 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from requests->simpletransformers) (2020.6.20)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from requests->simpletransformers) (2.8)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from requests->simpletransformers) (1.25.10)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from requests->simpletransformers) (3.0.4)\n",
      "Requirement already satisfied: pytz>=2017.2 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from pandas->simpletransformers) (2019.3)\n",
      "Requirement already satisfied: python-dateutil>=2.6.1 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from pandas->simpletransformers) (2.8.1)\n",
      "Requirement already satisfied: psutil>=5.0.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from wandb->simpletransformers) (5.6.7)\n",
      "Requirement already satisfied: six>=1.13.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from wandb->simpletransformers) (1.14.0)\n",
      "Collecting shortuuid>=0.5.0\n",
      "  Downloading shortuuid-1.0.1-py3-none-any.whl (7.5 kB)\n",
      "Requirement already satisfied: PyYAML in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from wandb->simpletransformers) (5.3.1)\n",
      "Collecting subprocess32>=3.5.3\n",
      "  Downloading subprocess32-3.5.4.tar.gz (97 kB)\n",
      "\u001b[K     |████████████████████████████████| 97 kB 12.4 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting promise<3,>=2.0\n",
      "  Downloading promise-2.3.tar.gz (19 kB)\n",
      "Collecting GitPython>=1.0.0\n",
      "  Using cached GitPython-3.1.9-py3-none-any.whl (159 kB)\n",
      "Requirement already satisfied: watchdog>=0.8.3 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from wandb->simpletransformers) (0.10.2)\n",
      "Collecting protobuf>=3.12.0\n",
      "  Downloading protobuf-3.13.0-cp36-cp36m-manylinux1_x86_64.whl (1.3 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.3 MB 69.4 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting docker-pycreds>=0.4.0\n",
      "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
      "Collecting sentry-sdk>=0.4.0\n",
      "  Downloading sentry_sdk-0.19.1-py2.py3-none-any.whl (120 kB)\n",
      "\u001b[K     |████████████████████████████████| 120 kB 59.9 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: Click>=7.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from wandb->simpletransformers) (7.0)\n",
      "Collecting configparser>=3.8.1\n",
      "  Downloading configparser-5.0.1-py3-none-any.whl (22 kB)\n",
      "Requirement already satisfied: packaging in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from transformers>=3.0.2->simpletransformers) (20.1)\n",
      "Requirement already satisfied: filelock in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from transformers>=3.0.2->simpletransformers) (3.0.12)\n",
      "Collecting sacremoses\n",
      "  Downloading sacremoses-0.0.43.tar.gz (883 kB)\n",
      "\u001b[K     |████████████████████████████████| 883 kB 63.0 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting sentencepiece!=0.1.92\n",
      "  Downloading sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.1 MB 62.6 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting dataclasses; python_version < \"3.7\"\n",
      "  Downloading dataclasses-0.7-py3-none-any.whl (18 kB)\n",
      "Collecting blinker\n",
      "  Downloading blinker-1.4.tar.gz (111 kB)\n",
      "\u001b[K     |████████████████████████████████| 111 kB 74.1 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: tornado>=5.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from streamlit->simpletransformers) (6.0.3)\n",
      "Collecting pyarrow\n",
      "  Downloading pyarrow-2.0.0-cp36-cp36m-manylinux2014_x86_64.whl (17.7 MB)\n",
      "\u001b[K     |████████████████████████████████| 17.7 MB 70.7 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting pydeck>=0.1.dev5\n",
      "  Downloading pydeck-0.5.0b1-py2.py3-none-any.whl (4.4 MB)\n",
      "\u001b[K     |████████████████████████████████| 4.4 MB 23.5 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting astor\n",
      "  Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB)\n",
      "Requirement already satisfied: botocore>=1.13.44 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from streamlit->simpletransformers) (1.18.16)\n",
      "Collecting altair>=3.2.0\n",
      "  Downloading altair-4.1.0-py3-none-any.whl (727 kB)\n",
      "\u001b[K     |████████████████████████████████| 727 kB 55.4 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: boto3 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from streamlit->simpletransformers) (1.15.16)\n",
      "Requirement already satisfied: toml in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from streamlit->simpletransformers) (0.10.1)\n",
      "Requirement already satisfied: pillow>=6.2.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from streamlit->simpletransformers) (7.0.0)\n",
      "Collecting enum-compat\n",
      "  Downloading enum_compat-0.0.3-py3-none-any.whl (1.3 kB)\n",
      "Collecting base58\n",
      "  Downloading base58-2.0.1-py3-none-any.whl (4.3 kB)\n",
      "Collecting cachetools>=4.0\n",
      "  Downloading cachetools-4.1.1-py3-none-any.whl (10 kB)\n",
      "Collecting tzlocal\n",
      "  Downloading tzlocal-2.1-py2.py3-none-any.whl (16 kB)\n",
      "Collecting validators\n",
      "  Downloading validators-0.18.1-py3-none-any.whl (19 kB)\n",
      "Requirement already satisfied: joblib>=0.11 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from scikit-learn->simpletransformers) (0.14.1)\n",
      "Collecting gitdb<5,>=4.0.1\n",
      "  Using cached gitdb-4.0.5-py3-none-any.whl (63 kB)\n",
      "Requirement already satisfied: pathtools>=0.1.1 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from watchdog>=0.8.3->wandb->simpletransformers) (0.1.2)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from protobuf>=3.12.0->wandb->simpletransformers) (45.2.0.post20200210)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from packaging->transformers>=3.0.2->simpletransformers) (2.4.6)\n",
      "Requirement already satisfied: ipykernel>=5.1.2; python_version >= \"3.4\" in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from pydeck>=0.1.dev5->streamlit->simpletransformers) (5.1.4)\n",
      "Requirement already satisfied: ipywidgets>=7.0.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from pydeck>=0.1.dev5->streamlit->simpletransformers) (7.5.1)\n",
      "Requirement already satisfied: jinja2>=2.10.1 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from pydeck>=0.1.dev5->streamlit->simpletransformers) (2.11.1)\n",
      "Requirement already satisfied: traitlets>=4.3.2 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from pydeck>=0.1.dev5->streamlit->simpletransformers) (4.3.3)\n",
      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from botocore>=1.13.44->streamlit->simpletransformers) (0.10.0)\n",
      "Requirement already satisfied: toolz in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from altair>=3.2.0->streamlit->simpletransformers) (0.10.0)\n",
      "Requirement already satisfied: jsonschema in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from altair>=3.2.0->streamlit->simpletransformers) (3.2.0)\n",
      "Requirement already satisfied: entrypoints in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from altair>=3.2.0->streamlit->simpletransformers) (0.3)\n",
      "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from boto3->streamlit->simpletransformers) (0.3.3)\n",
      "Requirement already satisfied: decorator>=3.4.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from validators->streamlit->simpletransformers) (4.4.1)\n",
      "Collecting smmap<4,>=3.0.1\n",
      "  Using cached smmap-3.0.4-py2.py3-none-any.whl (25 kB)\n",
      "Requirement already satisfied: jupyter-client in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (5.3.4)\n",
      "Requirement already satisfied: ipython>=5.0.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (7.12.0)\n",
      "Requirement already satisfied: nbformat>=4.2.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (5.0.4)\n",
      "Requirement already satisfied: widgetsnbextension~=3.5.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (3.5.1)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from jinja2>=2.10.1->pydeck>=0.1.dev5->streamlit->simpletransformers) (1.1.1)\n",
      "Requirement already satisfied: ipython-genutils in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from traitlets>=4.3.2->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.2.0)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from jsonschema->altair>=3.2.0->streamlit->simpletransformers) (0.15.7)\n",
      "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from jsonschema->altair>=3.2.0->streamlit->simpletransformers) (2.0.0)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from jsonschema->altair>=3.2.0->streamlit->simpletransformers) (19.3.0)\n",
      "Requirement already satisfied: jupyter-core>=4.6.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from jupyter-client->ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (4.6.1)\n",
      "Requirement already satisfied: pyzmq>=13 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from jupyter-client->ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (18.1.1)\n",
      "Requirement already satisfied: pexpect; sys_platform != \"win32\" in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (4.8.0)\n",
      "Requirement already satisfied: jedi>=0.10 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.14.1)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (3.0.3)\n",
      "Requirement already satisfied: backcall in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.1.0)\n",
      "Requirement already satisfied: pickleshare in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.7.5)\n",
      "Requirement already satisfied: pygments in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (2.5.2)\n",
      "Requirement already satisfied: notebook>=4.4.1 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (6.0.3)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from importlib-metadata; python_version < \"3.8\"->jsonschema->altair>=3.2.0->streamlit->simpletransformers) (2.2.0)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from pexpect; sys_platform != \"win32\"->ipython>=5.0.0->ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.6.0)\n",
      "Requirement already satisfied: parso>=0.5.0 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from jedi>=0.10->ipython>=5.0.0->ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.5.2)\n",
      "Requirement already satisfied: wcwidth in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.0.0->ipykernel>=5.1.2; python_version >= \"3.4\"->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.1.8)\n",
      "Requirement already satisfied: Send2Trash in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (1.5.0)\n",
      "Requirement already satisfied: prometheus-client in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.7.1)\n",
      "Requirement already satisfied: nbconvert in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (5.6.1)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: terminado>=0.8.1 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.8.3)\n",
      "Requirement already satisfied: testpath in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.4.4)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (1.4.2)\n",
      "Requirement already satisfied: defusedxml in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.6.0)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.8.4)\n",
      "Requirement already satisfied: bleach in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (3.2.1)\n",
      "Requirement already satisfied: webencodings in /home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.0.0->pydeck>=0.1.dev5->streamlit->simpletransformers) (0.5.1)\n",
      "Building wheels for collected packages: seqeval, subprocess32, promise, sacremoses, blinker\n",
      "  Building wheel for seqeval (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for seqeval: filename=seqeval-1.2.1-py3-none-any.whl size=16167 sha256=4dc407db4f5009fa6e6b695d724c104683d3e37945ab8bfddd9e3744a4147cc2\n",
      "  Stored in directory: /home/ec2-user/.cache/pip/wheels/41/ac/96/e9d5bceb83600d09ba2ca99693befff9b47bc35334943da786\n",
      "  Building wheel for subprocess32 (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for subprocess32: filename=subprocess32-3.5.4-py3-none-any.whl size=6489 sha256=ebeecb9acfb5accbd06935ebf43b9021fb086287bde7f27bb4248cccaaaab9e3\n",
      "  Stored in directory: /home/ec2-user/.cache/pip/wheels/44/3a/ab/102386d84fe551b6cedb628ed1e74c5f5be76af8b909aeda09\n",
      "  Building wheel for promise (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for promise: filename=promise-2.3-py3-none-any.whl size=21495 sha256=5dede3c08ddd83fba9aa751950b8736a6959907ec2d149797b41b105e32a0b76\n",
      "  Stored in directory: /home/ec2-user/.cache/pip/wheels/59/9a/1d/3f1afbbb5122d0410547bf9eb50955f4a7a98e53a6d8b99bd1\n",
      "  Building wheel for sacremoses (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for sacremoses: filename=sacremoses-0.0.43-py3-none-any.whl size=893259 sha256=77dfdd34a73938970f61b675f45a7357d59b4565be6ed8adbe2800f5442bbf2e\n",
      "  Stored in directory: /home/ec2-user/.cache/pip/wheels/49/25/98/cdea9c79b2d9a22ccc59540b1784b67f06b633378e97f58da2\n",
      "  Building wheel for blinker (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for blinker: filename=blinker-1.4-py3-none-any.whl size=13452 sha256=efa29cd674b8803e6ba0372045e63aec4c336e8428d78d904810dbbfcd6a129f\n",
      "  Stored in directory: /home/ec2-user/.cache/pip/wheels/4f/4a/93/c5ed8c11fedbe97fb8b8032b301eaa736248684b44087a7259\n",
      "Successfully built seqeval subprocess32 promise sacremoses blinker\n",
      "\u001b[31mERROR: seqeval 1.2.1 has requirement numpy==1.19.2, but you'll have numpy 1.18.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: seqeval 1.2.1 has requirement scikit-learn==0.23.2, but you'll have scikit-learn 0.22.1 which is incompatible.\u001b[0m\n",
      "Installing collected packages: regex, tokenizers, seqeval, shortuuid, subprocess32, promise, smmap, gitdb, GitPython, protobuf, docker-pycreds, sentry-sdk, configparser, wandb, tqdm, sacremoses, sentencepiece, dataclasses, transformers, blinker, pyarrow, pydeck, astor, altair, enum-compat, base58, cachetools, tzlocal, validators, streamlit, tensorboardx, simpletransformers\n",
      "  Attempting uninstall: protobuf\n",
      "    Found existing installation: protobuf 3.11.4\n",
      "    Uninstalling protobuf-3.11.4:\n",
      "      Successfully uninstalled protobuf-3.11.4\n",
      "  Attempting uninstall: tqdm\n",
      "    Found existing installation: tqdm 4.42.1\n",
      "    Uninstalling tqdm-4.42.1:\n",
      "      Successfully uninstalled tqdm-4.42.1\n",
      "Successfully installed GitPython-3.1.9 altair-4.1.0 astor-0.8.1 base58-2.0.1 blinker-1.4 cachetools-4.1.1 configparser-5.0.1 dataclasses-0.7 docker-pycreds-0.4.0 enum-compat-0.0.3 gitdb-4.0.5 promise-2.3 protobuf-3.13.0 pyarrow-2.0.0 pydeck-0.5.0b1 regex-2020.10.15 sacremoses-0.0.43 sentencepiece-0.1.91 sentry-sdk-0.19.1 seqeval-1.2.1 shortuuid-1.0.1 simpletransformers-0.48.14 smmap-3.0.4 streamlit-0.69.2 subprocess32-3.5.4 tensorboardx-2.1 tokenizers-0.9.2 tqdm-4.50.2 transformers-3.4.0 tzlocal-2.1 validators-0.18.1 wandb-0.10.7\n",
      "\u001b[33mWARNING: You are using pip version 20.0.2; however, version 20.2.4 is available.\n",
      "You should consider upgrading via the '/home/ec2-user/anaconda3/envs/pytorch_latest_p36/bin/python -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "! pip install simpletransformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train a Model with roberta as encoder and Bert as decoder.\n",
    "\n",
    "This is a default setting from the library. The following rules currently apply to Encoder-Decoder models:\n",
    "\n",
    "1. The decoder must be a bert model.\n",
    "2. The encoder can be one of [bert, roberta, distilbert, camembert, electra].\n",
    "3. The encoder and the decoder must be of the same \"size\". (E.g. roberta-base encoder and a bert-base-uncased decoder)\n",
    "\n",
    "We will be using a batch size of 16 set to 10 epochs and GPU(preferably P3's) to run the below code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "wandb: WARNING W&B installed but not logged in.  Run `wandb login` or set the WANDB_API_KEY env variable.\n"
     ]
    }
   ],
   "source": [
    "from simpletransformers.seq2seq import Seq2SeqModel\n",
    "import logging\n",
    "import pandas as pd\n",
    "import sklearn\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertLMHeadModel: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertLMHeadModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
      "- This IS NOT expected if you are initializing BertLMHeadModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertLMHeadModel were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['bert.encoder.layer.0.crossattention.self.query.weight', 'bert.encoder.layer.0.crossattention.self.query.bias', 'bert.encoder.layer.0.crossattention.self.key.weight', 'bert.encoder.layer.0.crossattention.self.key.bias', 'bert.encoder.layer.0.crossattention.self.value.weight', 'bert.encoder.layer.0.crossattention.self.value.bias', 'bert.encoder.layer.0.crossattention.output.dense.weight', 'bert.encoder.layer.0.crossattention.output.dense.bias', 'bert.encoder.layer.0.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.0.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.1.crossattention.self.query.weight', 'bert.encoder.layer.1.crossattention.self.query.bias', 'bert.encoder.layer.1.crossattention.self.key.weight', 'bert.encoder.layer.1.crossattention.self.key.bias', 'bert.encoder.layer.1.crossattention.self.value.weight', 'bert.encoder.layer.1.crossattention.self.value.bias', 'bert.encoder.layer.1.crossattention.output.dense.weight', 'bert.encoder.layer.1.crossattention.output.dense.bias', 'bert.encoder.layer.1.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.1.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.2.crossattention.self.query.weight', 'bert.encoder.layer.2.crossattention.self.query.bias', 'bert.encoder.layer.2.crossattention.self.key.weight', 'bert.encoder.layer.2.crossattention.self.key.bias', 'bert.encoder.layer.2.crossattention.self.value.weight', 'bert.encoder.layer.2.crossattention.self.value.bias', 'bert.encoder.layer.2.crossattention.output.dense.weight', 'bert.encoder.layer.2.crossattention.output.dense.bias', 'bert.encoder.layer.2.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.2.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.3.crossattention.self.query.weight', 'bert.encoder.layer.3.crossattention.self.query.bias', 'bert.encoder.layer.3.crossattention.self.key.weight', 'bert.encoder.layer.3.crossattention.self.key.bias', 'bert.encoder.layer.3.crossattention.self.value.weight', 'bert.encoder.layer.3.crossattention.self.value.bias', 'bert.encoder.layer.3.crossattention.output.dense.weight', 'bert.encoder.layer.3.crossattention.output.dense.bias', 'bert.encoder.layer.3.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.3.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.4.crossattention.self.query.weight', 'bert.encoder.layer.4.crossattention.self.query.bias', 'bert.encoder.layer.4.crossattention.self.key.weight', 'bert.encoder.layer.4.crossattention.self.key.bias', 'bert.encoder.layer.4.crossattention.self.value.weight', 'bert.encoder.layer.4.crossattention.self.value.bias', 'bert.encoder.layer.4.crossattention.output.dense.weight', 'bert.encoder.layer.4.crossattention.output.dense.bias', 'bert.encoder.layer.4.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.4.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.5.crossattention.self.query.weight', 'bert.encoder.layer.5.crossattention.self.query.bias', 'bert.encoder.layer.5.crossattention.self.key.weight', 'bert.encoder.layer.5.crossattention.self.key.bias', 'bert.encoder.layer.5.crossattention.self.value.weight', 'bert.encoder.layer.5.crossattention.self.value.bias', 'bert.encoder.layer.5.crossattention.output.dense.weight', 'bert.encoder.layer.5.crossattention.output.dense.bias', 'bert.encoder.layer.5.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.5.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.6.crossattention.self.query.weight', 'bert.encoder.layer.6.crossattention.self.query.bias', 'bert.encoder.layer.6.crossattention.self.key.weight', 'bert.encoder.layer.6.crossattention.self.key.bias', 'bert.encoder.layer.6.crossattention.self.value.weight', 'bert.encoder.layer.6.crossattention.self.value.bias', 'bert.encoder.layer.6.crossattention.output.dense.weight', 'bert.encoder.layer.6.crossattention.output.dense.bias', 'bert.encoder.layer.6.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.6.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.7.crossattention.self.query.weight', 'bert.encoder.layer.7.crossattention.self.query.bias', 'bert.encoder.layer.7.crossattention.self.key.weight', 'bert.encoder.layer.7.crossattention.self.key.bias', 'bert.encoder.layer.7.crossattention.self.value.weight', 'bert.encoder.layer.7.crossattention.self.value.bias', 'bert.encoder.layer.7.crossattention.output.dense.weight', 'bert.encoder.layer.7.crossattention.output.dense.bias', 'bert.encoder.layer.7.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.7.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.8.crossattention.self.query.weight', 'bert.encoder.layer.8.crossattention.self.query.bias', 'bert.encoder.layer.8.crossattention.self.key.weight', 'bert.encoder.layer.8.crossattention.self.key.bias', 'bert.encoder.layer.8.crossattention.self.value.weight', 'bert.encoder.layer.8.crossattention.self.value.bias', 'bert.encoder.layer.8.crossattention.output.dense.weight', 'bert.encoder.layer.8.crossattention.output.dense.bias', 'bert.encoder.layer.8.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.8.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.9.crossattention.self.query.weight', 'bert.encoder.layer.9.crossattention.self.query.bias', 'bert.encoder.layer.9.crossattention.self.key.weight', 'bert.encoder.layer.9.crossattention.self.key.bias', 'bert.encoder.layer.9.crossattention.self.value.weight', 'bert.encoder.layer.9.crossattention.self.value.bias', 'bert.encoder.layer.9.crossattention.output.dense.weight', 'bert.encoder.layer.9.crossattention.output.dense.bias', 'bert.encoder.layer.9.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.9.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.10.crossattention.self.query.weight', 'bert.encoder.layer.10.crossattention.self.query.bias', 'bert.encoder.layer.10.crossattention.self.key.weight', 'bert.encoder.layer.10.crossattention.self.key.bias', 'bert.encoder.layer.10.crossattention.self.value.weight', 'bert.encoder.layer.10.crossattention.self.value.bias', 'bert.encoder.layer.10.crossattention.output.dense.weight', 'bert.encoder.layer.10.crossattention.output.dense.bias', 'bert.encoder.layer.10.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.10.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.11.crossattention.self.query.weight', 'bert.encoder.layer.11.crossattention.self.query.bias', 'bert.encoder.layer.11.crossattention.self.key.weight', 'bert.encoder.layer.11.crossattention.self.key.bias', 'bert.encoder.layer.11.crossattention.self.value.weight', 'bert.encoder.layer.11.crossattention.self.value.bias', 'bert.encoder.layer.11.crossattention.output.dense.weight', 'bert.encoder.layer.11.crossattention.output.dense.bias', 'bert.encoder.layer.11.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.11.crossattention.output.LayerNorm.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "INFO:simpletransformers.seq2seq.seq2seq_utils: Creating features from dataset file at cache_dir/\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "28baa2b2941048c1aac28b7ba2493625",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=8498.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/tokenization_utils_base.py:1944: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
      "  FutureWarning,\n",
      "Exception ignored in: <bound method _MultiProcessingDataLoaderIter.__del__ of <torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7f0673ada0f0>>\n",
      "Traceback (most recent call last):\n",
      "  File \"/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/utils/data/dataloader.py\", line 1101, in __del__\n",
      "    self._shutdown_workers()\n",
      "  File \"/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/utils/data/dataloader.py\", line 1075, in _shutdown_workers\n",
      "    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)\n",
      "  File \"/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/multiprocessing/process.py\", line 124, in join\n",
      "    res = self._popen.wait(timeout)\n",
      "  File \"/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/multiprocessing/popen_fork.py\", line 47, in wait\n",
      "    if not wait([self.sentinel], timeout):\n",
      "  File \"/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/multiprocessing/connection.py\", line 911, in wait\n",
      "    ready = selector.select(timeout)\n",
      "  File \"/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/selectors.py\", line 376, in select\n",
      "    fd_event_list = self._poll.poll(timeout)\n",
      "KeyboardInterrupt: \n",
      "INFO:simpletransformers.seq2seq.seq2seq_utils: Saving features into cached file cache_dir/roberta-base-bert-base-cased_cached_158498\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:simpletransformers.seq2seq.seq2seq_model: Training started\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "31ad442c4fb94070bbe35c6989cf3b9d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Epoch'), FloatProgress(value=0.0, max=10.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7483dade45994191b15c179d2a2bd556",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Running Epoch 0 of 10'), FloatProgress(value=0.0, max=532.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\n",
      "  \"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\", UserWarning)\n",
      "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:231: UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.\n",
      "  warnings.warn(\"To get the last learning rate computed by the scheduler, \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b1e369dbe7841d4aef32734caf98e6f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Running Epoch 1 of 10'), FloatProgress(value=0.0, max=532.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "108fc9d12db14cea9e8d49facfd0b6cf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Running Epoch 2 of 10'), FloatProgress(value=0.0, max=532.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "adba762ffc24487aa01115c1aded5248",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Running Epoch 3 of 10'), FloatProgress(value=0.0, max=532.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:simpletransformers.seq2seq.seq2seq_model:Saving model into outputs/checkpoint-2000\n",
      "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:200: UserWarning: Please also save or load the state of the optimzer when saving or loading the scheduler.\n",
      "  warnings.warn(SAVE_STATE_WARNING, UserWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "58531395ef0e41e89fef68e04e39be63",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Running Epoch 4 of 10'), FloatProgress(value=0.0, max=532.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72b7ab3cd85748d198ecda14b74908e5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Running Epoch 5 of 10'), FloatProgress(value=0.0, max=532.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f79c07785be0467a8e0ce36bb896a3df",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Running Epoch 6 of 10'), FloatProgress(value=0.0, max=532.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "77488b0b4fcc4e82b89dffde5dde6800",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Running Epoch 7 of 10'), FloatProgress(value=0.0, max=532.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:simpletransformers.seq2seq.seq2seq_model:Saving model into outputs/checkpoint-4000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca35a2602002439f90029f01c525d47f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Running Epoch 8 of 10'), FloatProgress(value=0.0, max=532.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a8c4849c053d44acba721b93a751e538",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Running Epoch 9 of 10'), FloatProgress(value=0.0, max=532.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:simpletransformers.seq2seq.seq2seq_model:Saving model into outputs/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:simpletransformers.seq2seq.seq2seq_model: Training of roberta-base-bert-base-cased model complete. Saved to outputs/.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(5320, 0.709893412254637)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logging.basicConfig(level=logging.INFO)\n",
    "transformers_logger = logging.getLogger(\"transformers\")\n",
    "transformers_logger.setLevel(logging.WARNING)\n",
    "\n",
    "\n",
    "\n",
    "model_args = {\n",
    "    \"reprocess_input_data\": True,\n",
    "    \"overwrite_output_dir\": True,\n",
    "    \"max_seq_length\": 15,\n",
    "    \"train_batch_size\": 16,\n",
    "    \"num_train_epochs\": 10,\n",
    "    \"save_eval_checkpoints\": False,\n",
    "    \"save_model_every_epoch\": False,\n",
    "    \"evaluate_generated_text\": True,\n",
    "    \"evaluate_during_training_verbose\": True,\n",
    "    \"use_multiprocessing\": False,\n",
    "    \"max_length\": 15,\n",
    "    \"manual_seed\": 4,\n",
    "}\n",
    "\n",
    "encoder_type = \"roberta\"\n",
    "\n",
    "model = Seq2SeqModel(\n",
    "     encoder_type,\n",
    "    \"roberta-base\",\n",
    "    \"bert-base-cased\",\n",
    "    args=model_args,\n",
    "    use_cuda=True,\n",
    ")\n",
    "\n",
    "model.train_model(trainDF)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate the model on evalset by taking 1000 sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:simpletransformers.seq2seq.seq2seq_utils: Creating features from dataset file at cache_dir/\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "86b4e037af9044b9bb9a48068485788d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:simpletransformers.seq2seq.seq2seq_utils: Saving features into cached file cache_dir/roberta-base-bert-base-cased_cached_151000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09e0536a45624a9093fbb1c600eea2db",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value='Running Evaluation'), FloatProgress(value=0.0, max=125.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:simpletransformers.seq2seq.seq2seq_model:{'eval_loss': 0.488968826815486}\n"
     ]
    }
   ],
   "source": [
    "results = model.eval_model(evalDF[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run predictions on few unseen sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['What is your name?....']\n"
     ]
    }
   ],
   "source": [
    "print(model.predict([\"Wht is your name\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_latest_p36",
   "language": "python",
   "name": "conda_pytorch_latest_p36"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}