ch07/01_main-chapter-code/ch07.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "12e91914-5f51-43fa-b65b-625e73b4d17b",
   "metadata": {
    "id": "12e91914-5f51-43fa-b65b-625e73b4d17b"
   },
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp?1\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2520ec3-722f-4f44-bdd1-885b13e7afbf",
   "metadata": {
    "id": "c2520ec3-722f-4f44-bdd1-885b13e7afbf"
   },
   "source": [
    "# Chapter 7: Finetuning To Follow Instructions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4e19327b-6c02-4881-ad02-9b6d3ec0b1b4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4e19327b-6c02-4881-ad02-9b6d3ec0b1b4",
    "outputId": "bcdfe2cb-d084-4920-d703-503131aabec3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numpy version: 2.0.2\n",
      "matplotlib version: 3.10.0\n",
      "tiktoken version: 0.8.0\n",
      "torch version: 2.5.1+cu124\n",
      "tqdm version: 4.67.1\n",
      "tensorflow version: 2.18.0\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "pkgs = [\n",
    "    \"numpy\",       # PyTorch & TensorFlow dependency\n",
    "    \"matplotlib\",  # Plotting library\n",
    "    \"tiktoken\",    # Tokenizer\n",
    "    \"torch\",       # Deep learning library\n",
    "    \"tqdm\",        # Progress bar\n",
    "    \"tensorflow\",  # For OpenAI's pretrained weights\n",
    "]\n",
    "for p in pkgs:\n",
    "    print(f\"{p} version: {version(p)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "264fca98-2f9a-4193-b435-2abfa3b4142f",
   "metadata": {
    "id": "264fca98-2f9a-4193-b435-2abfa3b4142f"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/overview.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bbc68e9-75b3-41f1-ac2c-e071c3cd0813",
   "metadata": {
    "id": "8bbc68e9-75b3-41f1-ac2c-e071c3cd0813"
   },
   "source": [
    "## 7.1 Introduction to instruction finetuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53dba24a-6805-496c-9a7f-c75e2d3527ab",
   "metadata": {
    "id": "53dba24a-6805-496c-9a7f-c75e2d3527ab"
   },
   "source": [
    "- In chapter 5, we saw that pretraining an LLM involves a training procedure where it learns to generate one word at a time\n",
    "- Hence, a pretrained LLM is good at text completion, but it is not good at following instructions\n",
    "- In this chapter, we teach the LLM to follow instructions better"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18dc0535-0904-44ed-beaf-9b678292ef35",
   "metadata": {
    "id": "18dc0535-0904-44ed-beaf-9b678292ef35"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/instruction-following.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4698b23-12e0-4bd7-a140-ccb3dd71d4e8",
   "metadata": {
    "id": "b4698b23-12e0-4bd7-a140-ccb3dd71d4e8"
   },
   "source": [
    "- The topics covered in this chapter are summarized in the figure below\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-1.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5384f0cf-ef3c-4436-a5fa-59bd25649f86",
   "metadata": {
    "id": "5384f0cf-ef3c-4436-a5fa-59bd25649f86"
   },
   "source": [
    "## 7.2 Preparing a dataset for supervised instruction finetuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8b34ff8-619f-4e89-bd03-ce513269760d",
   "metadata": {
    "id": "f8b34ff8-619f-4e89-bd03-ce513269760d"
   },
   "source": [
    "- We will work with an instruction dataset I prepared for this chapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0G3axLw6kY1N",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0G3axLw6kY1N",
    "outputId": "07e1e4f9-026c-48c1-8a06-f2bfb1fb354e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 1100\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import os\n",
    "import urllib\n",
    "\n",
    "\n",
    "def download_and_load_file(file_path, url):\n",
    "\n",
    "    if not os.path.exists(file_path):\n",
    "        with urllib.request.urlopen(url) as response:\n",
    "            text_data = response.read().decode(\"utf-8\")\n",
    "        with open(file_path, \"w\", encoding=\"utf-8\") as file:\n",
    "            file.write(text_data)\n",
    "\n",
    "    # The book originally contained this unnecessary \"else\" clause:\n",
    "    #else:\n",
    "    #    with open(file_path, \"r\", encoding=\"utf-8\") as file:\n",
    "    #        text_data = file.read()\n",
    "\n",
    "    with open(file_path, \"r\", encoding=\"utf-8\") as file:\n",
    "        data = json.load(file)\n",
    "\n",
    "    return data\n",
    "\n",
    "\n",
    "file_path = \"instruction-data.json\"\n",
    "url = (\n",
    "    \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch\"\n",
    "    \"/main/ch07/01_main-chapter-code/instruction-data.json\"\n",
    ")\n",
    "\n",
    "data = download_and_load_file(file_path, url)\n",
    "print(\"Number of entries:\", len(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7af8176-4255-4e92-8c7d-998771733eb8",
   "metadata": {
    "id": "d7af8176-4255-4e92-8c7d-998771733eb8"
   },
   "source": [
    "- Each item in the `data` list we loaded from the JSON file above is a dictionary in the following form"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "-LiuBMsHkzQV",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-LiuBMsHkzQV",
    "outputId": "a4ee5c2d-db53-4a80-e5ee-0bbcf6fe0450"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example entry:\n",
      " {'instruction': 'Identify the correct spelling of the following word.', 'input': 'Ocassion', 'output': \"The correct spelling is 'Occasion.'\"}\n"
     ]
    }
   ],
   "source": [
    "print(\"Example entry:\\n\", data[50])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5a32b34-485a-4816-a77a-da14f9fe6e46",
   "metadata": {
    "id": "c5a32b34-485a-4816-a77a-da14f9fe6e46"
   },
   "source": [
    "- Note that the `'input'` field can be empty:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "uFInFxDDk2Je",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uFInFxDDk2Je",
    "outputId": "b4f84027-bb9e-4e51-b79e-1329c8bff093"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Another example entry:\n",
      " {'instruction': \"What is an antonym of 'complicated'?\", 'input': '', 'output': \"An antonym of 'complicated' is 'simple'.\"}\n"
     ]
    }
   ],
   "source": [
    "print(\"Another example entry:\\n\", data[999])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f034799a-6575-45fd-98c9-9d1012d0fd58",
   "metadata": {
    "id": "f034799a-6575-45fd-98c9-9d1012d0fd58"
   },
   "source": [
    "- Instruction finetuning is often referred to as \"supervised instruction finetuning\" because it involves training a model on a dataset where the input-output pairs are explicitly provided\n",
    "- There are different ways to format the entries as inputs to the LLM; the figure below illustrates two example formats that were used for training the Alpaca (https://crfm.stanford.edu/2023/03/13/alpaca.html) and Phi-3 (https://arxiv.org/abs/2404.14219) LLMs, respectively"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dffa4f70-44d4-4be4-89a9-2159f4885b10",
   "metadata": {
    "id": "dffa4f70-44d4-4be4-89a9-2159f4885b10"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/prompt-style.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd79a74e-befb-491c-be49-f777a6a5b6a6",
   "metadata": {
    "id": "dd79a74e-befb-491c-be49-f777a6a5b6a6"
   },
   "source": [
    "- In this chapter, we use Alpaca-style prompt formatting, which was the original prompt template for instruction finetuning\n",
    "- Below, we format the input that we will pass as input to the LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "Jhk37nnJnkBh",
   "metadata": {
    "id": "Jhk37nnJnkBh"
   },
   "outputs": [],
   "source": [
    "def format_input(entry):\n",
    "    instruction_text = (\n",
    "        f\"Below is an instruction that describes a task. \"\n",
    "        f\"Write a response that appropriately completes the request.\"\n",
    "        f\"\\n\\n### Instruction:\\n{entry['instruction']}\"\n",
    "    )\n",
    "\n",
    "    input_text = f\"\\n\\n### Input:\\n{entry['input']}\" if entry[\"input\"] else \"\"\n",
    "\n",
    "    return instruction_text + input_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "011e78b4-e89a-4653-a2ee-7b2739ca04d6",
   "metadata": {
    "id": "011e78b4-e89a-4653-a2ee-7b2739ca04d6"
   },
   "source": [
    "- A formatted response with input field looks like as shown below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "F9UQRfjzo4Js",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F9UQRfjzo4Js",
    "outputId": "7b615d35-2a5f-474d-9292-a69bc3850e16"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
      "\n",
      "### Instruction:\n",
      "Identify the correct spelling of the following word.\n",
      "\n",
      "### Input:\n",
      "Ocassion\n",
      "\n",
      "### Response:\n",
      "The correct spelling is 'Occasion.'\n"
     ]
    }
   ],
   "source": [
    "model_input = format_input(data[50])\n",
    "desired_response = f\"\\n\\n### Response:\\n{data[50]['output']}\"\n",
    "\n",
    "print(model_input + desired_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dc93ddf-431c-49c0-96f2-fb3a79c4d94c",
   "metadata": {
    "id": "4dc93ddf-431c-49c0-96f2-fb3a79c4d94c"
   },
   "source": [
    "- Below is a formatted response without an input field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a3891fa9-f738-41cd-946c-80ef9a99c346",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a3891fa9-f738-41cd-946c-80ef9a99c346",
    "outputId": "2142c5a4-b594-49c5-affe-2d963a7bd46b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
      "\n",
      "### Instruction:\n",
      "What is an antonym of 'complicated'?\n",
      "\n",
      "### Response:\n",
      "An antonym of 'complicated' is 'simple'.\n"
     ]
    }
   ],
   "source": [
    "model_input = format_input(data[999])\n",
    "desired_response = f\"\\n\\n### Response:\\n{data[999]['output']}\"\n",
    "\n",
    "print(model_input + desired_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4aa8afd5-2a21-49a5-90c3-6a03865a4771",
   "metadata": {
    "id": "4aa8afd5-2a21-49a5-90c3-6a03865a4771"
   },
   "source": [
    "- Lastly, before we prepare the PyTorch data loaders in the next section, we divide the dataset into a training, validation, and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "aFZVopbIlNfx",
   "metadata": {
    "id": "aFZVopbIlNfx"
   },
   "outputs": [],
   "source": [
    "train_portion = int(len(data) * 0.85)  # 85% for training\n",
    "test_portion = int(len(data) * 0.1)    # 10% for testing\n",
    "val_portion = len(data) - train_portion - test_portion  # Remaining 5% for validation\n",
    "\n",
    "train_data = data[:train_portion]\n",
    "test_data = data[train_portion:train_portion + test_portion]\n",
    "val_data = data[train_portion + test_portion:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "-zf6oht6bIUQ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-zf6oht6bIUQ",
    "outputId": "657ec5c6-4caa-4d1a-ba2e-23acd755ab07"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set length: 935\n",
      "Validation set length: 55\n",
      "Test set length: 110\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set length:\", len(train_data))\n",
    "print(\"Validation set length:\", len(val_data))\n",
    "print(\"Test set length:\", len(test_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcaaf606-f913-4445-8301-632ae10d387d",
   "metadata": {
    "id": "fcaaf606-f913-4445-8301-632ae10d387d"
   },
   "source": [
    "## 7.3 Organizing data into training batches"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "233f63bd-9755-4d07-8884-5e2e5345cf27",
   "metadata": {
    "id": "233f63bd-9755-4d07-8884-5e2e5345cf27"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-2.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c149fc1a-7757-4ec8-80cb-e2a3fb007a2c",
   "metadata": {
    "id": "c149fc1a-7757-4ec8-80cb-e2a3fb007a2c"
   },
   "source": [
    "- We tackle this dataset batching in several steps, as summarized in the figure below\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/detailed-batching.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9af423f-aad9-4b3c-bea5-153021c04862",
   "metadata": {
    "id": "b9af423f-aad9-4b3c-bea5-153021c04862"
   },
   "source": [
    "- First, we implement an `InstructionDataset` class that pre-tokenizes all inputs in the dataset, similar to the `SpamDataset` in chapter 6\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/pretokenizing.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "adc29dc4-f1c7-4c71-937b-95119d6239bb",
   "metadata": {
    "id": "adc29dc4-f1c7-4c71-937b-95119d6239bb"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "\n",
    "class InstructionDataset(Dataset):\n",
    "    def __init__(self, data, tokenizer):\n",
    "        self.data = data\n",
    "\n",
    "        # Pre-tokenize texts\n",
    "        self.encoded_texts = []\n",
    "        for entry in data:\n",
    "            instruction_plus_input = format_input(entry)\n",
    "            response_text = f\"\\n\\n### Response:\\n{entry['output']}\"\n",
    "            full_text = instruction_plus_input + response_text\n",
    "            self.encoded_texts.append(\n",
    "                tokenizer.encode(full_text)\n",
    "            )\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        return self.encoded_texts[index]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "384f0e69-4b22-41c0-a25d-f077527eddd1",
   "metadata": {
    "id": "384f0e69-4b22-41c0-a25d-f077527eddd1"
   },
   "source": [
    "- Similar to chapter 6, we want to collect multiple training examples in a batch to accelerate training; this requires padding all inputs to a similar length\n",
    "- Also similar to the previous chapter, we use the `<|endoftext|>` token as a padding token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ff24fe1a-5746-461c-ad3d-b6d84a1a7c96",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ff24fe1a-5746-461c-ad3d-b6d84a1a7c96",
    "outputId": "ac44227b-9ec2-4131-9df8-89caa6e879ca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50256]\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "\n",
    "print(tokenizer.encode(\"<|endoftext|>\", allowed_special={\"<|endoftext|>\"}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e5bd7bc-f347-4cf8-a0c2-94cb8799e427",
   "metadata": {
    "id": "9e5bd7bc-f347-4cf8-a0c2-94cb8799e427"
   },
   "source": [
    "- In chapter 6, we padded all examples in a dataset to the same length\n",
    "  - Here, we take a more sophisticated approach and develop a custom \"collate\" function that we can pass to the data loader\n",
    "  - This custom collate function pads the training examples in each batch to have the same length (but different batches can have different lengths)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65c4d943-4aa8-4a44-874e-05bc6831fbd3",
   "metadata": {
    "id": "65c4d943-4aa8-4a44-874e-05bc6831fbd3"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/padding.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "eb4c77dd-c956-4a1b-897b-b466909f18ca",
   "metadata": {
    "id": "eb4c77dd-c956-4a1b-897b-b466909f18ca"
   },
   "outputs": [],
   "source": [
    "def custom_collate_draft_1(\n",
    "    batch,\n",
    "    pad_token_id=50256,\n",
    "    device=\"cpu\"\n",
    "):\n",
    "    # Find the longest sequence in the batch\n",
    "    # and increase the max length by +1, which will add one extra\n",
    "    # padding token below\n",
    "    batch_max_length = max(len(item)+1 for item in batch)\n",
    "\n",
    "    # Pad and prepare inputs\n",
    "    inputs_lst = []\n",
    "\n",
    "    for item in batch:\n",
    "        new_item = item.copy()\n",
    "        # Add an <|endoftext|> token\n",
    "        new_item += [pad_token_id]\n",
    "        # Pad sequences to batch_max_length\n",
    "        padded = (\n",
    "            new_item + [pad_token_id] *\n",
    "            (batch_max_length - len(new_item))\n",
    "        )\n",
    "        # Via padded[:-1], we remove the extra padded token\n",
    "        # that has been added via the +1 setting in batch_max_length\n",
    "        # (the extra padding token will be relevant in later codes)\n",
    "        inputs = torch.tensor(padded[:-1])\n",
    "        inputs_lst.append(inputs)\n",
    "\n",
    "    # Convert list of inputs to tensor and transfer to target device\n",
    "    inputs_tensor = torch.stack(inputs_lst).to(device)\n",
    "    return inputs_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8fb02373-59b3-4f3a-b1d1-8181a2432645",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8fb02373-59b3-4f3a-b1d1-8181a2432645",
    "outputId": "93d987b9-e3ca-4857-9b28-b67d515a94d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[    0,     1,     2,     3,     4],\n",
      "        [    5,     6, 50256, 50256, 50256],\n",
      "        [    7,     8,     9, 50256, 50256]])\n"
     ]
    }
   ],
   "source": [
    "inputs_1 = [0, 1, 2, 3, 4]\n",
    "inputs_2 = [5, 6]\n",
    "inputs_3 = [7, 8, 9]\n",
    "\n",
    "batch = (\n",
    "    inputs_1,\n",
    "    inputs_2,\n",
    "    inputs_3\n",
    ")\n",
    "\n",
    "print(custom_collate_draft_1(batch))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c46832ab-39b7-45f8-b330-ac9adfa10d1b",
   "metadata": {
    "id": "c46832ab-39b7-45f8-b330-ac9adfa10d1b"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/batching-step-4.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17769a19-b961-4213-92ef-34f441b2d1d6",
   "metadata": {
    "id": "17769a19-b961-4213-92ef-34f441b2d1d6"
   },
   "source": [
    "- Above, we only returned the inputs to the LLM; however, for LLM training, we also need the target values\n",
    "- Similar to pretraining an LLM, the targets are the inputs shifted by 1 position to the right, so the LLM learns to predict the next token"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0386b6fe-3455-4e70-becd-a5a4681ba2ef",
   "metadata": {
    "id": "0386b6fe-3455-4e70-becd-a5a4681ba2ef"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/inputs-targets.webp?1\" width=400px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "74af192e-757c-4c0a-bdf9-b7eb25bf6ebc",
   "metadata": {
    "id": "74af192e-757c-4c0a-bdf9-b7eb25bf6ebc"
   },
   "outputs": [],
   "source": [
    "def custom_collate_draft_2(\n",
    "    batch,\n",
    "    pad_token_id=50256,\n",
    "    device=\"cpu\"\n",
    "):\n",
    "    # Find the longest sequence in the batch\n",
    "    batch_max_length = max(len(item)+1 for item in batch)\n",
    "\n",
    "    # Pad and prepare inputs\n",
    "    inputs_lst, targets_lst = [], []\n",
    "\n",
    "    for item in batch:\n",
    "        new_item = item.copy()\n",
    "        # Add an <|endoftext|> token\n",
    "        new_item += [pad_token_id]\n",
    "        # Pad sequences to max_length\n",
    "        padded = (\n",
    "            new_item + [pad_token_id] *\n",
    "            (batch_max_length - len(new_item))\n",
    "        )\n",
    "        inputs = torch.tensor(padded[:-1])  # Truncate the last token for inputs\n",
    "        targets = torch.tensor(padded[1:])  # Shift +1 to the right for targets\n",
    "        inputs_lst.append(inputs)\n",
    "        targets_lst.append(targets)\n",
    "\n",
    "    # Convert list of inputs to tensor and transfer to target device\n",
    "    inputs_tensor = torch.stack(inputs_lst).to(device)\n",
    "    targets_tensor = torch.stack(targets_lst).to(device)\n",
    "    return inputs_tensor, targets_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6eb2bce3-28a7-4f39-9d4b-5e972d69066c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6eb2bce3-28a7-4f39-9d4b-5e972d69066c",
    "outputId": "3d104439-c328-431b-ef7c-2639d86c2135"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[    0,     1,     2,     3,     4],\n",
      "        [    5,     6, 50256, 50256, 50256],\n",
      "        [    7,     8,     9, 50256, 50256]])\n",
      "tensor([[    1,     2,     3,     4, 50256],\n",
      "        [    6, 50256, 50256, 50256, 50256],\n",
      "        [    8,     9, 50256, 50256, 50256]])\n"
     ]
    }
   ],
   "source": [
    "inputs, targets = custom_collate_draft_2(batch)\n",
    "print(inputs)\n",
    "print(targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bf85703-a0e0-42aa-8f29-cbc28dbf4e15",
   "metadata": {
    "id": "3bf85703-a0e0-42aa-8f29-cbc28dbf4e15"
   },
   "source": [
    "- Next, we introduce an `ignore_index` value to replace all padding token IDs with a new value; the purpose of this `ignore_index` is that we can ignore padding values in the loss function (more on that later)\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/batching-step-5.webp?1\" width=500px>\n",
    "\n",
    "- Concretely, this means that we replace the token IDs corresponding to `50256` with `-100` as illustrated below"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd4bed33-956e-4b3f-a09c-586d8203109a",
   "metadata": {
    "id": "bd4bed33-956e-4b3f-a09c-586d8203109a"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/ignore-index.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5346513e-c3f4-44fe-af22-4ebd36497728",
   "metadata": {
    "id": "5346513e-c3f4-44fe-af22-4ebd36497728"
   },
   "source": [
    "- (In addition, we also introduce the `allowed_max_length` in case we want to limit the length of the samples; this will be useful if you plan to work with your own datasets that are longer than the 1024 token context size supported by the GPT-2 model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "41ec6e2d-9eb2-4124-913e-d2af39be4cf2",
   "metadata": {
    "id": "41ec6e2d-9eb2-4124-913e-d2af39be4cf2"
   },
   "outputs": [],
   "source": [
    "def custom_collate_fn(\n",
    "    batch,\n",
    "    pad_token_id=50256,\n",
    "    ignore_index=-100,\n",
    "    allowed_max_length=None,\n",
    "    device=\"cpu\"\n",
    "):\n",
    "    # Find the longest sequence in the batch\n",
    "    batch_max_length = max(len(item)+1 for item in batch)\n",
    "\n",
    "    # Pad and prepare inputs and targets\n",
    "    inputs_lst, targets_lst = [], []\n",
    "\n",
    "    for item in batch:\n",
    "        new_item = item.copy()\n",
    "        # Add an <|endoftext|> token\n",
    "        new_item += [pad_token_id]\n",
    "        # Pad sequences to max_length\n",
    "        padded = (\n",
    "            new_item + [pad_token_id] *\n",
    "            (batch_max_length - len(new_item))\n",
    "        )\n",
    "        inputs = torch.tensor(padded[:-1])  # Truncate the last token for inputs\n",
    "        targets = torch.tensor(padded[1:])  # Shift +1 to the right for targets\n",
    "\n",
    "        # New: Replace all but the first padding tokens in targets by ignore_index\n",
    "        mask = targets == pad_token_id\n",
    "        indices = torch.nonzero(mask).squeeze()\n",
    "        if indices.numel() > 1:\n",
    "            targets[indices[1:]] = ignore_index\n",
    "\n",
    "        # New: Optionally truncate to maximum sequence length\n",
    "        if allowed_max_length is not None:\n",
    "            inputs = inputs[:allowed_max_length]\n",
    "            targets = targets[:allowed_max_length]\n",
    "\n",
    "        inputs_lst.append(inputs)\n",
    "        targets_lst.append(targets)\n",
    "\n",
    "    # Convert list of inputs and targets to tensors and transfer to target device\n",
    "    inputs_tensor = torch.stack(inputs_lst).to(device)\n",
    "    targets_tensor = torch.stack(targets_lst).to(device)\n",
    "\n",
    "    return inputs_tensor, targets_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "cdf5eec4-9ebe-4be0-9fca-9a47bee88fdc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cdf5eec4-9ebe-4be0-9fca-9a47bee88fdc",
    "outputId": "e8f709b9-f4c5-428a-a6ac-2a4c1b9358ba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[    0,     1,     2,     3,     4],\n",
      "        [    5,     6, 50256, 50256, 50256],\n",
      "        [    7,     8,     9, 50256, 50256]])\n",
      "tensor([[    1,     2,     3,     4, 50256],\n",
      "        [    6, 50256,  -100,  -100,  -100],\n",
      "        [    8,     9, 50256,  -100,  -100]])\n"
     ]
    }
   ],
   "source": [
    "inputs, targets = custom_collate_fn(batch)\n",
    "print(inputs)\n",
    "print(targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26727c90-0d42-43b3-af21-0a66ad4fbbc7",
   "metadata": {
    "id": "26727c90-0d42-43b3-af21-0a66ad4fbbc7"
   },
   "source": [
    "- Let's see what this replacement by -100 accomplishes\n",
    "- For illustration purposes, let's assume we have a small classification task with 2 class labels, 0 and 1, similar to chapter 6\n",
    "- If we have the following logits values (outputs of the last layer of the model), we calculate the following loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "W2jvh-OP9MFV",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "W2jvh-OP9MFV",
    "outputId": "ccb3a703-59a7-4258-8841-57959a016e31"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.1269)\n"
     ]
    }
   ],
   "source": [
    "logits_1 = torch.tensor(\n",
    "    [[-1.0, 1.0],  # 1st training example\n",
    "     [-0.5, 1.5]]  # 2nd training example\n",
    ")\n",
    "targets_1 = torch.tensor([0, 1])\n",
    "\n",
    "\n",
    "loss_1 = torch.nn.functional.cross_entropy(logits_1, targets_1)\n",
    "print(loss_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5edd3244-8886-4505-92e9-367d28529e1e",
   "metadata": {
    "id": "5edd3244-8886-4505-92e9-367d28529e1e"
   },
   "source": [
    "- Now, adding one more training example will, as expected, influence the loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "nvVMuil89v9N",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nvVMuil89v9N",
    "outputId": "6d4683d4-5bfc-4a8c-de2a-95ecb2e716b9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.7936)\n"
     ]
    }
   ],
   "source": [
    "logits_2 = torch.tensor(\n",
    "    [[-1.0, 1.0],\n",
    "     [-0.5, 1.5],\n",
    "     [-0.5, 1.5]]  # New 3rd training example\n",
    ")\n",
    "targets_2 = torch.tensor([0, 1, 1])\n",
    "\n",
    "loss_2 = torch.nn.functional.cross_entropy(logits_2, targets_2)\n",
    "print(loss_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54dca331-40e0-468b-b690-189fe156ba8f",
   "metadata": {
    "id": "54dca331-40e0-468b-b690-189fe156ba8f"
   },
   "source": [
    "- Let's see what happens if we replace the class label of one of the examples with -100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "RTyB1vah9p56",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RTyB1vah9p56",
    "outputId": "da05302e-3fe0-439e-d1ed-82066bceb122"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.1269)\n",
      "loss_1 == loss_3: tensor(True)\n"
     ]
    }
   ],
   "source": [
    "targets_3 = torch.tensor([0, 1, -100])\n",
    "\n",
    "loss_3 = torch.nn.functional.cross_entropy(logits_2, targets_3)\n",
    "print(loss_3)\n",
    "print(\"loss_1 == loss_3:\", loss_1 == loss_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cef09d21-b652-4760-abea-4f76920e6a25",
   "metadata": {
    "id": "cef09d21-b652-4760-abea-4f76920e6a25"
   },
   "source": [
    "- As we can see, the resulting loss on these 3 training examples is the same as the loss we calculated from the 2 training examples, which means that the cross-entropy loss function ignored the training example with the -100 label\n",
    "- By default, PyTorch has the `cross_entropy(..., ignore_index=-100)` setting to ignore examples corresponding to the label -100\n",
    "- Using this -100 `ignore_index`, we can ignore the additional end-of-text (padding) tokens in the batches that we used to pad the training examples to equal length\n",
    "- However, we don't want to ignore the first instance of the end-of-text (padding) token (50256) because it can help signal to the LLM when the response is complete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a4e9c5f-7c49-4321-9f1b-a50468a84524",
   "metadata": {
    "id": "6a4e9c5f-7c49-4321-9f1b-a50468a84524"
   },
   "source": [
    "- In practice, it is also common to mask out the target token IDs that correspond to the instruction, as illustrated in the figure below (this is a recommended reader exercise after completing the chapter)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fab8f0ed-80e8-4fd9-bf84-e5d0e0bc0a39",
   "metadata": {
    "id": "fab8f0ed-80e8-4fd9-bf84-e5d0e0bc0a39"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/mask-instructions.webp?1\" width=600px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bccaf048-ec95-498c-9155-d5b3ccba6c96",
   "metadata": {
    "id": "bccaf048-ec95-498c-9155-d5b3ccba6c96"
   },
   "source": [
    "## 7.4 Creating data loaders for an instruction dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6b8e656-3af3-4db6-8dde-d8c216a12f50",
   "metadata": {
    "id": "e6b8e656-3af3-4db6-8dde-d8c216a12f50"
   },
   "source": [
    "- In this section, we use the `InstructionDataset` class and `custom_collate_fn` function to instantiate the training, validation, and test data loaders"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fffe390-b226-4d5c-983f-9f4da773cb82",
   "metadata": {
    "id": "9fffe390-b226-4d5c-983f-9f4da773cb82"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-3.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "932677e9-9317-42e8-b461-7b0269518f97",
   "metadata": {
    "id": "932677e9-9317-42e8-b461-7b0269518f97"
   },
   "source": [
    "- Another additional detail of the previous `custom_collate_fn` function is that we now directly move the data to the target device (e.g., GPU) instead of doing it in the main training loop, which improves efficiency because it can be carried out as a background process when we use the `custom_collate_fn` as part of the data loader\n",
    "- Using the `partial` function from Python's `functools` standard library, we create a new function with the `device` argument of the original function pre-filled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "etpqqWh8phKc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "etpqqWh8phKc",
    "outputId": "b4391c33-1a89-455b-faaa-5f874b6eb409"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device: cuda\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# Note:\n",
    "# Uncommenting the following lines will allow the code to run on Apple Silicon chips, if applicable,\n",
    "# which is much faster than on an Apple CPU (as measured on an M3 MacBook Air).\n",
    "# However, the resulting loss values may be slightly different.\n",
    "\n",
    "#if torch.cuda.is_available():\n",
    "#    device = torch.device(\"cuda\")\n",
    "#elif torch.backends.mps.is_available():\n",
    "#    device = torch.device(\"mps\")\n",
    "#else:\n",
    "#    device = torch.device(\"cpu\")\n",
    "\n",
    "print(\"Device:\", device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4e47fb30-c2c6-4e6d-a64c-76cc65be4a2c",
   "metadata": {
    "id": "4e47fb30-c2c6-4e6d-a64c-76cc65be4a2c"
   },
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "\n",
    "customized_collate_fn = partial(\n",
    "    custom_collate_fn,\n",
    "    device=device,\n",
    "    allowed_max_length=1024\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ff42c29-8b81-45e5-ae8d-b97cd1cf447a",
   "metadata": {
    "id": "8ff42c29-8b81-45e5-ae8d-b97cd1cf447a"
   },
   "source": [
    "- Next, we instantiate the data loaders similar to previous chapters, except that we now provide our own collate function for the batching process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "BtWkgir6Hlpe",
   "metadata": {
    "id": "BtWkgir6Hlpe"
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "num_workers = 0\n",
    "batch_size = 8\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "train_dataset = InstructionDataset(train_data, tokenizer)\n",
    "train_loader = DataLoader(\n",
    "    train_dataset,\n",
    "    batch_size=batch_size,\n",
    "    collate_fn=customized_collate_fn,\n",
    "    shuffle=True,\n",
    "    drop_last=True,\n",
    "    num_workers=num_workers\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "1d097dc8-ad34-4f05-b435-e4147965f532",
   "metadata": {
    "id": "1d097dc8-ad34-4f05-b435-e4147965f532"
   },
   "outputs": [],
   "source": [
    "val_dataset = InstructionDataset(val_data, tokenizer)\n",
    "val_loader = DataLoader(\n",
    "    val_dataset,\n",
    "    batch_size=batch_size,\n",
    "    collate_fn=customized_collate_fn,\n",
    "    shuffle=False,\n",
    "    drop_last=False,\n",
    "    num_workers=num_workers\n",
    ")\n",
    "\n",
    "test_dataset = InstructionDataset(test_data, tokenizer)\n",
    "test_loader = DataLoader(\n",
    "    test_dataset,\n",
    "    batch_size=batch_size,\n",
    "    collate_fn=customized_collate_fn,\n",
    "    shuffle=False,\n",
    "    drop_last=False,\n",
    "    num_workers=num_workers\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f67c147-b1a2-4a95-9807-e2d0de0324c0",
   "metadata": {
    "id": "3f67c147-b1a2-4a95-9807-e2d0de0324c0"
   },
   "source": [
    "- Let's see what the dimensions of the resulting input and target batches look like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "GGs1AI3vHpnX",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GGs1AI3vHpnX",
    "outputId": "f6a74c8b-1af3-4bc1-b48c-eda64b0200d1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train loader:\n",
      "torch.Size([8, 61]) torch.Size([8, 61])\n",
      "torch.Size([8, 76]) torch.Size([8, 76])\n",
      "torch.Size([8, 73]) torch.Size([8, 73])\n",
      "torch.Size([8, 68]) torch.Size([8, 68])\n",
      "torch.Size([8, 65]) torch.Size([8, 65])\n",
      "torch.Size([8, 72]) torch.Size([8, 72])\n",
      "torch.Size([8, 80]) torch.Size([8, 80])\n",
      "torch.Size([8, 67]) torch.Size([8, 67])\n",
      "torch.Size([8, 62]) torch.Size([8, 62])\n",
      "torch.Size([8, 75]) torch.Size([8, 75])\n",
      "torch.Size([8, 62]) torch.Size([8, 62])\n",
      "torch.Size([8, 68]) torch.Size([8, 68])\n",
      "torch.Size([8, 67]) torch.Size([8, 67])\n",
      "torch.Size([8, 77]) torch.Size([8, 77])\n",
      "torch.Size([8, 69]) torch.Size([8, 69])\n",
      "torch.Size([8, 79]) torch.Size([8, 79])\n",
      "torch.Size([8, 71]) torch.Size([8, 71])\n",
      "torch.Size([8, 66]) torch.Size([8, 66])\n",
      "torch.Size([8, 83]) torch.Size([8, 83])\n",
      "torch.Size([8, 68]) torch.Size([8, 68])\n",
      "torch.Size([8, 80]) torch.Size([8, 80])\n",
      "torch.Size([8, 71]) torch.Size([8, 71])\n",
      "torch.Size([8, 69]) torch.Size([8, 69])\n",
      "torch.Size([8, 65]) torch.Size([8, 65])\n",
      "torch.Size([8, 68]) torch.Size([8, 68])\n",
      "torch.Size([8, 60]) torch.Size([8, 60])\n",
      "torch.Size([8, 59]) torch.Size([8, 59])\n",
      "torch.Size([8, 69]) torch.Size([8, 69])\n",
      "torch.Size([8, 63]) torch.Size([8, 63])\n",
      "torch.Size([8, 65]) torch.Size([8, 65])\n",
      "torch.Size([8, 76]) torch.Size([8, 76])\n",
      "torch.Size([8, 66]) torch.Size([8, 66])\n",
      "torch.Size([8, 71]) torch.Size([8, 71])\n",
      "torch.Size([8, 91]) torch.Size([8, 91])\n",
      "torch.Size([8, 65]) torch.Size([8, 65])\n",
      "torch.Size([8, 64]) torch.Size([8, 64])\n",
      "torch.Size([8, 67]) torch.Size([8, 67])\n",
      "torch.Size([8, 66]) torch.Size([8, 66])\n",
      "torch.Size([8, 64]) torch.Size([8, 64])\n",
      "torch.Size([8, 65]) torch.Size([8, 65])\n",
      "torch.Size([8, 75]) torch.Size([8, 75])\n",
      "torch.Size([8, 89]) torch.Size([8, 89])\n",
      "torch.Size([8, 59]) torch.Size([8, 59])\n",
      "torch.Size([8, 88]) torch.Size([8, 88])\n",
      "torch.Size([8, 83]) torch.Size([8, 83])\n",
      "torch.Size([8, 83]) torch.Size([8, 83])\n",
      "torch.Size([8, 70]) torch.Size([8, 70])\n",
      "torch.Size([8, 65]) torch.Size([8, 65])\n",
      "torch.Size([8, 74]) torch.Size([8, 74])\n",
      "torch.Size([8, 76]) torch.Size([8, 76])\n",
      "torch.Size([8, 67]) torch.Size([8, 67])\n",
      "torch.Size([8, 75]) torch.Size([8, 75])\n",
      "torch.Size([8, 83]) torch.Size([8, 83])\n",
      "torch.Size([8, 69]) torch.Size([8, 69])\n",
      "torch.Size([8, 67]) torch.Size([8, 67])\n",
      "torch.Size([8, 60]) torch.Size([8, 60])\n",
      "torch.Size([8, 60]) torch.Size([8, 60])\n",
      "torch.Size([8, 66]) torch.Size([8, 66])\n",
      "torch.Size([8, 80]) torch.Size([8, 80])\n",
      "torch.Size([8, 71]) torch.Size([8, 71])\n",
      "torch.Size([8, 61]) torch.Size([8, 61])\n",
      "torch.Size([8, 58]) torch.Size([8, 58])\n",
      "torch.Size([8, 71]) torch.Size([8, 71])\n",
      "torch.Size([8, 67]) torch.Size([8, 67])\n",
      "torch.Size([8, 68]) torch.Size([8, 68])\n",
      "torch.Size([8, 63]) torch.Size([8, 63])\n",
      "torch.Size([8, 87]) torch.Size([8, 87])\n",
      "torch.Size([8, 68]) torch.Size([8, 68])\n",
      "torch.Size([8, 64]) torch.Size([8, 64])\n",
      "torch.Size([8, 68]) torch.Size([8, 68])\n",
      "torch.Size([8, 71]) torch.Size([8, 71])\n",
      "torch.Size([8, 68]) torch.Size([8, 68])\n",
      "torch.Size([8, 71]) torch.Size([8, 71])\n",
      "torch.Size([8, 61]) torch.Size([8, 61])\n",
      "torch.Size([8, 65]) torch.Size([8, 65])\n",
      "torch.Size([8, 67]) torch.Size([8, 67])\n",
      "torch.Size([8, 65]) torch.Size([8, 65])\n",
      "torch.Size([8, 64]) torch.Size([8, 64])\n",
      "torch.Size([8, 60]) torch.Size([8, 60])\n",
      "torch.Size([8, 72]) torch.Size([8, 72])\n",
      "torch.Size([8, 64]) torch.Size([8, 64])\n",
      "torch.Size([8, 70]) torch.Size([8, 70])\n",
      "torch.Size([8, 57]) torch.Size([8, 57])\n",
      "torch.Size([8, 72]) torch.Size([8, 72])\n",
      "torch.Size([8, 64]) torch.Size([8, 64])\n",
      "torch.Size([8, 68]) torch.Size([8, 68])\n",
      "torch.Size([8, 62]) torch.Size([8, 62])\n",
      "torch.Size([8, 74]) torch.Size([8, 74])\n",
      "torch.Size([8, 80]) torch.Size([8, 80])\n",
      "torch.Size([8, 68]) torch.Size([8, 68])\n",
      "torch.Size([8, 70]) torch.Size([8, 70])\n",
      "torch.Size([8, 91]) torch.Size([8, 91])\n",
      "torch.Size([8, 61]) torch.Size([8, 61])\n",
      "torch.Size([8, 66]) torch.Size([8, 66])\n",
      "torch.Size([8, 80]) torch.Size([8, 80])\n",
      "torch.Size([8, 81]) torch.Size([8, 81])\n",
      "torch.Size([8, 74]) torch.Size([8, 74])\n",
      "torch.Size([8, 82]) torch.Size([8, 82])\n",
      "torch.Size([8, 63]) torch.Size([8, 63])\n",
      "torch.Size([8, 83]) torch.Size([8, 83])\n",
      "torch.Size([8, 68]) torch.Size([8, 68])\n",
      "torch.Size([8, 67]) torch.Size([8, 67])\n",
      "torch.Size([8, 77]) torch.Size([8, 77])\n",
      "torch.Size([8, 91]) torch.Size([8, 91])\n",
      "torch.Size([8, 64]) torch.Size([8, 64])\n",
      "torch.Size([8, 61]) torch.Size([8, 61])\n",
      "torch.Size([8, 75]) torch.Size([8, 75])\n",
      "torch.Size([8, 64]) torch.Size([8, 64])\n",
      "torch.Size([8, 66]) torch.Size([8, 66])\n",
      "torch.Size([8, 78]) torch.Size([8, 78])\n",
      "torch.Size([8, 66]) torch.Size([8, 66])\n",
      "torch.Size([8, 64]) torch.Size([8, 64])\n",
      "torch.Size([8, 83]) torch.Size([8, 83])\n",
      "torch.Size([8, 66]) torch.Size([8, 66])\n",
      "torch.Size([8, 74]) torch.Size([8, 74])\n",
      "torch.Size([8, 69]) torch.Size([8, 69])\n"
     ]
    }
   ],
   "source": [
    "print(\"Train loader:\")\n",
    "for inputs, targets in train_loader:\n",
    "    print(inputs.shape, targets.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c8e8dd7-d46a-4cc3-8a7e-c1d31e1b4657",
   "metadata": {
    "id": "0c8e8dd7-d46a-4cc3-8a7e-c1d31e1b4657"
   },
   "source": [
    "- As we can see based on the output above, all batches have a batch size of 8 but a different length, as expected\n",
    "- Let's also double-check that the inputs contain the `<|endoftext|>` padding tokens corresponding to token ID 50256 by printing the contents of the first training example in the `inputs` batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "21b8fd02-014f-4481-9b71-5bfee8f9dfcd",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "21b8fd02-014f-4481-9b71-5bfee8f9dfcd",
    "outputId": "1b8ad342-2b5b-4f12-ad1a-3cb2a6c712ff"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([21106,   318,   281, 12064,   326,  8477,   257,  4876,    13, 19430,\n",
      "          257,  2882,   326, 20431, 32543,   262,  2581,    13,   198,   198,\n",
      "        21017, 46486,    25,   198, 30003,  6525,   262,  6827,  1262,   257,\n",
      "          985,   576,    13,   198,   198, 21017, 23412,    25,   198,   464,\n",
      "         5156,   318,   845, 13779,    13,   198,   198, 21017, 18261,    25,\n",
      "          198,   464,  5156,   318,   355, 13779,   355,   257,  4936,    13,\n",
      "        50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256],\n",
      "       device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "print(inputs[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f1f3647-8971-4006-89e0-6a2a1ec1d360",
   "metadata": {
    "id": "5f1f3647-8971-4006-89e0-6a2a1ec1d360"
   },
   "source": [
    "- Similarly, we visually double-check that the targets contain the -100 placeholder tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "51649ab4-1a7e-4a9e-92c5-950a24fde211",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "51649ab4-1a7e-4a9e-92c5-950a24fde211",
    "outputId": "5e8c23f8-6a05-4c13-9f92-373b75b57ea6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([  318,   281, 12064,   326,  8477,   257,  4876,    13, 19430,   257,\n",
      "         2882,   326, 20431, 32543,   262,  2581,    13,   198,   198, 21017,\n",
      "        46486,    25,   198, 30003,  6525,   262,  6827,  1262,   257,   985,\n",
      "          576,    13,   198,   198, 21017, 23412,    25,   198,   464,  5156,\n",
      "          318,   845, 13779,    13,   198,   198, 21017, 18261,    25,   198,\n",
      "          464,  5156,   318,   355, 13779,   355,   257,  4936,    13, 50256,\n",
      "         -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100],\n",
      "       device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "print(targets[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6aad445-8f19-4238-b9bf-db80767fb91a",
   "metadata": {
    "id": "d6aad445-8f19-4238-b9bf-db80767fb91a"
   },
   "source": [
    "## 7.5 Loading a pretrained LLM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a5c07d1-4fc9-4846-94cf-b11a085a667b",
   "metadata": {
    "id": "5a5c07d1-4fc9-4846-94cf-b11a085a667b"
   },
   "source": [
    "- In this section, we load a pretrained GPT model using the same code that we used in section 5.5 of chapter 5 and section 6.4 in chapter 6"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d1b438f-88af-413f-96a9-f059c6c55fc4",
   "metadata": {
    "id": "8d1b438f-88af-413f-96a9-f059c6c55fc4"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-4.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c68eda7-e02e-4caa-846b-ca6dbd396ca2",
   "metadata": {
    "id": "8c68eda7-e02e-4caa-846b-ca6dbd396ca2"
   },
   "source": [
    "- However, instead of loading the smallest 124 million parameter model, we load the medium version with 355 million parameters since the 124 million model is too small for achieving qualitatively reasonable results via instruction finetuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "0d249d67-5eba-414e-9bd2-972ebf01329d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0d249d67-5eba-414e-9bd2-972ebf01329d",
    "outputId": "386ebd49-51d7-4a62-c590-91cdccce5fb8"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025-02-08 23:37:19.420934: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",
      "2025-02-08 23:37:19.439459: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
      "E0000 00:00:1739057839.462005    4402 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "E0000 00:00:1739057839.468845    4402 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2025-02-08 23:37:19.492335: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "checkpoint: 100%|██████████| 77.0/77.0 [00:00<00:00, 125kiB/s]\n",
      "encoder.json: 100%|██████████| 1.04M/1.04M [00:00<00:00, 5.37MiB/s]\n",
      "hparams.json: 100%|██████████| 91.0/91.0 [00:00<00:00, 161kiB/s]\n",
      "model.ckpt.data-00000-of-00001: 100%|██████████| 1.42G/1.42G [01:00<00:00, 23.6MiB/s]\n",
      "model.ckpt.index: 100%|██████████| 10.4k/10.4k [00:00<00:00, 17.5MiB/s]\n",
      "model.ckpt.meta: 100%|██████████| 927k/927k [00:00<00:00, 6.38MiB/s]\n",
      "vocab.bpe: 100%|██████████| 456k/456k [00:00<00:00, 2.69MiB/s]\n"
     ]
    }
   ],
   "source": [
    "from gpt_download import download_and_load_gpt2\n",
    "from previous_chapters import GPTModel, load_weights_into_gpt\n",
    "# If the `previous_chapters.py` file is not available locally,\n",
    "# you can import it from the `llms-from-scratch` PyPI package.\n",
    "# For details, see: https://github.com/rasbt/LLMs-from-scratch/tree/main/pkg\n",
    "# E.g.,\n",
    "# from llms_from_scratch.ch04 import GPTModel\n",
    "# from llms_from_scratch.ch05 import download_and_load_gpt2, load_weights_into_gpt\n",
    "\n",
    "\n",
    "BASE_CONFIG = {\n",
    "    \"vocab_size\": 50257,     # Vocabulary size\n",
    "    \"context_length\": 1024,  # Context length\n",
    "    \"drop_rate\": 0.0,        # Dropout rate\n",
    "    \"qkv_bias\": True         # Query-key-value bias\n",
    "}\n",
    "\n",
    "model_configs = {\n",
    "    \"gpt2-small (124M)\": {\"emb_dim\": 768, \"n_layers\": 12, \"n_heads\": 12},\n",
    "    \"gpt2-medium (355M)\": {\"emb_dim\": 1024, \"n_layers\": 24, \"n_heads\": 16},\n",
    "    \"gpt2-large (774M)\": {\"emb_dim\": 1280, \"n_layers\": 36, \"n_heads\": 20},\n",
    "    \"gpt2-xl (1558M)\": {\"emb_dim\": 1600, \"n_layers\": 48, \"n_heads\": 25},\n",
    "}\n",
    "\n",
    "CHOOSE_MODEL = \"gpt2-medium (355M)\"\n",
    "\n",
    "BASE_CONFIG.update(model_configs[CHOOSE_MODEL])\n",
    "\n",
    "model_size = CHOOSE_MODEL.split(\" \")[-1].lstrip(\"(\").rstrip(\")\")\n",
    "settings, params = download_and_load_gpt2(\n",
    "    model_size=model_size,\n",
    "    models_dir=\"gpt2\"\n",
    ")\n",
    "\n",
    "model = GPTModel(BASE_CONFIG)\n",
    "load_weights_into_gpt(model, params)\n",
    "model.eval();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbf3afed-bc8e-4d3a-ad9d-eb6f57bb7af5",
   "metadata": {
    "id": "dbf3afed-bc8e-4d3a-ad9d-eb6f57bb7af5"
   },
   "source": [
    "- Before we start finetuning the model in the next section, let's see how it performs on one of the validation tasks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "7bd32b7c-5b44-4d25-a09f-46836802ca74",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7bd32b7c-5b44-4d25-a09f-46836802ca74",
    "outputId": "c1276a91-e7da-495b-be0f-70a96872dbe6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
      "\n",
      "### Instruction:\n",
      "Convert the active sentence to passive: 'The chef cooks the meal every day.'\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "input_text = format_input(val_data[0])\n",
    "print(input_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "2e3e68e0-2627-4c65-b4e7-1e0667e4f6fa",
   "metadata": {
    "id": "2e3e68e0-2627-4c65-b4e7-1e0667e4f6fa"
   },
   "outputs": [],
   "source": [
    "from previous_chapters import (\n",
    "    generate,\n",
    "    text_to_token_ids,\n",
    "    token_ids_to_text\n",
    ")\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch05 import (\n",
    "#    generate_text_simple,\n",
    "#    text_to_token_ids,\n",
    "#    token_ids_to_text\n",
    "# )\n",
    "\n",
    "\n",
    "token_ids = generate(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(input_text, tokenizer),\n",
    "    max_new_tokens=35,\n",
    "    context_size=BASE_CONFIG[\"context_length\"],\n",
    "    eos_id=50256,\n",
    ")\n",
    "generated_text = token_ids_to_text(token_ids, tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36e2fda5-f796-4954-8f72-1dd1123e3344",
   "metadata": {
    "id": "36e2fda5-f796-4954-8f72-1dd1123e3344"
   },
   "source": [
    "- Note that the `generate` function we used in previous chapters returns the combined input and output text, which was convenient in the previous section for creating legible text\n",
    "- To isolate the response, we can subtract the length of the instruction from the start of the `generated_text`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "ba4a55bf-a245-48d8-beda-2838a58fb5ba",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ba4a55bf-a245-48d8-beda-2838a58fb5ba",
    "outputId": "3e231f03-c5dc-4397-8778-4995731176a3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The chef cooks the meal every day.\n",
      "\n",
      "### Instruction:\n",
      "\n",
      "Convert the active sentence to passive: 'The chef cooks the\n"
     ]
    }
   ],
   "source": [
    "response_text = (\n",
    "    generated_text[len(input_text):]\n",
    "    .replace(\"### Response:\", \"\")\n",
    "    .strip()\n",
    ")\n",
    "print(response_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44080b2-a4c5-4520-a797-549519f66a3e",
   "metadata": {
    "id": "d44080b2-a4c5-4520-a797-549519f66a3e"
   },
   "source": [
    "- As we can see, the model is not capable of following the instructions, yet; it creates a \"Response\" section but it simply repeats the original input sentence as well as the instruction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70d27b9d-a942-4cf5-b797-848c5f01e723",
   "metadata": {
    "id": "70d27b9d-a942-4cf5-b797-848c5f01e723"
   },
   "source": [
    "## 7.6 Finetuning the LLM on instruction data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "314b2a39-88b4-44d8-8c85-1c5b0cd6cc4a",
   "metadata": {
    "id": "314b2a39-88b4-44d8-8c85-1c5b0cd6cc4a"
   },
   "source": [
    "- In this section, we finetune the model\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-5.webp?1\" width=500px>\n",
    "\n",
    "- Note that we can reuse all the loss calculation and training functions that we used in previous chapters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "65444865-df87-4d98-9faf-875e1c4be860",
   "metadata": {
    "id": "65444865-df87-4d98-9faf-875e1c4be860"
   },
   "outputs": [],
   "source": [
    "from previous_chapters import (\n",
    "    calc_loss_loader,\n",
    "    train_model_simple\n",
    ")\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch05 import (\n",
    "#    calc_loss_loader,\n",
    "#    train_model_simple,\n",
    "# )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00083059-aa41-4d37-8a17-1c72d1b1ca00",
   "metadata": {
    "id": "00083059-aa41-4d37-8a17-1c72d1b1ca00"
   },
   "source": [
    "- Let's calculate the initial training and validation set loss before we start training (as in previous chapters, the goal is to minimize the loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "d99fc6f8-63b2-43da-adbb-a7b6b92c8dd5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "d99fc6f8-63b2-43da-adbb-a7b6b92c8dd5",
    "outputId": "a3f5e1b0-093a-4c51-e7fc-c9cac48c2ea2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 3.8259087562561036\n",
      "Validation loss: 3.761933708190918\n"
     ]
    }
   ],
   "source": [
    "model.to(device)\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "with torch.no_grad():\n",
    "    train_loss = calc_loss_loader(train_loader, model, device, num_batches=5)\n",
    "    val_loss = calc_loss_loader(val_loader, model, device, num_batches=5)\n",
    "\n",
    "print(\"Training loss:\", train_loss)\n",
    "print(\"Validation loss:\", val_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12a6da8f-15b3-42b0-a136-619b7a35c3e9",
   "metadata": {
    "id": "12a6da8f-15b3-42b0-a136-619b7a35c3e9"
   },
   "source": [
    "- Note that the training is a bit more expensive than in previous chapters since we are using a larger model (355 million instead of 124 million parameters)\n",
    "- The runtimes for various devices are shown for reference below (running this notebook on a compatible GPU device requires no changes to the code)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db4b57fb-e689-4550-931c-6d34a932487c",
   "metadata": {
    "id": "db4b57fb-e689-4550-931c-6d34a932487c"
   },
   "source": [
    "<div style=\"text-align: left;\">\n",
    "    \n",
    "| Model              | Device                | Runtime for 2 Epochs |\n",
    "|--------------------|-----------------------|----------------------|\n",
    "| gpt2-medium (355M) | CPU (M3 MacBook Air)  | 15.78 minutes        |\n",
    "| gpt2-medium (355M) | GPU (M3 MacBook Air)  | 10.77 minutes        |\n",
    "| gpt2-medium (355M) | GPU (L4)              | 1.83 minutes         |\n",
    "| gpt2-medium (355M) | GPU (A100)            | 0.86 minutes         |\n",
    "| gpt2-small (124M)  | CPU (M3 MacBook Air)  | 5.74 minutes         |\n",
    "| gpt2-small (124M)  | GPU (M3 MacBook Air)  | 3.73 minutes         |\n",
    "| gpt2-small (124M)  | GPU (L4)              | 0.69 minutes         |\n",
    "| gpt2-small (124M)  | GPU (A100)            | 0.39 minutes         |\n",
    "\n",
    "</div>\n",
    "\n",
    "- I ran this notebook using the `\"gpt2-medium (355M)\"` model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "78bcf83a-1fff-4540-97c1-765c4016d5e3",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "78bcf83a-1fff-4540-97c1-765c4016d5e3",
    "outputId": "ecb9a3dd-97c0-492d-8a51-fbd175bb139b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep 1 (Step 000000): Train loss 2.637, Val loss 2.626\n",
      "Ep 1 (Step 000005): Train loss 1.174, Val loss 1.103\n",
      "Ep 1 (Step 000010): Train loss 0.872, Val loss 0.944\n",
      "Ep 1 (Step 000015): Train loss 0.857, Val loss 0.906\n",
      "Ep 1 (Step 000020): Train loss 0.776, Val loss 0.881\n",
      "Ep 1 (Step 000025): Train loss 0.754, Val loss 0.859\n",
      "Ep 1 (Step 000030): Train loss 0.800, Val loss 0.836\n",
      "Ep 1 (Step 000035): Train loss 0.714, Val loss 0.809\n",
      "Ep 1 (Step 000040): Train loss 0.672, Val loss 0.806\n",
      "Ep 1 (Step 000045): Train loss 0.633, Val loss 0.789\n",
      "Ep 1 (Step 000050): Train loss 0.663, Val loss 0.782\n",
      "Ep 1 (Step 000055): Train loss 0.760, Val loss 0.763\n",
      "Ep 1 (Step 000060): Train loss 0.719, Val loss 0.743\n",
      "Ep 1 (Step 000065): Train loss 0.653, Val loss 0.735\n",
      "Ep 1 (Step 000070): Train loss 0.536, Val loss 0.732\n",
      "Ep 1 (Step 000075): Train loss 0.569, Val loss 0.739\n",
      "Ep 1 (Step 000080): Train loss 0.603, Val loss 0.734\n",
      "Ep 1 (Step 000085): Train loss 0.518, Val loss 0.717\n",
      "Ep 1 (Step 000090): Train loss 0.575, Val loss 0.699\n",
      "Ep 1 (Step 000095): Train loss 0.505, Val loss 0.689\n",
      "Ep 1 (Step 000100): Train loss 0.507, Val loss 0.683\n",
      "Ep 1 (Step 000105): Train loss 0.570, Val loss 0.676\n",
      "Ep 1 (Step 000110): Train loss 0.564, Val loss 0.671\n",
      "Ep 1 (Step 000115): Train loss 0.522, Val loss 0.666\n",
      "Below is an instruction that describes a task. Write a response that appropriately completes the request.  ### Instruction: Convert the active sentence to passive: 'The chef cooks the meal every day.'  ### Response: The meal is prepared every day by the chef.<|endoftext|>The following is an instruction that describes a task. Write a response that appropriately completes the request.  ### Instruction: Convert the active sentence to passive:\n",
      "Ep 2 (Step 000120): Train loss 0.439, Val loss 0.671\n",
      "Ep 2 (Step 000125): Train loss 0.454, Val loss 0.685\n",
      "Ep 2 (Step 000130): Train loss 0.448, Val loss 0.681\n",
      "Ep 2 (Step 000135): Train loss 0.406, Val loss 0.678\n",
      "Ep 2 (Step 000140): Train loss 0.412, Val loss 0.678\n",
      "Ep 2 (Step 000145): Train loss 0.372, Val loss 0.680\n",
      "Ep 2 (Step 000150): Train loss 0.381, Val loss 0.674\n",
      "Ep 2 (Step 000155): Train loss 0.419, Val loss 0.672\n",
      "Ep 2 (Step 000160): Train loss 0.417, Val loss 0.680\n",
      "Ep 2 (Step 000165): Train loss 0.383, Val loss 0.683\n",
      "Ep 2 (Step 000170): Train loss 0.328, Val loss 0.679\n",
      "Ep 2 (Step 000175): Train loss 0.334, Val loss 0.668\n",
      "Ep 2 (Step 000180): Train loss 0.391, Val loss 0.656\n",
      "Ep 2 (Step 000185): Train loss 0.418, Val loss 0.657\n",
      "Ep 2 (Step 000190): Train loss 0.341, Val loss 0.648\n",
      "Ep 2 (Step 000195): Train loss 0.330, Val loss 0.633\n",
      "Ep 2 (Step 000200): Train loss 0.313, Val loss 0.631\n",
      "Ep 2 (Step 000205): Train loss 0.354, Val loss 0.628\n",
      "Ep 2 (Step 000210): Train loss 0.365, Val loss 0.629\n",
      "Ep 2 (Step 000215): Train loss 0.394, Val loss 0.634\n",
      "Ep 2 (Step 000220): Train loss 0.301, Val loss 0.647\n",
      "Ep 2 (Step 000225): Train loss 0.347, Val loss 0.661\n",
      "Ep 2 (Step 000230): Train loss 0.297, Val loss 0.659\n",
      "Below is an instruction that describes a task. Write a response that appropriately completes the request.  ### Instruction: Convert the active sentence to passive: 'The chef cooks the meal every day.'  ### Response: The meal is cooked every day by the chef.<|endoftext|>The following is an instruction that describes a task. Write a response that appropriately completes the request.  ### Instruction: What is the capital of the United Kingdom\n",
      "Training completed in 0.93 minutes.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=0.00005, weight_decay=0.1)\n",
    "\n",
    "num_epochs = 2\n",
    "\n",
    "train_losses, val_losses, tokens_seen = train_model_simple(\n",
    "    model, train_loader, val_loader, optimizer, device,\n",
    "    num_epochs=num_epochs, eval_freq=5, eval_iter=5,\n",
    "    start_context=format_input(val_data[0]), tokenizer=tokenizer\n",
    ")\n",
    "\n",
    "end_time = time.time()\n",
    "execution_time_minutes = (end_time - start_time) / 60\n",
    "print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Ise3wGjlB-iq",
   "metadata": {
    "id": "Ise3wGjlB-iq"
   },
   "source": [
    "- As we can see based on the outputs above, the model trains well, as we can tell based on the decreasing training loss and validation loss values\n",
    "- Furthermore, based on the response text printed after each epoch, we can see that the model correctly follows the instruction to convert the input sentence `'The chef cooks the meal every day.'` into passive voice `'The meal is cooked every day by the chef.'` (We will properly format and evaluate the responses in a later section)\n",
    "- Finally, let's take a look at the training and validation loss curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "4acd368b-1403-4807-a218-9102e35bfdbb",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "id": "4acd368b-1403-4807-a218-9102e35bfdbb",
    "outputId": "2f5c99e0-7ed0-4f42-d67c-e07c375e6158"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAWdFJREFUeJzt3Xd4VFX6wPHvTMokk94LIRAgkgAhhCpEBAWpohTFRVbAgqtSZFF0+aEIuooKKiqIbSW7KoKIICLF0BWQHnpvoSShpPdk5vz+GJgwlJAyYZLwfp7nPpm599x73zOEvHPuPfccjVJKIYQQQohqSWvrAIQQQghxc5KohRBCiGpMErUQQghRjUmiFkIIIaoxSdRCCCFENSaJWgghhKjGJFELIYQQ1ZgkaiGEEKIak0QthBBCVGOSqIWoRU6ePIlGoyEhIcHWoQghrEQStRDVjEajKXWZNGmSrUMUQtxG9rYOQAhhKSkpyfx63rx5TJw4kUOHDpnXubq62iIsIYSNSItaiGomMDDQvHh4eKDRaMzv/f39+fDDDwkJCUGn09GiRQuWL19+02MZDAaeeuopIiIiSExMBOCXX36hZcuWODk50aBBAyZPnkxxcbF5H41Gw9dff02/fv3Q6/WEh4ezePFi8/a0tDQGDx6Mn58fzs7OhIeHM3v27JvG8NNPPxEVFYWzszM+Pj507dqVnJwc8/avv/6ayMhInJyciIiI4LPPPrPY//Tp0wwcOBBPT0+8vb15+OGHOXnypHn7sGHD6Nu3L9OmTSMoKAgfHx9GjBhBUVFRmT9zIao1JYSotmbPnq08PDzM7z/88EPl7u6ufvjhB3Xw4EH1yiuvKAcHB3X48GGllFInTpxQgNq5c6fKz89X/fr1UzExMer8+fNKKaXWr1+v3N3dVVxcnDp27Jj6/fffVf369dWkSZPM5wBUSEiImjNnjjpy5IgaPXq0cnV1VZcuXVJKKTVixAjVokULtXXrVnXixAkVHx+vFi9efMP4z507p+zt7dWHH36oTpw4oXbv3q1mzpypsrKylFJKfffddyooKEgtWLBAHT9+XC1YsEB5e3uruLg4pZRShYWFKjIyUj311FNq9+7dav/+/erxxx9XjRs3VgUFBUoppYYOHarc3d3Vc889pw4cOKB+/fVXpdfr1ZdffmndfwwhbEQStRDV2LWJOjg4WL399tsWZdq0aaNeeOEFpVRJov7jjz9Uly5d1D333KPS09PNZbt06aLeeecdi/2//fZbFRQUZH4PqNdee838Pjs7WwFq2bJlSiml+vTpo5588skyxb99+3YFqJMnT95we8OGDdWcOXMs1r311luqffv25tgaN26sjEajeXtBQYFydnZWK1asUEqZEnW9evVUcXGxucyjjz6qHnvssTLFKER1J/eohaghMjMzOXfuHLGxsRbrY2Nj2bVrl8W6QYMGERISwurVq3F2djav37VrFxs2bODtt982rzMYDOTn55Obm4terwegefPm5u0uLi64u7tz/vx5AJ5//nkGDBjAjh076NatG3379qVDhw43jDk6OpouXboQFRVF9+7d6datG4888gheXl7k5ORw7Ngxnn76aYYPH27ep7i4GA8PD3O8R48exc3NzeK4+fn5HDt2zPy+adOm2NnZmd8HBQWxZ8+eUj5NIWoOSdRC1EK9evXiu+++Y9OmTdx///3m9dnZ2UyePJn+/ftft4+Tk5P5tYODg8U2jUaD0WgEoGfPnpw6dYqlS5cSHx9Ply5dGDFiBNOmTbvumHZ2dsTHx7Nx40Z+//13Pv30UyZMmMDmzZvNXwq++uor2rVrd91+V+Jt1aoV33///XXH9vPzK1O8QtR0kqiFqCHc3d0JDg5mw4YNdOrUybx+w4YNtG3b1qLs888/T7NmzXjooYf47bffzOVbtmzJoUOHaNSoUaVi8fPzY+jQoQwdOpSOHTsybty4GyZqMCXN2NhYYmNjmThxIvXq1WPhwoWMHTuW4OBgjh8/zuDBg2+4b8uWLZk3bx7+/v64u7tXKmYhaipJ1ELUIOPGjeONN96gYcOGtGjRgtmzZ5OQkHDDFueoUaMwGAw8+OCDLFu2jHvuuYeJEyfy4IMPEhoayiOPPIJWq2XXrl3s3buXf//732WKYeLEibRq1YqmTZtSUFDAkiVLiIyMvGHZzZs3s2rVKrp164a/vz+bN2/mwoUL5vKTJ09m9OjReHh40KNHDwoKCti2bRtpaWmMHTuWwYMHM3XqVB5++GHefPNNQkJCOHXqFD///DOvvPIKISEhFf8whaghJFELUYOMHj2ajIwMXnrpJc6fP0+TJk1YvHgx4eHhNyw/ZswYjEYjvXr1Yvny5XTv3p0lS5bw5ptv8t577+Hg4EBERATPPPNMmWNwdHRk/PjxnDx5EmdnZzp27MjcuXNvWNbd3Z3169czffp0MjMzqVevHh988AE9e/YE4JlnnkGv1zN16lTGjRuHi4sLUVFRjBkzBgC9Xs/69et59dVX6d+/P1lZWdSpU4cuXbpIC1vcMTRKKWXrIIQQQghxYzLgiRBCCFGNSaIWQgghqjFJ1EIIIUQ1JolaCCGEqMYkUQshhBDVmCRqIYQQohqTRF0BM2fOpH79+jg5OdGuXTu2bNli65AsTJkyhTZt2uDm5oa/vz99+/a1mM8YTGMljxgxAh8fH1xdXRkwYAApKSkWZRITE+nduzd6vR5/f3/GjRtnMR0iwNq1a2nZsiU6nY5GjRoRFxd3XTy38/N699130Wg05udwofbV9ezZs/z973/Hx8cHZ2dnoqKi2LZtm3m7UoqJEycSFBSEs7MzXbt25ciRIxbHSE1NZfDgwbi7u+Pp6cnTTz9Ndna2RZndu3fTsWNHnJycqFu3Lu+///51scyfP5+IiAicnJyIiopi6dKlVqunwWDg9ddfJywsDGdnZxo2bMhbb73F1U+U1uS6rl+/nj59+hAcHIxGo2HRokUW26tT3coSS0XrWlRUxKuvvkpUVBQuLi4EBwczZMgQzp07VyPrWiVsNx9IzTR37lzl6OiovvnmG7Vv3z41fPhw5enpqVJSUmwdmln37t3V7Nmz1d69e1VCQoLq1auXCg0NVdnZ2eYyzz33nKpbt65atWqV2rZtm7r77rtVhw4dzNuLi4tVs2bNVNeuXdXOnTvV0qVLla+vrxo/fry5zPHjx5Ver1djx45V+/fvV59++qmys7NTy5cvN5e5nZ/Xli1bVP369VXz5s3Viy++WCvrmpqaqurVq6eGDRumNm/erI4fP65WrFihjh49ai7z7rvvKg8PD7Vo0SK1a9cu9dBDD6mwsDCVl5dnLtOjRw8VHR2t/vrrL/XHH3+oRo0aqUGDBpm3Z2RkqICAADV48GC1d+9e9cMPPyhnZ2f1xRdfmMts2LBB2dnZqffff1/t379fvfbaa8rBwUHt2bPHKnV9++23lY+Pj1qyZIk6ceKEmj9/vnJ1dVUff/xxrajr0qVL1YQJE9TPP/+sALVw4UKL7dWpbmWJpaJ1TU9PV127dlXz5s1TBw8eVJs2bVJt27ZVrVq1sjhGTalrVZBEXU5t27ZVI0aMML83GAwqODhYTZkyxYZRle78+fMKUOvWrVNKmf5jODg4qPnz55vLHDhwQAFq06ZNSinTfyytVquSk5PNZWbNmqXc3d3N8wC/8sorqmnTphbneuyxx1T37t3N72/X55WVlaXCw8NVfHy86tSpkzlR17a6vvrqq+qee+656Xaj0agCAwPV1KlTzevS09OVTqdTP/zwg1JKqf379ytAbd261Vxm2bJlSqPRqLNnzyqllPrss8+Ul5eXuf5Xzt24cWPz+4EDB6revXtbnL9du3bqH//4R+UqeVnv3r3VU089ZbGuf//+avDgwbWurtcmr+pUt7LEUpm63siWLVsUoE6dOlWj62otcum7HAoLC9m+fTtdu3Y1r9NqtXTt2pVNmzbZMLLSZWRkAODt7Q3A9u3bKSoqsqhHREQEoaGh5nps2rSJqKgoAgICzGW6d+9OZmYm+/btM5e5+hhXylw5xu38vEaMGEHv3r2vi6e21XXx4sW0bt2aRx99FH9/f2JiYvjqq6/M20+cOEFycrJFHB4eHrRr186ivp6enrRu3dpcpmvXrmi1WjZv3mwuc++99+Lo6GhR30OHDpGWlmYuU9pnUlkdOnRg1apVHD58GDBNefnnn3+ahx+tTXW9VnWqW1lisbaMjAw0Gg2enp61vq5lIYm6HC5evIjBYLD4gw4QEBBAcnKyjaIqndFoZMyYMcTGxtKsWTMAkpOTcXR0NP8nuOLqeiQnJ9+wnle2lVYmMzOTvLy82/Z5zZ07lx07djBlypTrttW2uh4/fpxZs2YRHh7OihUreP755xk9ejT//e9/LeItLY7k5GT8/f0tttvb2+Pt7W2Vz8Ra9f3Xv/7F3/72NyIiInBwcCAmJoYxY8aYZ9qqTXW9VnWqW1lisab8/HxeffVVBg0aZB7PvbbWtaxkUo5absSIEezdu5c///zT1qFUidOnT/Piiy8SHx9vMZ9ybWU0GmndujXvvPMOADExMezdu5fPP/+coUOH2jg66/rxxx/5/vvvmTNnDk2bNiUhIYExY8YQHBxc6+oqTIqKihg4cCBKKWbNmmXrcKoNaVGXg6+vL3Z2dtf1GE5JSSEwMNBGUd3cyJEjWbJkCWvWrLGYDjAwMJDCwkLS09Mtyl9dj8DAwBvW88q20sq4u7vj7Ox8Wz6v7du3c/78eVq2bIm9vT329vasW7eOTz75BHt7ewICAmpNXQGCgoJo0qSJxbrIyEgSExMt4i0tjsDAQM6fP2+xvbi4mNTUVKt8Jtaq77hx48yt6qioKJ544gn++c9/mq+c1Ka6Xqs61a0ssVjDlSR96tQp4uPjLWZHq211LS9J1OXg6OhIq1atWLVqlXmd0Whk1apVtG/f3oaRWVJKMXLkSBYuXMjq1asJCwuz2N6qVSscHBws6nHo0CESExPN9Wjfvj179uyx+M9x5T/PlUTRvn17i2NcKXPlGLfj8+rSpQt79uwhISHBvLRu3ZrBgwebX9eWugLExsZe96jd4cOHqVevHgBhYWEEBgZaxJGZmcnmzZst6puens727dvNZVavXo3RaKRdu3bmMuvXr6eoqMiivo0bN8bLy8tcprTPpLJyc3PRai3/RNnZ2WE0GmtdXa9VnepWllgq60qSPnLkCCtXrsTHx8die22qa4XYrBtbDTV37lyl0+lUXFyc2r9/v3r22WeVp6enRY9hW3v++eeVh4eHWrt2rUpKSjIvubm55jLPPfecCg0NVatXr1bbtm1T7du3V+3btzdvv/LIUrdu3VRCQoJavny58vPzu+EjS+PGjVMHDhxQM2fOvOEjS7f787q613dtq+uWLVuUvb29evvtt9WRI0fU999/r/R6vfruu+/MZd59913l6empfvnlF7V792718MMP3/CxnpiYGLV582b1559/qvDwcItHXdLT01VAQIB64okn1N69e9XcuXOVXq+/7lEXe3t7NW3aNHXgwAH1xhtvWPXxrKFDh6o6deqYH8/6+eefla+vr3rllVdqRV2zsrLUzp071c6dOxWgPvzwQ7Vz505zT+fqVLeyxFLRuhYWFqqHHnpIhYSEqISEBIu/WVf34K4pda0Kkqgr4NNPP1WhoaHK0dFRtW3bVv3111+2DskCcMNl9uzZ5jJ5eXnqhRdeUF5eXkqv16t+/fqppKQki+OcPHlS9ezZUzk7OytfX1/10ksvqaKiIosya9asUS1atFCOjo6qQYMGFue44nZ/Xtcm6tpW119//VU1a9ZM6XQ6FRERob788kuL7UajUb3++usqICBA6XQ61aVLF3Xo0CGLMpcuXVKDBg1Srq6uyt3dXT355JMqKyvLosyuXbvUPffco3Q6napTp4569913r4vlxx9/VHfddZdydHRUTZs2Vb/99pvV6pmZmalefPFFFRoaqpycnFSDBg3UhAkTLP541+S6rlmz5ob/T4cOHVrt6laWWCpa1xMnTtz0b9aaNWtqXF2rgkapq4b5EUIIIUS1IveohRBCiGpMErUQQghRjUmiFkIIIaoxSdRCCCFENSaJWgghhKjGJFELIYQQ1Zgk6goqKChg0qRJFBQU2DqUKncn1RXurPpKXWuvO6m+tb2u8hx1BWVmZuLh4UFGRobFmLS10Z1UV7iz6it1rb3upPrW9rpKi1oIIYSoxiRRCyGEENXYHTcfdXFxMTt37iQgIOC6mXnKIysrC4CzZ8+SmZlprfCqpTuprnBn1VfqWnvdSfWtiXU1Go2kpKQQExODvX3pqfiOu0e9detW2rZta+swhBBCCLZs2UKbNm1KLXPHtagDAgIA04cTFBRk42iEEELciZKSkmjbtq05J5XmjkvUVy53BwUFERISYuNohBBC3MnKcgtWOpMJIYQQ1ZgkaiGEEKIak0QthBBCVGN33D1qIYQojcFgoKioyNZhiBrOwcEBOzs7qxxLEnUl7D2bwbn0PKLrehLg7mTrcIQQlaCUIjk5mfT0dFuHImoJT09PAgMD0Wg0lTqOJOpKeHPJfracSGXG4zE82DzY1uEIISrhSpL29/dHr9dX+o+ruHMppcjNzeX8+fMAlX4UWBJ1JXRS22hrtwtNkhYkUQtRYxkMBnOS9vHxsXU4ohZwdnYG4Pz58/j7+1fqMrh0JquEjnmreNlhPi4p22wdihCiEq7ck9br9TaORNQmV36fKtvnQRJ1JRidvEwvclNtG4gQwirkcrewJmv9PkmirgTl7A2AJj/NxpEIIYSorSRRV4LWxXQvy7FQErUQovaoX78+06dPL3P5tWvXotFoqrzHfFxcHJ6enlV6jurIpol6ypQptGnTBjc3N/z9/enbty+HDh0qdZ+4uDg0Go3F4uRkm0ejHNx8AdAVZtjk/EKIO9u1fwuvXSZNmlSh427dupVnn322zOU7dOhAUlISHh4eFTqfKJ1Ne32vW7eOESNG0KZNG4qLi/m///s/unXrxv79+3Fxcbnpfu7u7hYJ3Vb3lXTupkStN0iiFkLcfklJSebX8+bNY+LEiRZ/G11dXc2vlVIYDIZbzn0M4OfnV644HB0dCQwMLNc+ouxs2qJevnw5w4YNo2nTpkRHRxMXF0diYiLbt28vdT+NRkNgYKB5Kcs0YVXBxcMfADdjzZioXAhRu1z9d9DDw8Pib+PBgwdxc3Nj2bJltGrVCp1Ox59//smxY8d4+OGHCQgIwNXVlTZt2rBy5UqL41576Vuj0fD111/Tr18/9Ho94eHhLF682Lz92kvfVy5Rr1ixgsjISFxdXenRo4fFF4vi4mJGjx6Np6cnPj4+vPrqqwwdOpS+ffuW6zOYNWsWDRs2xNHRkcaNG/Ptt9+atymlmDRpEqGhoeh0OoKDgxk9erR5+2effUZ4eDhOTk4EBATwyCOPlOvct0u1ukedkWFqmXp7e5daLjs7m3r16lG3bl0efvhh9u3bdzvCu46rt+lbpydZ5BUabBKDEKJqKKXILSy2yaKUslo9/vWvf/Huu+9y4MABmjdvTnZ2Nr169WLVqlXs3LmTHj160KdPHxITE0s9zuTJkxk4cCC7d++mV69eDB48mNTUmz/xkpuby7Rp0/j2229Zv349iYmJvPzyy+bt7733Ht9//z2zZ89mw4YNZGZmsmjRonLVbeHChbz44ou89NJL7N27l3/84x88+eSTrFmzBoAFCxbw0Ucf8cUXX3DkyBEWLVpEVFQUANu2bWP06NG8+eabHDp0iOXLl3PvvfeW6/y3S7UZ8MRoNDJmzBhiY2Np1qzZTcs1btyYb775hubNm5ORkcG0adPo0KED+/btu+H80gUFBRQUFJjfZ2VlWS1mF09Ti9pFU8DZzCzq+Hpa7dhCCNvKKzLQZOIKm5x7/5vd0Tta58/zm2++yQMPPGB+7+3tTXR0tPn9W2+9xcKFC1m8eDEjR4686XGGDRvGoEGDAHjnnXf45JNP2LJlCz169Lhh+aKiIj7//HMaNmwIwMiRI3nzzTfN2z/99FPGjx9Pv379AJgxYwZLly4tV92mTZvGsGHDeOGFFwAYO3Ysf/31F9OmTeO+++4jMTGRwMBAunbtioODA6GhobRt2xaAxMREXFxcePDBB3Fzc6NevXrExMSU6/y3S7VpUY8YMYK9e/cyd+7cUsu1b9+eIUOG0KJFCzp16sTPP/+Mn58fX3zxxQ3LT5kyBQ8PD/PSpEkTq8WscfKk+PJHmJWaYrXjCiGEtbRu3drifXZ2Ni+//DKRkZF4enri6urKgQMHbtmibt68ufm1i4sL7u7u5iEyb0Sv15uTNJiG0bxSPiMjg5SUFHPSBLCzs6NVq1blqtuBAweIjY21WBcbG8uBAwcAePTRR8nLy6NBgwYMHz6chQsXUlxcDMADDzxAvXr1aNCgAU888QTff/89ubm55Tr/7VItWtQjR45kyZIlrF+//oat4tI4ODgQExPD0aNHb7h9/PjxjB071vz+7Nmz1kvWGg3ZGjc8VQbZ6ReAxtY5rhDC5pwd7Nj/Znebndtaru2Y+/LLLxMfH8+0adNo1KgRzs7OPPLIIxQWFpZ6HAcHB4v3Go0Go9FYrvLWvKRfFnXr1uXQoUOsXLmS+Ph4XnjhBaZOncq6detwc3Njx44drF27lt9//52JEycyadIktm7dWu0eAbNpi1opxciRI1m4cCGrV68mLCys3McwGAzs2bPnpoOe63Q63N3dzYubm1tlw7aQY+cOQEHGzb9ZCiFqHo1Gg97R3iZLVT7JsmHDBoYNG0a/fv2IiooiMDCQkydPVtn5bsTDw4OAgAC2bt1qXmcwGNixY0e5jhMZGcmGDRss1m3YsMGiMebs7EyfPn345JNPWLt2LZs2bWLPnj0A2Nvb07VrV95//312797NyZMnWb16dSVqVjVs2qIeMWIEc+bM4ZdffsHNzY3k5GTA9I94ZUDzIUOGUKdOHaZMmQKY7rfcfffdNGrUiPT0dKZOncqpU6d45plnbFKH87r6ZBZqyMiXzmRCiOovPDycn3/+mT59+qDRaHj99ddLbRlXlVGjRjFlyhQaNWpEREQEn376KWlpaeX6kjJu3DgGDhxITEwMXbt25ddff+Xnn38292KPi4vDYDDQrl079Ho93333Hc7OztSrV48lS5Zw/Phx7r33Xry8vFi6dClGo5HGjavflVGbJupZs2YB0LlzZ4v1s2fPZtiwYYDphr9WW9LwT0tLY/jw4SQnJ+Pl5UWrVq3YuHGjVe89l8eC8Cl891cio3WN6GWTCIQQouw+/PBDnnrqKTp06ICvry+vvvoqmZm3/xHTV199leTkZIYMGYKdnR3PPvss3bt3L9csU3379uXjjz9m2rRpvPjii4SFhTF79mxzTvH09OTdd99l7NixGAwGoqKi+PXXX/Hx8cHT05Off/6ZSZMmkZ+fT3h4OD/88ANNmzatohpXnEbd7psGNnbmzBnq1q3L6dOny30//EY+/P0Qn6w+yt/vDuXffaOsEKEQ4nbLz8/nxIkThIWF2Wykwzud0WgkMjKSgQMH8tZbb9k6HKso7feqPLmoWnQmq8m8XBwBSMup3DRmQghxJzl16hS///47nTp1oqCggBkzZnDixAkef/xxW4dW7VSbx7Nqquapy1nl+BIPnvvY1qEIIUSNodVqiYuLo02bNsTGxrJnzx5WrlxJZGSkrUOrdqRFXUludgYaapO4UHDW1qEIIUSNUbdu3et6bIsbk0RdScZGD/DYH/kU2gey0NbBCCGEqHUkUVeSm38om1UkDnmmh/ltNZOXEEKI2knuUVeSt97UmazIoMguKLZxNEIIIWobaVFXkrO2mCcdV+JqyCQt617cnBxuvZMQQghRRpKoK0uj5Q3tN6CFPekTwM+6Q5QKIYS4s8ml78qycyBHowcgJ03G+xZCCGFdkqitIEdrmpgjL+OCjSMRQojy69y5M2PGjDG/r1+/PtOnTy91H41Gw6JFiyp9bmsdpzSTJk2iRYsWVXqOqiSJ2gryHDwAKMq6aONIhBB3kj59+tCjR48bbvvjjz/QaDTs3r273MfdunUrzz77bGXDs3CzZJmUlETPnj2teq7aRhK1FRQ5egJQnH3JtoEIIe4oTz/9NPHx8Zw5c+a6bbNnz6Z169Y0b9683Mf18/NDr9dbI8RbCgwMRKfT3ZZz1VSSqK2gWOcNgMpLtXEkQog7yYMPPoifnx9xcXEW67Ozs5k/fz5PP/00ly5dYtCgQdSpUwe9Xk9UVBQ//PBDqce99tL3kSNHuPfee3FycqJJkybEx8dft8+rr77KXXfdhV6vp0GDBrz++usUFZnmQIiLi2Py5Mns2rULjUaDRqMxx3ztpe89e/Zw//334+zsjI+PD88++yzZ2dnm7cOGDaNv375MmzaNoKAgfHx8GDFihPlcZWE0GnnzzTcJCQlBp9PRokULli9fbt5eWFjIyJEjCQoKwsnJiXr16pmnWlZKMWnSJEJDQ9HpdAQHBzN69Ogyn7sipNe3FShnLwC0kqiFqH0Kc8q/j50O7C7/eTUUg6EANFpwcL71cR1dynwae3t7hgwZQlxcHBMmTDAPuDR//nwMBgODBg0iOzubVq1a8eqrr+Lu7s5vv/3GE088QcOGDWnbtu0tz2E0Gunfvz8BAQFs3ryZjIwMi/vZV7i5uREXF0dwcDB79uxh+PDhuLm58corr/DYY4+xd+9eli9fbp4r2sPD47pj5OTk0L17d9q3b8/WrVs5f/48zzzzDCNHjrT4MrJmzRqCgoJYs2YNR48e5bHHHqNFixYMHz68TJ/bxx9/zAcffMAXX3xBTEwM33zzDQ899BD79u0jPDycTz75hMWLF/Pjjz8SGhrK6dOnOX36NAALFizgo48+Yu7cuTRt2pTk5GR27dpVpvNWlCRqK9C6mFrUDgXptg1ECGF97wSXf59H46BpP9Prg7/C/GFQ7x548reSMtOjIPcGt8smZZTrVE899RRTp05l3bp15nmYZ8+ezYABA/Dw8MDDw4OXX37ZXH7UqFGsWLGCH3/8sUyJeuXKlRw8eJAVK1YQHGz6LN55553r7iu/9tpr5tf169fn5ZdfZu7cubzyyis4Ozvj6uqKvb09gYGBNz3XnDlzyM/P53//+x8uLqYvLDNmzKBPnz689957BAQEAODl5cWMGTOws7MjIiKC3r17s2rVqjIn6mnTpvHqq6/yt7/9DYD33nuPNWvWMH36dGbOnEliYiLh4eHcc889aDQa6tWrZ943MTGRwMBAunbtioODA6GhoWX6HCtDLn1bgYOrLwC6onTbBiKEuONERETQoUMHvvnmGwCOHj3KH3/8wdNPPw2AwWDgrbfeIioqCm9vb1xdXVmxYgWJiYllOv6BAweoW7euOUkDtG/f/rpy8+bNIzY2lsDAQFxdXXnttdfKfI6rzxUdHW1O0gCxsbEYjUYOHTpkXte0aVPs7OzM74OCgjh/vmyPx2ZmZnLu3DliY2Mt1sfGxnLgwAHAdHk9ISGBxo0bM3r0aH7//XdzuUcffZS8vDwaNGjA8OHDWbhwIcXFVTsqpbSorUDnbkrUzsXl+yYshKgB/u9c+fexu6pzVEQf0zE017SLxuypXFxXefrppxk1ahQzZ85k9uzZNGzYkE6dOgEwdepUPv74Y6ZPn05UVBQuLi6MGTOGwsJCq51/06ZNDB48mMmTJ9O9e3c8PDyYO3cuH3zwgdXOcTUHB8sRIDUaDUaj0WrHb9myJSdOnGDZsmWsXLmSgQMH0rVrV3766Sfq1q3LoUOHWLlyJfHx8bzwwgvmKxrXxmUt0qK2Ar2nPwBuxiyMRmXjaIQQVuXoUv7F7qo2kJ29ad3V96dLO24FDBw4EK1Wy5w5c/jf//7HU089Zb5fvWHDBh5++GH+/ve/Ex0dTYMGDTh8+HCZjx0ZGcnp06dJSkoyr/vrr78symzcuJF69eoxYcIEWrduTXh4OKdOnbKsrqMjBoPhlufatWsXOTkl9+83bNiAVqulcePGZY65NO7u7gQHB183xeaGDRto0qSJRbnHHnuMr776innz5rFgwQJSU039kJydnenTpw+ffPIJa9euZdOmTezZY70vXteSFrUVuHiZErWnJovM/CI8L0/UIYQQt4OrqyuPPfYY48ePJzMzk2HDhpm3hYeH89NPP7Fx40a8vLz48MMPSUlJsUhKpenatSt33XUXQ4cOZerUqWRmZjJhwgSLMuHh4SQmJjJ37lzatGnDb7/9xsKFlhP/1q9fnxMnTpCQkEBISAhubm7XPZY1ePBg3njjDYYOHcqkSZO4cOECo0aN4oknnjDfn7aGcePG8cYbb9CwYUNatGjB7NmzSUhI4Pvvvwfgww8/JCgoiJiYGLRaLfPnzycwMBBPT0/i4uIwGAy0a9cOvV7Pd999h7Ozs8V9bGuTFrUVOLr5k6x8SFI+pOZY73KSEEKU1dNPP01aWhrdu3e3uJ/82muv0bJlS7p3707nzp0JDAykb9++ZT6uVqtl4cKF5OXl0bZtW5555hnefvttizIPPfQQ//znPxk5ciQtWrRg48aNvP766xZlBgwYQI8ePbjvvvvw8/O74SNier2eFStWkJqaSps2bXjkkUfo0qULM2bMKN+HcQujR49m7NixvPTSS0RFRbF8+XIWL15MeHg4YOrB/v7779O6dWvatGnDyZMnWbp0KVqtFk9PT7766itiY2Np3rw5K1eu5Ndff8XHx8eqMV5No5S6o67Vnjlzhrp163L69GlCQkKsdtyO76/mdGoeC55vT6t63lY7rhCi6uXn53PixAnCwsJwcnKydTiilijt96o8uUha1FZyZV7q1JyyP3QvhBBC3IokaivxcjEl6rRcufQthBDCeiRRW8mI9GmsdhyL85kNty4shBBClJEkaivxNV6kgTYZY1ayrUMRQghRi9g0UU+ZMoU2bdrg5uaGv78/ffv2tRh95mbmz59PREQETk5OREVFsXTp0tsQbem2NRrNwILX2ekQY+tQhBBC1CI2TdTr1q1jxIgR/PXXX8THx1NUVES3bt0sHna/1saNGxk0aBBPP/00O3fupG/fvvTt25e9e/fexsivVxTYki0qkjMFFRuwQAhhe9Yc3UoIa/0+2XTAk6unFQPTVGj+/v5s376de++994b7fPzxx/To0YNx48YB8NZbbxEfH8+MGTP4/PPPqzzmm/F2MQ0dJ53JhKh5HB0d0Wq1nDt3Dj8/PxwdHc0jewlRXkopCgsLuXDhAlqtFkfHyg2CVa1GJsvIMI2V7e198+eQN23axNixYy3Wde/e3WI+U1sIKjrDE3a/o80IBDrYNBYhRPlotVrCwsJISkri3LkKjO0txA3o9XpCQ0PRait38braJGqj0ciYMWOIjY2lWbNmNy2XnJx83VByAQEBJCffuBNXQUEBBQUF5vdZWVnWCfgaAVl7ecshjk35zYHxVXIOIUTVcXR0JDQ0lOLi4luOSS3ErdjZ2WFvb2+VKzPVJlGPGDGCvXv38ueff1r1uFOmTGHy5MlWPeaNOHv4AeBqzKTYYMTeTjrUC1HTaDQaHBwcqmwWJCEqolpkk5EjR7JkyRLWrFlzy6HUAgMDSUlJsViXkpJy08nIx48fT0ZGhnnZv3+/1eK+2pWJObw02WTkyehkQgghrMOmiVopxciRI1m4cCGrV68mLCzslvu0b9+eVatWWayLj4+/4UTmADqdDnd3d/Pi5uZmldivZe9iGpDdiyzpUCaEEMJqbHrpe8SIEcyZM4dffvkFNzc3831mDw8PnJ1Nc7cOGTKEOnXqMGXKFABefPFFOnXqxAcffEDv3r2ZO3cu27Zt48svv7RZPQDQmzrAuWgKSMvMBv+q+UIghBDizmLTFvWsWbPIyMigc+fOBAUFmZd58+aZyyQmJlpMWN6hQwfmzJnDl19+SXR0ND/99BOLFi0qtQPabaHzwHD548xOO2/bWIQQQtQaNm1Rl2WGzbVr11637tFHH+XRRx+tgogqQaslV+uGmzGDvIwLto5GCCFELVEtOpPVFrn2HgAUZF20cSRCCCFqC0nUVlTo6AmAQRK1EEIIK5FEbUXFTl4AqLxUG0cihBCitpBEbU2XE7VGErUQQggrkURtRZrLz1LbF6TbNhAhhBC1hiRqK7LzCOas8iG9SIYfFEIIYR3VZqzv2qCozXPcvz4SN+wZZutghBBC1ArSorYibxfTnKNZBcUUFssE9EIIISpPErUVuTs5oL08o1m6jPcthBDCCuTStxVps86ySDcJo7GY1NyO+Ls72TokIYQQNZy0qK3JzpHmHKa55gSpWXm2jkYIIUQtIC1qa3L2ZqrnRLakwLBcmZNaCCFE5UmL2prs7Dnq04mtKoLUPIOtoxFCCFELSKK2sis9v9NypDOZEEKIypNL31YWU7gDB7sdaC5pgHBbhyOEEKKGkxa1lbW/MI83Hf6Ld2qCrUMRQghRC0iitjLl5A2AJi/NxpEIIYSoDSRRW5nGxZSo7QslUQshhKg8SdRWZu/qC4CuMN22gQghhKgVJFFbmc7NDwDn4gwbRyKEEKI2kERtZc6epha1u8oir1CepRZCCFE5FUrUp0+f5syZM+b3W7ZsYcyYMXz55ZdWC6ymcnI3tag9ySZNJuYQQghRSRVK1I8//jhr1qwBIDk5mQceeIAtW7YwYcIE3nzzTasGWNNo9KbOZN6aLFJl0BMhhBCVVKFEvXfvXtq2bQvAjz/+SLNmzdi4cSPff/89cXFx1oyv5nE2JWpPskjLKbBxMEIIIWq6CiXqoqIidDodACtXruShhx4CICIigqSkJOtFVxNdblE7agxkZqTbNhYhhBA1XoUSddOmTfn888/5448/iI+Pp0ePHgCcO3cOHx8fqwZY4zjoKdSYxvvOyzhv42CEEELUdBVK1O+99x5ffPEFnTt3ZtCgQURHRwOwePFi8yXxsli/fj19+vQhODgYjUbDokWLSi2/du1aNBrNdUtycnJFqlE1NBry7D0AKMi8aONghBBC1HQVmpSjc+fOXLx4kczMTLy8vMzrn332WfR6fZmPk5OTQ3R0NE899RT9+/cv836HDh3C3d3d/N7f37/M+94O2U5BZBUqcvLybB2KEEKIGq5CiTovLw+llDlJnzp1ioULFxIZGUn37t3LfJyePXvSs2fPcp/f398fT0/Pcu93u/x+9/+Y/Ot+emuCbB2KEEKIGq5Cl74ffvhh/ve//wGQnp5Ou3bt+OCDD+jbty+zZs2yaoA30qJFC4KCgnjggQfYsGFDqWULCgrIzMw0L1lZWVUen8xJLYQQwloqlKh37NhBx44dAfjpp58ICAjg1KlT/O9//+OTTz6xaoBXCwoK4vPPP2fBggUsWLCAunXr0rlzZ3bs2HHTfaZMmYKHh4d5adKkSZXFd4WX3pSo5TlqIYQQlVWhS9+5ubm4ubkB8Pvvv9O/f3+0Wi133303p06dsmqAV2vcuDGNGzc2v+/QoQPHjh3jo48+4ttvv73hPuPHj2fs2LHm92fPnq3yZN3g7GIWOc5kc1Yb4N4qPZcQQojarUIt6kaNGrFo0SJOnz7NihUr6NatGwDnz5+36OR1O7Rt25ajR4/edLtOp8Pd3d28XPmCUZVcVRYttMcILkpEKVXl5xNCCFF7VShRT5w4kZdffpn69evTtm1b2rdvD5ha1zExMVYN8FYSEhIICqpenbacmvRmeOFYPi16mByZmEMIIUQlVOjS9yOPPMI999xDUlKS+RlqgC5dutCvX78yHyc7O9uiNXzixAkSEhLw9vYmNDSU8ePHc/bsWXPHtenTpxMWFkbTpk3Jz8/n66+/ZvXq1fz+++8VqUaVcQoM5w+7tuQXGUnLKcRVV6GPWQghhKhYogYIDAwkMDDQPItWSEhIuQY7Adi2bRv33Xef+f2Ve8lDhw4lLi6OpKQkEhMTzdsLCwt56aWXOHv2LHq9nubNm7Ny5UqLY1QX3npHzmXkk5pTSF3vsj9bLoQQQlytQonaaDTy73//mw8++IDs7GwA3NzceOmll5gwYQJabdmuqHfu3LnUe7jXTvDxyiuv8Morr1Qk5NurMJe+9hvJtLtEam4bW0cjhBCiBqtQop4wYQL/+c9/ePfdd4mNjQXgzz//ZNKkSeTn5/P2229bNcgapzifV3KmgQMszBoNVK+R04QQQtQcFUrU//3vf/n666/Ns2YBNG/enDp16vDCCy9IonbyxIgWLUZy0y8ADWwdkRBCiBqqQr2+U1NTiYiIuG59REQEqamplQ6qxtNqybM3PaZWmHnBxsEIIYSoySqUqKOjo5kxY8Z162fMmEHz5s0rHVRtUOBgmkGrKPuSjSMRQghRk1Xo0vf7779P7969WblypfkZ6k2bNnH69GmWLl1q1QBrqmKdF+SdQuXIFQYhhBAVV6EWdadOnTh8+DD9+vUjPT2d9PR0+vfvz759+246lOedxuh0efrPfEnUQgghKq7Cz1EHBwdf12ls165d/Oc//+HLL7+sdGA1ncbFBwD7/DQbRyKEEKImq1CLWtya/eVErStMt20gQgghajRJ1FXE0d0XAOfiDIxGmZhDCCFExUiiriJOlxO1B1lk5RfbOBohhBA1VbnuUffv37/U7enp6ZWJpVZxcDUlai9NNqm5hXjoHWwckRBCiJqoXInaw8PjltuHDBlSqYBqDb03AJ5kk5pTSJivi40DEkIIUROVK1HPnj27quKoffQ+5GmcyUNHWk6hraMRQghRQ8k96qriH8nzoYt5sPAdUnMlUQshhKgYSdRVyFvvCCAtaiGEEBUmiboKebmYErW0qIUQQlSUJOoqNODMuyxyfB1N0i5bhyKEEKKGkkRdheoVnaCF9hiJJ4+RUyDPUgshhCg/SdRVSN9zEhOc/o8thWEs3ZNk63CEEELUQJKoq5CmUReC2w3gIh7M337G1uEIIYSogSRRV7H+Leug1cCWE6mcupRj63CEEELUMJKoq9KlYwSdXMy44D0A/CStaiGEEOUkiboq5VyAhc/y/KUpdNFuZ8H2MzKTlhBCiHKRRF2VQu+Gu0cAMM3xS4ozkth47JKNgxJCCFGTSKKual3fgMDmeJHFhw6f8dO2U7aOSAghRA1i00S9fv16+vTpQ3BwMBqNhkWLFt1yn7Vr19KyZUt0Oh2NGjUiLi6uyuOsFHsdPPINBntn7rHbR8j+r8jIK7J1VEIIIWoImybqnJwcoqOjmTlzZpnKnzhxgt69e3PfffeRkJDAmDFjeOaZZ1ixYkUVR1pJvuFoe74HwIvaH9m0vprHK4QQotoo1zSX1tazZ0969uxZ5vKff/45YWFhfPDBBwBERkby559/8tFHH9G9e/eqCtMqNC2HcGzzrzQ8H0/0lpehcyfQudk6LCGEENVcjbpHvWnTJrp27Wqxrnv37mzatOmm+xQUFJCZmWlesrKyqjrMG9NocH90JmeVL0GGJDIXvGibOIQQQtQoNSpRJycnExAQYLEuICCAzMxM8vLybrjPlClT8PDwMC9NmjS5HaHekJ9fAN8Gv4ZBaXA/vAB2/2izWIQQQtQMNSpRV8T48ePJyMgwL/v377dpPC1ie/JJcX8A1JKxkHrCpvEIIYSo3mpUog4MDCQlJcViXUpKCu7u7jg7O99wH51Oh7u7u3lxc7PtfeH7I/yZoxvIVuNdaAqzYMEzYDTaNCYhhBDVV41K1O3bt2fVqlUW6+Lj42nfvr2NIio/R3stD8bUZUzhCM47BMM9/wRtjfpnEEIIcRvZNENkZ2eTkJBAQkICYHr8KiEhgcTERMB02XrIkCHm8s899xzHjx/nlVde4eDBg3z22Wf8+OOP/POf/7RF+BX2aKu6nMWPTrlTSQvtVrLhxyGwcjJkX7BdcEIIIaoVmybqbdu2ERMTQ0xMDABjx44lJiaGiRMnApCUlGRO2gBhYWH89ttvxMfHEx0dzQcffMDXX39d7R/NulaTYHeaBruTZ9DwS8JZ08rUE7D/F9jwMWg0JYXz0kHJ+OBCCHGnsulz1J07d0aVkoRuNOpY586d2blzZxVGdXs82iqEfef2M3/7GYbFhoFrAAz4D1w6Ci6+JQXnPg4Zp6Hu3VC3LYS0gYBmYGfTfzohhBC3ify1t5GHWtTh7aUH2Hcuk/3nMmkS7A5Rj1gWys+EczuhKBfSE2HP5ce5HPQQ3BLqtoGQtqYEfnVyF0IIUWtIorYRbxdHukYGsGxvMjPXHuXJDvUJ9dHj56pDc+XSt5M7vHwETm+GM1vh9BY4sw0KMuDUn6blCvc6ENgcgppDzN/BM9Q2FRNCCGFVkqht6NHWISzbm8xvu5P4bXcSAM4OdoR66wn10VPPW089Hz0t6rYmqnMX005GI1w8DGe2XE7cW+HCQcg8a1oOL4PGvUoS9aFlcPJPuKsHhHW0UU2FEEJUlCRqG+p8lz+j72/EtlNpnLqUS1JGHnlFBg6lZHEoxXKo07/fHcqEXk1wdrQD/wjT0vJyj/j8TEjZC0m7IXkP+EeW7HjwN9j5rWkWryuJOucS/DENglpAUDT4hoPW7vZUWgghRLlIorYhrVbD2G6Nze8Li42cScvlVGouiZdyOXUpl2MXsll3+ALf/ZXIxmOX+PixGKJCPCwP5OQO9TqYlms17mVK0g3vL1l3bif89VnJewc9BMdcPkas6Z63o4uVayuEEKIiNKq0bte10JkzZ6hbty6nT58mJCTE1uGUyR9HLvDy/F2kZBZgr9Xwzwfu4rlODbHTam69840k7zW1ss8lmFrgRTmW27X2lok79G5w8rjhoYQQQpRfeXKRJOoaIi2nkP9buIdle5MBaFvfmw8GRlPXW1+5AxsNpJ/ejy55K85n/4KTGyDzjGUZjRaaPAyPxpWs++lpKMiCnu+Bd5hpXfJe0/1y7zDwbgjOnpWLTQghaqny5CK59F1DeLk48tnglvy0/QyTFu9jy8lUen38B2/2bUrfFnVKeoqXkdGoWH/kAnM2J7Lq4HlcdSF88OhbdO0fAGmn4NRGOLXBtKQev360tONrIPcSPDC5ZN2BX2HduyXv9T7g3cCUtH0agqs/6NxNl+qdPE2v9d7yaJkQQpRCWtQ1UOKlXP75YwLbT6UB0Lt5EANa1qFZHQ/83ZxK3fd8Vj7zt53hhy2JnEm7fmrQ4R3DeKVHBA52Vw1al5kEuRchMKpk3d6foTAbIh8qaTlvj4Nd8yD1GGRbTp5yU3XbwdO/l7z/7SWwd4K7nwePmvnvI4QQtyKXvktRGxI1QLHByGdrj/HxqiMYjCX/hP5uOqLqeNC0jgdRdTxoVsedADcnNh2/xPebT/H7vhSKL5d3d7JnQKsQHm1Vl5+2n+GbDaYpN2NCPfl0UAwhXpW4rF6QbWqJpx67/PM45KZCfoapl3pBhul1aHt4fJ5pH0MxvBMEhkJ4cRd41Tet3/IVRQdXkOQURmCjljgGNwPfu0yd5IQQogaSRF2K2pKor9h1Op3/bjrJnjMZHLuQjfEG/5rODnbkFRnM71uGevJ4u3o82DwIJ4eSx7JW7Evm5fm7yMovxsPZgQ8ejaZrk4CqrYBSJWObF+XDrjlw8Qh0e9s8q1jWt3/H7divFrsZNfZofBuh8W8CAU3Av4npsTS3YHAo/aqCEELYmiTqUtS2RH213MJiDiRlsudMBnvPZbL3bAZHzmdjMCrcdPb0janD4+1CiQxyv+kxTqfmMnLODnadyQBucin8NvrzyEU+/e5HwosPE6k9TbjmNBGa07hrcm++k4MeWj8F3d82vS/Kh99fM90L7/yvknLHVkPmOdA6mMZO1zqAMpha/PkZN17qtYeuk0z7Gw2w6AXTo2zd/g2Ol69AXDgMxXmme/M61yr5XIQQNZt0JrtD6R3taVXPm1b1vM3r8osMnLyUQ6i3Hr3jrf+563rrmf9cB95ddpBvNpzgqz9OsO1UWuUvhVfA3C2JvLZoL8XGehjrt+Cff2/FiYs5vLfjDFt37yW48DiNNWdorD1NtMNZ6qsz2Kki09joV8u9CFu/AjtHy0S9+UvTSG7l4Rte8rowB3bPNb3u/k7J+g3TIeF702vXQFNHOu8G4NPI9Non3PTe3rF85xZC3JEkUddyTg52RATevAV9I472Wib2aUK7Bt6Mm7+LnYnp9Jz+Bw+1CKZ/yxBahnqWu5d5eRiNivdWHOSLdccB6NsimPceaY7O3g4fVx2t63tT8FBT1hy8wKKdZ5l98DyFuUZAUde5iLH3+NKnVcOSX257J7h3nCmxXi04xtSCNhSBsdj0U6Mx9Uh3cjc9O37t4lsyQA1ae3jgLdMXg6vvl9vrTD3ecy9BdrJpObXB8txae1Oy9mtsOmbYvdCgk5U/SSFEbSCXvkWpTqfmMuqHnSScTjevC/N1oX9MHfrG1Kn8c9zXyCs08M95CSzfZ3pefEzXcF7sEl7qF4P03EJ+25PEfzee5HBKNgARgW5M7NOEDg1t+OhXXhpcutyh7tKxyz+Pmu7BF2Zblm37D+j1vul1VrJpelP3OvDYtyVlEjebOtq5+IHOzXTJ3dFVpjwVogaSe9SlkERdfgajYtOxS/y84wzL9iZbdExrF+bNgJYh9IwKxM3JoVLnOZ+Vz/D/bmPXmQwc7bS890gU/WLK/m9UbDDyw5ZEPog/THpuEQA9mgYyoXek1b9QVIpSpnvjFw7ChUOmn3d1h4jepu3ndsKXncEtCF46WLLfNz0gcdP1x7N3upy0LyduR5fLz6t7mIaQbf6oqVxRPhxaalrf8P6STnz5maYWvoNzyTpryUuH9FOmaVrTLv9MP2V6nZ9h6vjn4GI6t4OzafKY9i9cjjcPVr0Fdg5w/+slX0iOroTUE6ZbGRrt5Zg1N/6pjKYvNx51oeF9pv2NRlg/FYxFcM/Ykr4FyXtNE9vo3E1fhJwu/9Q6mMbC19qDxs7cyfE6SsHZHZCfBvU7llxlObQMjq0xxXFlKS4wXcExFJiu5thfrr+D3vSZeNWH2BdLjn10ZcmIgTJKYK0giboUkqgrJ6egmOV7k1mw4wybjl/iym+Pk4OWLhEBPNg8iPsi/C16k9+KUopdZzIY8f0Ozqbn4al34MsnWtM2zPvWO99Aem4hH8Uf5rvNiRiMCkd7Lc92bMAL9zUs0316m8tLM40QZyyGpn1L1i8YDud2QM4F0+NvynDTQ5jFjikZlCbtJHwcbUoKryWXlPnuETgaD2hMicLRxZS8riR98zrXy+tdTM+/R/Yx7Z+bCr+NNSWev31/g+OWQ+un4MGPTK9zLsHUBqbXE1NLJo6Z/yTs+7l8x418qOTqhFIw2dP0+uWj4Opner10HGz5smzHu5K0wzrC3xeUrP93oKkj4dWPF8a/Yeq3UB5BLeAf60ref9wC0k7Ak8tNHRoBEn6ALV+Aiz+4BZquwLgHg0edktc6t/Kd905WePkW1pXfs8Ic0zo7B9OXQjtH0zYrfZmVzmSiyrjoTM9eD2gVwtn0PBbtPMuCHWc4fiGH3/Yk8dueJFwc7ejaJIAHmwdz712+6OyvT9pn0nLZeOwSm45dYuOxi6RkFgCmy+rfDGtDmG/FJwXx1Dsy+eFmPN6uHpN/3cfGY5eYseYoP20/w+SHm9K9aWCFj31bOHtB5IPXrx/wVclrpUwts8Ic02X0wpyS1wXZUHC553pQC8t96sWakszVzJ3vlGnc96IcuOZ2/nVa55QkamWEfQtNr42Gkj90Ln4lPz1DwbOe6afX5Z96H1PLsjDH1HouyisZjhZMne3u+afpC8DVs7uFtDZ9SSkuNJ0bZarbdT8x/VG1czS1RK/QaKD106ZjXt2hzy3I9HkVZJqGx83PNLV4b8RYDFzu13C1gCaXW83FJevCOpo+cztH0/nsrlrsdaaEX5x/+TPINb2+drQ+v8amsm5X/e5eOmq6+lIanbspYTt7mz4r30bw8MyS7bN7m64i/G2OKXaAE3+YrgK4+Jj+jfS+pnjcg01fAGrqTHtF+aa6pp00/Z7e1a1k24y2cPEQPL8RApqa1u34Fpa/ankMO0d4/ZpRGm8DaVGLSlNKsfdsJkt2n2PJ7iTOppeMeObmZE/3poH0igoku8DApmMX2XjsEqcuWfbMdrTXcn9jf6b0j8LLxXq9oZVS/L4/hX//tp/TqXloNPDhwOhyXVKv9YxGU4IovJykr7QkzK+vWYpyIKSNafx3MCXMbd+YWtrRg0ouUedcMl3GrckzsRUXmpKysdj05cBovOp1sekPt5uNvvilnYTzB02jAGYlm8bozzx3eTlr+qJ2rWtb6tOjTLcjnlll+gIEsH4arH7rxufU2ptuI3jVM33x8qpveu3dEIJblJQrzCn5clKFHU8BUwLOSzOdMy8VMs6YlsyzJa8zzpie/rjCJxxGbSt5/0UnSEqAQfOgcQ/Tuo0z4PcJludycIEJ56wStlz6LoUk6qqllGJHYjpLdp9j6Z4kc0v5WnZaDc1DPOjQ0IfYhr60rOdVrsvl5ZVfZGDyr/v5YUsiWg18MiiGB5sHV9n5hLC5gmzISjIlqfx0U+J09oL695SUObvd1Lr0jyy5TH5iPRyJN93SyL0IORdNt1syz5nu69+IXySM+Kvk/UfNIOM0DF8NdVqZ1v31Ofwx7fK4BQ5XXVJ2KFkHppa/Uqaf7sGWHSq/Hwjn98OA/0BoO9O6zV/CsnFl+0wcXExXc/zugoH/K1mfdtJ079/Zy7K8UpefCCksuVJy5VZJJcmlb2EzGo2GVvW8aFXPi9d7N2HryVSW7E5i9cHzuDs7ENvQhw6NfGhT37vSnc/Kw8nBjrf7NsNoVMzbdpoX5yZgr9XSo1k1vwwuREXpXEEXbvns/7WuJNGrhd1rWq5lNJgSf9rJyx0DL3cKTDtpGh/gaoZC00+7q66OFWSZEn555IZZvs9ONn0BuPpqgaPedPvA0cWUbD1CTIt7nZLXV947e924hX+lP8G1NJqSLxXY7sqQtKjFHcVoVLz80y5+3nEWBzsNswa3qvQwqZn5Raw/fIGV+1PYeOwSne7yY9JDTXHRyfdgcYcqyjMlawcXy1sh2SmXW6ZFptb5ldeGy681GlNPfi7/dHSxHF8gaZephevTqKT3u9F4eb8qvsRuZdKiFuImtFoNUx+JptigWLzrHC98v4Mvh7Sic2P/ch3ndGouKw+ksOrAef46fsk80QnA/O1n2JGYxozHW5Y6XKsQtdaVx+2u5uJjWiojKPr6dTd7XK4WkUQt7jh2Wg0fDoymyGBk2d5knv12O7OHtSG20c0HRzEYFQmn01h98DyrDpznYHKWxfaGfi50bRJARKAb7y07xLELOfSduYE3+jRlUNu6ZRrJ7VByFjPWHOVwchajujSid1SQ1UeAMxoVS/cmcTApC73ODledPa46e1wu/7zy2sfF0aqd+oQQFSeXvsUdq7DYyAvf72DlgRScHLTEPdmWuxuUfOO/lF3AusMXWHvoAuuPXDAPogKmZN+6nhcPNAmgS2SAxeNkqTmFjP0xgbWHTPfj+kQH806/Zje9J38oOYtPVh3htz1JFus7hvvy1sPNqF+JR9Wu9seRC7yz9CAHkjLLVL5tfW/6t6xDr+ZBuN/G/gRC3AlqXK/vmTNnMnXqVJKTk4mOjubTTz+lbdu2NywbFxfHk08+abFOp9ORn59fpnNJohZXKyg28I9vt7P20AX0jna80y+Kk5dyWHPoArvPpHP1/w53J3s63uXHA5EBdG7sh6f+5i1Oo1Hx1R/HeX/FIQxGRX0fPTMeb0mzOiWjSh1MzuSTVUdYuqdk8JGezQIJ83Xh6z9OUGgw4miv5flODXm+c8MK94o/kJTJlGUHWX/Y9MXBTWfPg9FBFBsU2QXF5iWnoJjsfNPrzPyS54B19lq6NQ1kQMs63NPIF3sbzaQmRG1SoxL1vHnzGDJkCJ9//jnt2rVj+vTpzJ8/n0OHDuHvf/19w7i4OF588UUOHTpkXqfRaAgIKFuHIEnU4lr5RQaG/28bfxy5eN22JkHu3Bfhx32N/WlR17PcSWr7qTRGzdnBuYx8HO20vP5gJK3re/PJqiMs21uSoHtFBTLq/nDzPe0TF3OY+Mtec0z1ffS8+XAz7r2r7I+GJGXk8eHvh/lpxxmUAgc7DX+/ux6j7g/H+xaXtZMy8li08xwLdpzh6PmSccn93HT0i6lD/5Z1yj3ZixCiRI1K1O3ataNNmzbMmDEDAKPRSN26dRk1ahT/+te/risfFxfHmDFjSE9Pr9D5JFGLG8krNPD899vZcSqN2Ea+3NfYn06N/Qhwd6r0sdNzC3l5/m5WHki5blvvqCBGdWl0w6SnlOK3PUm8+et+zmeZnkfv3TyI13s3IdDj5nFl5Rfx+bpj/OfPE+QXGc3neaVHY+r5lO8yulKKPWczWLD9DIt3nSPtqsv/vaICmdKvOR56uSwuRHnVmERdWFiIXq/np59+om/fvub1Q4cOJT09nV9++eW6feLi4njmmWeoU6cORqORli1b8s4779C0adMynVMStbAFpRT/+fME7y0/SLFR0SsqiNH3h9M48NZjMWflF/Fh/GH+u/EkRgXODnb4uekoNhgpNiqKjYoigxGDUVFsUBQZjeZL9m3qe/F/vSKJCfUq/SRlUFhsZO2h8yzYcYZVB85TbFQEezjx8aAY2tSv2LjsQtypaszjWRcvXsRgMFx32TogIICDBw/ecJ/GjRvzzTff0Lx5czIyMpg2bRodOnRg3759N6xsQUEBBQUlo2NlZWVdV0aIqqbRaHimYwMeuPzMdnlatm5ODrzRpykDWobw2qK9JJxOJzE1t9R9Gvi58K8eETzQJMBqPccdL9+r7tY0kN1n0hn9w05OXsrlsS82MbpLOCPvayT3r4WoAjXu8az27dvTvn178/sOHToQGRnJF198wVtvXT8+7ZQpU5g8efLtDFGImyrvpeerNavjwc/Pd2DfuUwKDUbstRrs7TQ42Gmx02pw0GqxtzOt83PVWf3Rrqs1D/FkyeiOTPxlLz/vOMv0lUfYePQSH/2tBXU8nW99ACFEmdn066+vry92dnakpFjeu0tJSSEwsGxDOzo4OBATE8PRo0dvuH38+PFkZGSYl/3791c6biFsRavVEBXiQat6XkTX9aRpsAd3BbjR0M+VUB89wZ7O+Ls5VWmSvsJVZ8+HA1sw/bEWuDjaseVkKr0+/oPle5NuuW9GXhGnU3O5lF1AfpGBavDwiRDVlk1b1I6OjrRq1YpVq1aZ71EbjUZWrVrFyJEjy3QMg8HAnj176NWr1w2363Q6dDqd+X1mZtmeIRVClE3fmDrEhHoy+oed7DqTwXPf7eDxdqE8c08Y59LzSUzNJTE1l9OXfyam5pKRZzm5g71Wg97Rzjzgil5nj6ezA/dH+PNQdLAMviLuaDbv9T1v3jyGDh3KF198Qdu2bZk+fTo//vgjBw8eJCAggCFDhlCnTh2mTJkCwJtvvsndd99No0aNSE9PZ+rUqSxatIjt27fTpEmTW55POpMJUTUKi418GH+Yz9cdK1N5nb2WgmLjLcs52GnoEhHAgFYhdG7sh4PcBxe1QI3pTAbw2GOPceHCBSZOnEhycjItWrRg+fLl5g5miYmJaK8ayzUtLY3hw4eTnJyMl5cXrVq1YuPGjWVK0kKIquNor+VfPSOIbeTD+J/3cCGrgFBvPXW99RY/Q731hHg546Kzx2BU5BQWk1tgMA+6klNQTE6hgVOXcvh5x1n2J2WyfF8yy/cl4+PiyMMt6jCgVR2aBnvcNJYig5GcgmKcHe3Q2Vt/+tQrj63F708hLbcQe60WrcbUP0Cr0WCnBTutFjuNBl83Rzrd5UeIl97qcVSGUoozaXlsPZnKufQ8HmwebLVR8IR12bxFfbtJi1qIqnflz4o17pUfSMpkwfYzLEo4x8Xskic4IgLdqOPpTNaVUdUu/8zKLza31PWOdnRvGsjDLYIrPaqa0ahIOJPOsj1JLN2TzNn0vHLtHxHoRpdIf+6PCKBFXU/stLd3tieDUXEwOZNtJ9PYcjKVbSdTLeaLd3awY3yvCP7erh7a2xzbnajGPEdtC5KohaiZig1G1h+5wE/bz7By/3kKDbe+bH41HxdHejcP4uEWdWgZ6lmmLxFGo2JHYhpL9ySzbG8SSRklQxXrHe24L8Kfhn6uGI0Kg1IYjJZLsVFx9HwW20+lcdUEa3i7ONK5sR9dIgLoeJdvlY2lXmQwsnDnWX7bncSOU2lkFRRbbLe/3DlRA+xITAfgnka+vPdI8xrRe18pRWJqLn8dv8Rfx1PRO9rxzwfuwtdVd+udbUwSdSkkUQtR86XnFrL64HmKDEZcdQ646OxwcyqZBcxN54BeZ8fuMxksTjjLkt1JXMopNO9f19uZh6PrcO9dfuQWFpOaU8il7EIu5RRyKbvA9D6nkDNpuVzMLtnPVWdPl0h/ejYLotNdfjg7lu2yelpOIesOX2DVwfOsPXSerKvGUne00zKgVR2e69SwUo/vXa3YYGRRwjk+XX2EU5dKnrl31dnTsp4Xbep50SbMm+gQT5wd7TAaFd/+dYopyw6QX2TETWfPxD5NeKRVyG15gqA8Tqfmsun4JVNyPnaJcxmW8zwEezjxxROtiQq5+a2R6kASdSkkUQtx5yk2GPnz6EUWJ5xjxb5kcgoNZd7XTWfPA00C6BkVRMdw3wpPjnJFkcHI9lNprDqQwqqD5zl+IQcArQYeig5mxH2NCA+49Yh1N2IwKhbvOssnq45y4qLpuD4ujjx1Txid7vIjMsi91Evuxy9k89L8Xey83LruGhnAO/2b4e9W+aF0y0IpRVpuEReyCjiflc+FrILLrwtIycwn4XQ6Z9Isbzk42GmIDvGkXQNvlu1J5vjFHHT2Wt4b0Jy+MXVuS9wVIYm6FJKohbiz5RUaWHkghV8SzrL/XCYeekd8XR3xdjEtvq46vF0c8XFxxNdNR9Ng9yrpkHbF1pOpzFh9lHWXZzcD6NE0kJH3N7KYba00BqNpXPiPVx7m2OXE76V34LlODXmifT30jmXvN2wwKr5Yf4yP4g9TZFB46R14u18UvaKCylexMlBKkXA6nV8SzrHqYArJGfkUGUpPSfZaDc1DPGjf0Ie7G/jQqp6XuX4ZeUWMmbuTNZenmH323ga82iOiTP0BMvOL2HTsEo38XWno51r5yt2CJOpSSKIWQlRHe85kMHPNUZbvK5lVrdNdfoy4rxGN/F3JKSgmt9Bg7iWfU1hMbmExaTlF/LAlkSOXZznz1DswvGMDhnaoj6uu4g/2HEjKZOyPu8zzl4d4OePnpsPX1bT4uZq+yFx5H+ThRLCnc5mS4vEL2SxKOMfihLOcvHT9cLheegf83HT4uzld/mk6R3iAK23qe+NSSr0MRsUHvx/is7WmxwQ7hvvy6aCYG05Lq5Ri95kM5mxOZPGuc+QVma60dLrLj6fuCePecN8qu/QviboUkqiFENXZ4ZQsPltzlMW7zll0QLsVdyd7hndswLDY+rhZqXNaYbGRT1cf4bO1xzCUIRhHey31ffSE+boQ5utKA18XwvxcCPN1wWhU/Lo7iV8SzrL7TIZ5H2cHOx5oEsDDLYKJDHLH11WHo33ln5Vfsvsc4+bvJq/IQD0fPV8+0do8CU52QTG/JJxlzuZE9p0rGQSrjqcz5zLyzJPaNPJ3ZViH+vRvWadcVyXKQhJ1KSRRCyFqglOXcvh83TEWbD9LocGIk4MWF0d79Do7009HO9Mobo52RNXxYEiH+lXWe/xCVgGJqTlcyCrkYnZByXL5/YXsApLS88vcE99Oq6FjuC99W9ThgSYBpbaQK2P/uUyG/28bZ9Pz0DvaMb5XJPvPZbI44ay5n4KjvZZezQJ5vF092tT3IjE1l7iNJ5m/7QzZl3vJezg7MKhtKEPa1yPYSr3hJVGXQhK1EKImKTIYLw+iUr16X1/LYFScS8/j+MUcTlzI5sTFHNPrizmcTTe1UmNCPenbog69mwfdtkeoUnMKGfH9DjYdv2SxvoGvC4+3C2VAy5AbDlGblV/E/G1niNt40jxbnZ1WQ89mgbx2iznhy0ISdSkkUQshxO2VX2Qgr9BgszHbiwxG3l12kHlbT3NfhD+Ptw3l7gbeZbr/bDAqVh88zzd/nmDT8Uu46ezZ9H9dKnX/H2rYEKJCCCFqNycHu0o/1lYZDnZaXn+wCa8/WP6hpu20Gh5oEsADTQLYfy6TYxeyK52ky0sStRBCCFEGTYLdaRLsftvPK9PQCCGEENWYJGohhBCiGpNELYQQQlRjkqiFEEKIakwStRBCCFGN3XG9vo1G08g5SUlJNo5ECCHEnepKDrqSk0pzxyXqlJQUANq2bWvjSIQQQtzpUlJSCA0NLbXMHTcyWXFxMTt37iQgIACttnJX/rOysmjSpAn79+/Hza1i88cKURPJ7764E1nz995oNJKSkkJMTAz29qW3me+4RG1NmZmZeHh4kJGRgbv77X8IXghbkd99cSey1e+9dCYTQgghqjFJ1EIIIUQ1Jom6EnQ6HW+88QY63e2Zrk2I6kJ+98WdyFa/93KPWgghhKjGpEUthBBCVGOSqIUQQohqTBK1EEIIUY1Joq6EmTNnUr9+fZycnGjXrh1btmyxdUhCVKn169fTp08fgoOD0Wg0LFq0yNYhCVHlpkyZQps2bXBzc8Pf35++ffty6NCh23Z+SdQVNG/ePMaOHcsbb7zBjh07iI6Opnv37pw/f97WoQlRZXJycoiOjmbmzJm2DkWI22bdunWMGDGCv/76i/j4eIqKiujWrRs5OTm35fzS67uC2rVrR5s2bZgxYwZgGg6ubt26jBo1in/96182jk6IqqfRaFi4cCF9+/a1dShC3FYXLlzA39+fdevWce+991b5+aRFXQGFhYVs376drl27mtdptVq6du3Kpk2bbBiZEEKIqpaRkQGAt7f3bTmfJOoKuHjxIgaDgYCAAIv1AQEBJCcn2ygqIYQQVc1oNDJmzBhiY2Np1qzZbTnnHTfNpRBCCFFRI0aMYO/evfz555+37ZySqCvA19cXOzs789zWV6SkpBAYGGijqIQQQlSlkSNHsmTJEtavX09ISMhtO69c+q4AR0dHWrVqxapVq8zrjEYjq1aton379jaMTAghhLUppRg5ciQLFy5k9erVhIWF3dbzS4u6gsaOHcvQoUNp3bo1bdu2Zfr06eTk5PDkk0/aOjQhqkx2djZHjx41vz9x4gQJCQl4e3sTGhpqw8iEqDojRoxgzpw5/PLLL7i5uZn7Inl4eODs7Fzl55fHsyphxowZTJ06leTkZFq0aMEnn3xCu3btbB2WEFVm7dq13HfffdetHzp0KHFxcbc/ICFuA41Gc8P1s2fPZtiwYVV/fknUQgghRPUl96iFEEKIakwStRBCCFGNSaIWQgghqjFJ1EIIIUQ1JolaCCGEqMYkUQshhBDVmCRqIYQQohqTRC2EEEJUY5KohRBVRqPRsGjRIluHIUSNJolaiFpq2LBhaDSa65YePXrYOjQhRDnIpBxC1GI9evRg9uzZFut0Op2NohFCVIS0qIWoxXQ6HYGBgRaLl5cXYLosPWvWLHr27ImzszMNGjTgp59+sth/z5493H///Tg7O+Pj48Ozzz5Ldna2RZlvvvmGpk2botPpCAoKYuTIkRbbL168SL9+/dDr9YSHh7N48WLztrS0NAYPHoyfnx/Ozs6Eh4df98VCiDudJGoh7mCvv/46AwYMYNeuXQwePJi//e1vHDhwAICcnBy6d++Ol5cXW7duZf78+axcudIiEc+aNYsRI0bw7LPPsmfPHhYvXkyjRo0szjF58mQGDhzI7t276dWrF4MHDyY1NdV8/v3797Ns2TIOHDjArFmz8PX1vX0fgBA1gRJC1EpDhw5VdnZ2ysXFxWJ5++23lVJKAeq5556z2Kddu3bq+eefV0op9eWXXyovLy+VnZ1t3v7bb78prVarkpOTlVJKBQcHqwkTJtw0BkC99tpr5vfZ2dkKUMuWLVNKKdWnTx/15JNPWqfCQtRSco9aiFrsvvvuY9asWRbrvL29za/bt29vsa19+/YkJCQAcODAAaKjo3FxcTFvj42NxWg0cujQITQaDefOnaNLly6lxtC8eXPzaxcXF9zd3Tl//jwAzz//PAMGDGDHjh1069aNvn370qFDhwrVVYjaShK1ELWYi4vLdZeircXZ2blM5RwcHCzeazQajEYjAD179uTUqVMsXbqU+Ph4unTpwogRI5g2bZrV4xWippJ71ELcwf7666/r3kdGRgIQGRnJrl27yMnJMW/fsGEDWq2Wxo0b4+bmRv369Vm1alWlYvDz82Po0KF89913TJ8+nS+//LJSxxOitpEWtRC1WEFBAcnJyRbr7O3tzR225s+fT+vWrbnnnnv4/vvv2bJlC//5z38AGDx4MG+88QZDhw5l0qRJXLhwgVGjRvHEE08QEBAAwKRJk3juuefw9/enZ8+eZGVlsWHDBkaNGlWm+CZOnEirVq1o2rQpBQUFLFmyxPxFQQhhIolaiFps+fLlBAUFWaxr3LgxBw8eBEw9sufOncsLL7xAUFAQP/zwA02aNAFAr9ezYsUKXnzxRdq0aYNer2fAgAF8+OGH5mMNHTqU/Px8PvroI15++WV8fX155JFHyhyfo6Mj48eP5+TJkzg7O9OxY0fmzp1rhZoLUXtolFLK1kEIIW4/jUbDwoUL6du3r61DEUKUQu5RCyGEENWYJGohhBCiGpN71ELcoeSulxA1g7SohRBCiGpMErUQQghRjUmiFkIIIaoxSdRCCCFENSaJWgghhKjGJFELIYQQ1ZgkaiGEEKIak0QthBBCVGOSqIUQQohq7P8B1Sh5DnsXWQkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from previous_chapters import plot_losses\n",
    "# Alternatively:\n",
    "# from llms_from_scratch.ch05 import plot_losses\n",
    "\n",
    "epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))\n",
    "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6777e0c4-d82c-46d8-84fb-1376c4f8bae0",
   "metadata": {
    "id": "6777e0c4-d82c-46d8-84fb-1376c4f8bae0"
   },
   "source": [
    "- As we can see, the loss decreases sharply at the beginning of the first epoch, which means the model starts learning quickly\n",
    "- We can see that slight overfitting sets in at around 1 training epoch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87b79a47-13f9-4d1f-87b1-3339bafaf2a3",
   "metadata": {
    "id": "87b79a47-13f9-4d1f-87b1-3339bafaf2a3"
   },
   "source": [
    "## 7.7 Extracting and saving responses"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a25cc88-1758-4dd0-b8bf-c044cbf2dd49",
   "metadata": {
    "id": "5a25cc88-1758-4dd0-b8bf-c044cbf2dd49"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-6.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17510e9d-7727-4d58-ba9a-d82ec23c1427",
   "metadata": {
    "id": "17510e9d-7727-4d58-ba9a-d82ec23c1427"
   },
   "source": [
    "- In this section, we save the test set responses for scoring in the next section\n",
    "- We also save a copy of the model for future use\n",
    "- But first, let's take a brief look at the responses generated by the finetuned model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "VQ2NZMbfucAc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VQ2NZMbfucAc",
    "outputId": "066c56ff-b52a-4ee6-eae7-1bddfc74d0c1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
      "\n",
      "### Instruction:\n",
      "Rewrite the sentence using a simile.\n",
      "\n",
      "### Input:\n",
      "The car is very fast.\n",
      "\n",
      "Correct response:\n",
      ">> The car is as fast as lightning.\n",
      "\n",
      "Model response:\n",
      ">> The car is as fast as a bullet.\n",
      "-------------------------------------\n",
      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
      "\n",
      "### Instruction:\n",
      "What type of cloud is typically associated with thunderstorms?\n",
      "\n",
      "Correct response:\n",
      ">> The type of cloud typically associated with thunderstorms is cumulonimbus.\n",
      "\n",
      "Model response:\n",
      ">> The type of cloud associated with thunderstorms is a cumulus cloud.\n",
      "-------------------------------------\n",
      "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
      "\n",
      "### Instruction:\n",
      "Name the author of 'Pride and Prejudice'.\n",
      "\n",
      "Correct response:\n",
      ">> Jane Austen.\n",
      "\n",
      "Model response:\n",
      ">> The author of 'Pride and Prejudice' is Jane Austen.\n",
      "-------------------------------------\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "\n",
    "for entry in test_data[:3]:\n",
    "\n",
    "    input_text = format_input(entry)\n",
    "\n",
    "    token_ids = generate(\n",
    "        model=model,\n",
    "        idx=text_to_token_ids(input_text, tokenizer).to(device),\n",
    "        max_new_tokens=256,\n",
    "        context_size=BASE_CONFIG[\"context_length\"],\n",
    "        eos_id=50256\n",
    "    )\n",
    "    generated_text = token_ids_to_text(token_ids, tokenizer)\n",
    "    response_text = (\n",
    "        generated_text[len(input_text):]\n",
    "        .replace(\"### Response:\", \"\")\n",
    "        .strip()\n",
    ")\n",
    "\n",
    "    print(input_text)\n",
    "    print(f\"\\nCorrect response:\\n>> {entry['output']}\")\n",
    "    print(f\"\\nModel response:\\n>> {response_text.strip()}\")\n",
    "    print(\"-------------------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49ab64c1-586f-4939-8def-23feeb1b3599",
   "metadata": {
    "id": "49ab64c1-586f-4939-8def-23feeb1b3599"
   },
   "source": [
    "- As we can see based on the test set instructions, given responses, and the model's responses, the model performs relatively well\n",
    "- The answers to the first and last instructions are clearly correct\n",
    "- The second answer is close; the model answers with \"cumulus cloud\" instead of \"cumulonimbus\" (however, note that cumulus clouds can develop into cumulonimbus clouds, which are capable of producing thunderstorms)\n",
    "- Most importantly, we can see that model evaluation is not as straightforward as in the previous chapter, where we just had to calculate the percentage of correct spam/non-spam class labels to obtain the classification accuracy\n",
    "- In practice, instruction-finetuned LLMs such as chatbots are evaluated via multiple approaches\n",
    "  - short-answer and multiple choice benchmarks such as MMLU (\"Measuring Massive Multitask Language Understanding\", [https://arxiv.org/abs/2009.03300](https://arxiv.org/abs/2009.03300)), which test the knowledge of a model\n",
    "  - human preference comparison to other LLMs, such as LMSYS chatbot arena ([https://arena.lmsys.org](https://arena.lmsys.org))\n",
    "  - automated conversational benchmarks, where another LLM like GPT-4 is used to evaluate the responses, such as AlpacaEval ([https://tatsu-lab.github.io/alpaca_eval/](https://tatsu-lab.github.io/alpaca_eval/))\n",
    "\n",
    "- In the next section, we will use an approach similar to AlpacaEval and use another LLM to evaluate the responses of our model; however, we will use our own test set instead of using a publicly available benchmark dataset\n",
    "- For this, we add the model response to the `test_data` dictionary and save it as a `\"instruction-data-with-response.json\"` file for record-keeping so that we can load and analyze it in separate Python sessions if needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "-PNGKzY4snKP",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-PNGKzY4snKP",
    "outputId": "37b22a62-9860-40b7-c46f-b297782b944c"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 110/110 [01:20<00:00,  1.37it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "for i, entry in tqdm(enumerate(test_data), total=len(test_data)):\n",
    "\n",
    "    input_text = format_input(entry)\n",
    "\n",
    "    token_ids = generate(\n",
    "        model=model,\n",
    "        idx=text_to_token_ids(input_text, tokenizer).to(device),\n",
    "        max_new_tokens=256,\n",
    "        context_size=BASE_CONFIG[\"context_length\"],\n",
    "        eos_id=50256\n",
    "    )\n",
    "    generated_text = token_ids_to_text(token_ids, tokenizer)\n",
    "    response_text = generated_text[len(input_text):].replace(\"### Response:\", \"\").strip()\n",
    "\n",
    "    test_data[i][\"model_response\"] = response_text\n",
    "\n",
    "\n",
    "with open(\"instruction-data-with-response.json\", \"w\") as file:\n",
    "    json.dump(test_data, file, indent=4)  # \"indent\" for pretty-printing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "228d6fa7-d162-44c3-bef1-4013c027b155",
   "metadata": {
    "id": "228d6fa7-d162-44c3-bef1-4013c027b155"
   },
   "source": [
    "- Let's double-check one of the entries to see whether the responses have been added to the `test_data` dictionary correctly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "u-AvCCMTnPSE",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "u-AvCCMTnPSE",
    "outputId": "7bcd9600-1446-4829-b773-5259b13d256a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'instruction': 'Rewrite the sentence using a simile.', 'input': 'The car is very fast.', 'output': 'The car is as fast as lightning.', 'model_response': 'The car is as fast as a bullet.'}\n"
     ]
    }
   ],
   "source": [
    "print(test_data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1b2f3f6-8569-405a-9db6-d47cba65608a",
   "metadata": {
    "id": "c1b2f3f6-8569-405a-9db6-d47cba65608a"
   },
   "source": [
    "- Finally, we also save the model in case we want to reuse it in the future"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "8cBU0iHmVfOI",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8cBU0iHmVfOI",
    "outputId": "135849ed-9acd-43a2-f438-053d07dae9b2",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model saved as gpt2-medium355M-sft.pth\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "\n",
    "file_name = f\"{re.sub(r'[ ()]', '', CHOOSE_MODEL) }-sft.pth\"\n",
    "torch.save(model.state_dict(), file_name)\n",
    "print(f\"Model saved as {file_name}\")\n",
    "\n",
    "# Load model via\n",
    "# model.load_state_dict(torch.load(\"gpt2-medium355M-sft.pth\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "obgoGI89dgPm",
   "metadata": {
    "id": "obgoGI89dgPm"
   },
   "source": [
    "## 7.8 Evaluating the finetuned LLM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "805b9d30-7336-499f-abb5-4a21be3129f5",
   "metadata": {
    "id": "805b9d30-7336-499f-abb5-4a21be3129f5"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-7.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68d2b9d3-b6ff-4533-a89d-7b66079b4fd1",
   "metadata": {
    "id": "68d2b9d3-b6ff-4533-a89d-7b66079b4fd1"
   },
   "source": [
    "- In this section, we automate the response evaluation of the finetuned LLM using another, larger LLM\n",
    "- In particular, we use an instruction-finetuned 8-billion-parameter Llama 3 model by Meta AI that can be run locally via ollama ([https://ollama.com](https://ollama.com))\n",
    "- (Alternatively, if you prefer using a more capable LLM like GPT-4 via the OpenAI API, please see the [llm-instruction-eval-openai.ipynb](../03_model-evaluation/llm-instruction-eval-openai.ipynb) notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea427a30-36ba-44e3-bb1f-eb0d7008d6e9",
   "metadata": {
    "id": "ea427a30-36ba-44e3-bb1f-eb0d7008d6e9"
   },
   "source": [
    "- Ollama is an application to run LLMs efficiently\n",
    "- It is a wrapper around llama.cpp ([https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)), which implements LLMs in pure C/C++ to maximize efficiency\n",
    "- Note that it is a tool for using LLMs to generate text (inference), not training or finetuning LLMs\n",
    "- Before running the code below, install ollama by visiting [https://ollama.com](https://ollama.com) and following the instructions (for instance, clicking on the \"Download\" button and downloading the ollama application for your operating system)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "747a2fc7-282d-47ec-a987-ed0a23ed6822",
   "metadata": {
    "id": "747a2fc7-282d-47ec-a987-ed0a23ed6822"
   },
   "source": [
    "- For macOS and Windows users, click on the ollama application you downloaded; if it prompts you to install the command line usage, say \"yes\"\n",
    "- Linux users can use the installation command provided on the ollama website\n",
    "\n",
    "- In general, before we can use ollama from the command line, we have to either start the ollama application or run `ollama serve` in a separate terminal\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/ollama-run.webp?1\" width=700px>\n",
    "\n",
    "\n",
    "- With the ollama application or `ollama serve` running in a different terminal, on the command line, execute the following command to try out the 8-billion-parameter Llama 3 model (the model, which takes up 4.7 GB of storage space, will be automatically downloaded the first time you execute this command)\n",
    "\n",
    "```bash\n",
    "# 8B model\n",
    "ollama run llama3\n",
    "```\n",
    "\n",
    "\n",
    "The output looks like as follows\n",
    "\n",
    "```\n",
    "$ ollama run llama3\n",
    "pulling manifest\n",
    "pulling 6a0746a1ec1a... 100% ▕████████████████▏ 4.7 GB\n",
    "pulling 4fa551d4f938... 100% ▕████████████████▏  12 KB\n",
    "pulling 8ab4849b038c... 100% ▕████████████████▏  254 B\n",
    "pulling 577073ffcc6c... 100% ▕████████████████▏  110 B\n",
    "pulling 3f8eb4da87fa... 100% ▕████████████████▏  485 B\n",
    "verifying sha256 digest\n",
    "writing manifest\n",
    "removing any unused layers\n",
    "success\n",
    "```\n",
    "\n",
    "- Note that `llama3` refers to the instruction finetuned 8-billion-parameter Llama 3 model\n",
    "\n",
    "- Using ollama with the `\"llama3\"` model (a 8B parameter model) requires 16 GB of RAM; if this is not supported by your machine, you can try the smaller model, such as the 3.8B parameter phi-3 model by setting `model = \"phi-3\"`, which only requires 8 GB of RAM\n",
    "\n",
    "- Alternatively, you can also use the larger 70-billion-parameter Llama 3 model, if your machine supports it, by replacing `llama3` with `llama3:70b`\n",
    "\n",
    "- After the download has been completed, you will see a command line prompt that allows you to chat with the model\n",
    "\n",
    "- Try a prompt like \"What do llamas eat?\", which should return an output similar to the following\n",
    "\n",
    "```\n",
    ">>> What do llamas eat?\n",
    "Llamas are ruminant animals, which means they have a four-chambered\n",
    "stomach and eat plants that are high in fiber. In the wild, llamas\n",
    "typically feed on:\n",
    "1. Grasses: They love to graze on various types of grasses, including tall\n",
    "grasses, wheat, oats, and barley.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b7b341c-ba0e-40bb-a52c-cb328bbd1fe4",
   "metadata": {
    "id": "7b7b341c-ba0e-40bb-a52c-cb328bbd1fe4"
   },
   "source": [
    "- You can end this session using the input `/bye`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faaf3e02-8ca0-4edf-be23-60625a5b14e3",
   "metadata": {
    "id": "faaf3e02-8ca0-4edf-be23-60625a5b14e3"
   },
   "source": [
    "- The following code checks whether the ollama session is running correctly before proceeding to use ollama to evaluate the test set responses we generated in the previous section"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "026e8570-071e-48a2-aa38-64d7be35f288",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 193
    },
    "id": "026e8570-071e-48a2-aa38-64d7be35f288",
    "outputId": "e30d3533-e1f5-4aa9-b24f-33273fc7b30e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ollama running: True\n"
     ]
    }
   ],
   "source": [
    "import psutil\n",
    "\n",
    "def check_if_running(process_name):\n",
    "    running = False\n",
    "    for proc in psutil.process_iter([\"name\"]):\n",
    "        if process_name in proc.info[\"name\"]:\n",
    "            running = True\n",
    "            break\n",
    "    return running\n",
    "\n",
    "ollama_running = check_if_running(\"ollama\")\n",
    "\n",
    "if not ollama_running:\n",
    "    raise RuntimeError(\"Ollama not running. Launch ollama before proceeding.\")\n",
    "print(\"Ollama running:\", check_if_running(\"ollama\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "723c9b00-e3cd-4092-83c3-6e48b5cf65b0",
   "metadata": {
    "id": "723c9b00-e3cd-4092-83c3-6e48b5cf65b0"
   },
   "outputs": [],
   "source": [
    "# This cell is optional; it allows you to restart the notebook\n",
    "# and only run section 7.7 without rerunning any of the previous code\n",
    "import json\n",
    "from tqdm import tqdm\n",
    "\n",
    "file_path = \"instruction-data-with-response.json\"\n",
    "\n",
    "with open(file_path, \"r\") as file:\n",
    "    test_data = json.load(file)\n",
    "\n",
    "\n",
    "def format_input(entry):\n",
    "    instruction_text = (\n",
    "        f\"Below is an instruction that describes a task. \"\n",
    "        f\"Write a response that appropriately completes the request.\"\n",
    "        f\"\\n\\n### Instruction:\\n{entry['instruction']}\"\n",
    "    )\n",
    "\n",
    "    input_text = f\"\\n\\n### Input:\\n{entry['input']}\" if entry[\"input\"] else \"\"\n",
    "\n",
    "    return instruction_text + input_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3464705-d026-4594-977f-fb357e51c3a9",
   "metadata": {
    "id": "b3464705-d026-4594-977f-fb357e51c3a9"
   },
   "source": [
    "- Now, an alternative way to the `ollama run` command we used earlier to interact with the model is via its REST API in Python via the following function\n",
    "- Before you run the next cells in this notebook, make sure that ollama is still running (the previous code cells should print `\"Ollama running: True\"`)\n",
    "- Next, run the following code cell to query the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3ae0e10-2b28-42ce-8ea2-d9366a58088f",
   "metadata": {
    "id": "e3ae0e10-2b28-42ce-8ea2-d9366a58088f",
    "outputId": "cc43acb3-8216-43cf-c77d-71d4089dc96c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Llamas are herbivores, which means they primarily feed on plant-based foods. Their diet typically consists of:\n",
      "\n",
      "1. Grasses: Llamas love to graze on various types of grasses, including tall grasses, short grasses, and even weeds.\n",
      "2. Hay: High-quality hay, such as alfalfa or timothy hay, is a staple in a llama's diet. They enjoy the sweet taste and texture of fresh hay.\n",
      "3. Grains: Llamas may receive grains like oats, barley, or corn as part of their daily ration. However, it's essential to provide these grains in moderation, as they can be high in calories.\n",
      "4. Fruits and vegetables: Llamas enjoy a variety of fruits and veggies, such as apples, carrots, sweet potatoes, and leafy greens like kale or spinach.\n",
      "5. Minerals: Llamas require access to mineral supplements, which help maintain their overall health and well-being.\n",
      "\n",
      "In the wild, llamas might also eat:\n",
      "\n",
      "1. Leaves: They'll munch on leaves from trees and shrubs, including plants like willow, alder, and birch.\n",
      "2. Bark: In some cases, llamas may eat the bark of certain trees, like aspen or cottonwood.\n",
      "3. Mosses and lichens: These non-vascular plants can be a tasty snack for llamas.\n",
      "\n",
      "In captivity, llama owners typically provide a balanced diet that includes a mix of hay, grains, and fruits/vegetables. It's essential to consult with a veterinarian or experienced llama breeder to determine the best feeding plan for your llama.\n"
     ]
    }
   ],
   "source": [
    "import urllib.request\n",
    "\n",
    "def query_model(\n",
    "    prompt,\n",
    "    model=\"llama3\",\n",
    "    url=\"http://localhost:11434/api/chat\"\n",
    "):\n",
    "    # Create the data payload as a dictionary\n",
    "    data = {\n",
    "        \"model\": model,\n",
    "        \"messages\": [\n",
    "            {\"role\": \"user\", \"content\": prompt}\n",
    "        ],\n",
    "        \"options\": {     # Settings below are required for deterministic responses\n",
    "            \"seed\": 123,\n",
    "            \"temperature\": 0,\n",
    "            \"num_ctx\": 2048\n",
    "        }\n",
    "    }\n",
    "\n",
    "\n",
    "    # Convert the dictionary to a JSON formatted string and encode it to bytes\n",
    "    payload = json.dumps(data).encode(\"utf-8\")\n",
    "\n",
    "    # Create a request object, setting the method to POST and adding necessary headers\n",
    "    request = urllib.request.Request(\n",
    "        url,\n",
    "        data=payload,\n",
    "        method=\"POST\"\n",
    "    )\n",
    "    request.add_header(\"Content-Type\", \"application/json\")\n",
    "\n",
    "    # Send the request and capture the response\n",
    "    response_data = \"\"\n",
    "    with urllib.request.urlopen(request) as response:\n",
    "        # Read and decode the response\n",
    "        while True:\n",
    "            line = response.readline().decode(\"utf-8\")\n",
    "            if not line:\n",
    "                break\n",
    "            response_json = json.loads(line)\n",
    "            response_data += response_json[\"message\"][\"content\"]\n",
    "\n",
    "    return response_data\n",
    "\n",
    "\n",
    "model = \"llama3\"\n",
    "result = query_model(\"What do Llamas eat?\", model)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "207ae28f-0f8c-4fda-aeef-e7e3046249cc",
   "metadata": {
    "id": "207ae28f-0f8c-4fda-aeef-e7e3046249cc"
   },
   "source": [
    "- Now, using the `query_model` function we defined above, we can evaluate the responses of our finetuned model; let's try it out on the first 3 test set responses we looked at in a previous section"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86b839d4-064d-4178-b2d7-01691b452e5e",
   "metadata": {
    "id": "86b839d4-064d-4178-b2d7-01691b452e5e",
    "outputId": "1c755ee1-bded-4450-9b84-1466724f389a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Dataset response:\n",
      ">> The car is as fast as lightning.\n",
      "\n",
      "Model response:\n",
      ">> The car is as fast as a bullet.\n",
      "\n",
      "Score:\n",
      ">> I'd rate the model response \"The car is as fast as a bullet.\" an 85 out of 100.\n",
      "\n",
      "Here's why:\n",
      "\n",
      "* The response uses a simile correctly, comparing the speed of the car to something else (in this case, a bullet).\n",
      "* The comparison is relevant and makes sense, as bullets are known for their high velocity.\n",
      "* The phrase \"as fast as\" is used correctly to introduce the simile.\n",
      "\n",
      "The only reason I wouldn't give it a perfect score is that some people might find the comparison slightly less vivid or evocative than others. For example, comparing something to lightning (as in the original response) can be more dramatic and attention-grabbing. However, \"as fast as a bullet\" is still a strong and effective simile that effectively conveys the idea of the car's speed.\n",
      "\n",
      "Overall, I think the model did a great job!\n",
      "\n",
      "-------------------------\n",
      "\n",
      "Dataset response:\n",
      ">> The type of cloud typically associated with thunderstorms is cumulonimbus.\n",
      "\n",
      "Model response:\n",
      ">> The type of cloud associated with thunderstorms is a cumulus cloud.\n",
      "\n",
      "Score:\n",
      ">> I'd score this model response as 40 out of 100.\n",
      "\n",
      "Here's why:\n",
      "\n",
      "* The model correctly identifies that thunderstorms are related to clouds (correctly identifying the type of phenomenon).\n",
      "* However, it incorrectly specifies the type of cloud associated with thunderstorms. Cumulus clouds are not typically associated with thunderstorms; cumulonimbus clouds are.\n",
      "* The response lacks precision and accuracy in its description.\n",
      "\n",
      "Overall, while the model attempts to address the instruction, it provides an incorrect answer, which is a significant error.\n",
      "\n",
      "-------------------------\n",
      "\n",
      "Dataset response:\n",
      ">> Jane Austen.\n",
      "\n",
      "Model response:\n",
      ">> The author of 'Pride and Prejudice' is Jane Austen.\n",
      "\n",
      "Score:\n",
      ">> I'd rate my own response as 95 out of 100. Here's why:\n",
      "\n",
      "* The response accurately answers the question by naming the author of 'Pride and Prejudice' as Jane Austen.\n",
      "* The response is concise and clear, making it easy to understand.\n",
      "* There are no grammatical errors or ambiguities that could lead to confusion.\n",
      "\n",
      "The only reason I wouldn't give myself a perfect score is that the response is slightly redundant - it's not necessary to rephrase the question in the answer. A more concise response would be simply \"Jane Austen.\"\n",
      "\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "for entry in test_data[:3]:\n",
    "    prompt = (\n",
    "        f\"Given the input `{format_input(entry)}` \"\n",
    "        f\"and correct output `{entry['output']}`, \"\n",
    "        f\"score the model response `{entry['model_response']}`\"\n",
    "        f\" on a scale from 0 to 100, where 100 is the best score. \"\n",
    "    )\n",
    "    print(\"\\nDataset response:\")\n",
    "    print(\">>\", entry['output'])\n",
    "    print(\"\\nModel response:\")\n",
    "    print(\">>\", entry[\"model_response\"])\n",
    "    print(\"\\nScore:\")\n",
    "    print(\">>\", query_model(prompt))\n",
    "    print(\"\\n-------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24fec453-631f-4ff5-a922-44c3c451942d",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Note: Better evaluation prompt**\n",
    "\n",
    "- [A reader (Ayoosh Kathuria) suggested](https://github.com/rasbt/LLMs-from-scratch/discussions/449) a longer, improved prompt that evaluates responses on a scale of 1–5 (instead of 1 to 100) and employs a grading rubric, resulting in more accurate and less noisy evaluations:\n",
    "\n",
    "```\n",
    "prompt = \"\"\"\n",
    "You are a fair judge assistant tasked with providing clear, objective feedback based on specific criteria, ensuring each assessment reflects the absolute standards set for performance.\n",
    "You will be given an instruction, a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing the evaluation criteria.\n",
    "Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.\n",
    "Please do not generate any other opening, closing, and explanations.\n",
    "\n",
    "Here is the rubric you should use to build your answer:\n",
    "1: The response fails to address the instructions, providing irrelevant, incorrect, or excessively verbose information that detracts from the user's request.\n",
    "2: The response partially addresses the instructions but includes significant inaccuracies, irrelevant details, or excessive elaboration that detracts from the main task.\n",
    "3: The response follows the instructions with some minor inaccuracies or omissions. It is generally relevant and clear, but may include some unnecessary details or could be more concise.\n",
    "4: The response adheres to the instructions, offering clear, accurate, and relevant information in a concise manner, with only occasional, minor instances of excessive detail or slight lack of clarity.\n",
    "5: The response fully adheres to the instructions, providing a clear, accurate, and relevant answer in a concise and efficient manner. It addresses all aspects of the request without unnecessary details or elaboration\n",
    "\n",
    "Provide your feedback as follows:\n",
    "\n",
    "Feedback:::\n",
    "Evaluation: (your rationale for the rating, as a text)\n",
    "Total rating: (your rating, as a number between 1 and 5)\n",
    "\n",
    "You MUST provide values for 'Evaluation:' and 'Total rating:' in your answer.\n",
    "\n",
    "Now here is the instruction, the reference answer, and the response.\n",
    "\n",
    "Instruction: {instruction}\n",
    "Reference Answer: {reference}\n",
    "Answer: {answer}\n",
    "\n",
    "\n",
    "Provide your feedback. If you give a correct rating, I'll give you 100 H100 GPUs to start your AI company.\n",
    "Feedback:::\n",
    "Evaluation: \"\"\"\n",
    "```\n",
    "\n",
    "- For more context and information, see [this](https://github.com/rasbt/LLMs-from-scratch/discussions/449) GitHub discussion\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b114fd65-9cfb-45f6-ab74-8331da136bf3",
   "metadata": {
    "id": "b114fd65-9cfb-45f6-ab74-8331da136bf3"
   },
   "source": [
    "- As we can see, the Llama 3 model provides a reasonable evaluation and also gives partial points if a model is not entirely correct, as we can see based on the \"cumulus cloud\" answer\n",
    "- Note that the previous prompt returns very verbose evaluations; we can tweak the prompt to generate integer responses in the range between 0 and 100 (where 100 is best) to calculate an average score for our model\n",
    "- The evaluation of the 110 entries in the test set takes about 1 minute on an M3 MacBook Air laptop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d7bca69-97c4-47a5-9aa0-32f116fa37eb",
   "metadata": {
    "id": "9d7bca69-97c4-47a5-9aa0-32f116fa37eb",
    "outputId": "110223c0-90ca-481d-b2d2-f6ac46d3c4f0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Scoring entries: 100%|████████████████████████| 110/110 [01:10<00:00,  1.57it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of scores: 110 of 110\n",
      "Average score: 50.32\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "def generate_model_scores(json_data, json_key, model=\"llama3\"):\n",
    "    scores = []\n",
    "    for entry in tqdm(json_data, desc=\"Scoring entries\"):\n",
    "        prompt = (\n",
    "            f\"Given the input `{format_input(entry)}` \"\n",
    "            f\"and correct output `{entry['output']}`, \"\n",
    "            f\"score the model response `{entry[json_key]}`\"\n",
    "            f\" on a scale from 0 to 100, where 100 is the best score. \"\n",
    "            f\"Respond with the integer number only.\"\n",
    "        )\n",
    "        score = query_model(prompt, model)\n",
    "        try:\n",
    "            scores.append(int(score))\n",
    "        except ValueError:\n",
    "            print(f\"Could not convert score: {score}\")\n",
    "            continue\n",
    "\n",
    "    return scores\n",
    "\n",
    "\n",
    "scores = generate_model_scores(test_data, \"model_response\")\n",
    "print(f\"Number of scores: {len(scores)} of {len(test_data)}\")\n",
    "print(f\"Average score: {sum(scores)/len(scores):.2f}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "407f08d5-9ada-4301-9ebc-f0533c76d3f2",
   "metadata": {
    "id": "407f08d5-9ada-4301-9ebc-f0533c76d3f2"
   },
   "source": [
    "- Our model achieves an average score of above 50, which we can use as a reference point to compare the model to other models or to try out other training settings that may improve the model\n",
    "- Note that ollama is not fully deterministic across operating systems (as of this writing), so the numbers you are getting might slightly differ from the ones shown above"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6408768b-2784-44f1-b48e-aed0c1eb9b94",
   "metadata": {
    "id": "6408768b-2784-44f1-b48e-aed0c1eb9b94"
   },
   "source": [
    "- For reference, the original\n",
    "  - Llama 3 8B base model achieves a score of 58.51\n",
    "  - Llama 3 8B instruct model achieves a score of 82.65"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "412d7325-284a-446c-92a1-5aa8acc52dee",
   "metadata": {
    "id": "412d7325-284a-446c-92a1-5aa8acc52dee"
   },
   "source": [
    "## 7.9 Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tIbNMluCDjVM",
   "metadata": {
    "id": "tIbNMluCDjVM"
   },
   "source": [
    "### 7.9.1 What's next\n",
    "\n",
    "- This marks the final chapter of this book\n",
    "- We covered the major steps of the LLM development cycle: implementing an LLM architecture, pretraining an LLM, and finetuning it\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/final-overview.webp?1\" width=500px>\n",
    "\n",
    "- An optional step that is sometimes followed after instruction finetuning, as described in this chapter, is preference finetuning\n",
    "- Preference finetuning process can be particularly useful for customizing a model to better align with specific user preferences; see the [../04_preference-tuning-with-dpo](../04_preference-tuning-with-dpo) folder if you are interested in this\n",
    "\n",
    "- This GitHub repository also contains a large selection of additional bonus material you may enjoy; for more information, please see the [Bonus Material](https://github.com/rasbt/LLMs-from-scratch?tab=readme-ov-file#bonus-material) section on this repository's README page\n",
    "\n",
    "### 7.9.2 Staying up to date in a fast-moving field\n",
    "\n",
    "- No code in this section\n",
    "\n",
    "### 7.9.3 Final words\n",
    "\n",
    "- I hope you enjoyed this journey of implementing an LLM from the ground up and coding the pretraining and finetuning functions\n",
    "- In my opinion, implementing an LLM from scratch is the best way to understand how LLMs work; I hope you gained a better understanding through this approach\n",
    "- While this book serves educational purposes, you may be interested in using different and more powerful LLMs for real-world applications\n",
    "  - For this, you may consider popular tools such as axolotl ([https://github.com/OpenAccess-AI-Collective/axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)) or LitGPT ([https://github.com/Lightning-AI/litgpt](https://github.com/Lightning-AI/litgpt)), which I help developing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9853e7f-a81a-4806-9728-be1690807185",
   "metadata": {
    "id": "f9853e7f-a81a-4806-9728-be1690807185"
   },
   "source": [
    "## Summary and takeaways\n",
    "\n",
    "- See the [./gpt_instruction_finetuning.py](./gpt_instruction_finetuning.py) script, a self-contained script for instruction finetuning\n",
    "- [./ollama_evaluate.py](./ollama_evaluate.py) is a standalone script based on section 7.8 that evaluates a JSON file containing \"output\" and \"response\" keys via Ollama and Llama 3\n",
    "- The [./load-finetuned-model.ipynb](./load-finetuned-model.ipynb) notebook illustrates how to load the finetuned model in a new session\n",
    "- You can find the exercise solutions in [./exercise-solutions.ipynb](./exercise-solutions.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9cc51ec-e06c-4470-b626-48401a037851",
   "metadata": {
    "id": "b9cc51ec-e06c-4470-b626-48401a037851"
   },
   "source": [
    "## What's next?\n",
    "\n",
    "- Congrats on completing the book; in case you are looking for additional resources, I added several bonus sections to this GitHub repository that you might find interesting\n",
    "- The complete list of bonus materials can be viewed in the main README's [Bonus Material](https://github.com/rasbt/LLMs-from-scratch?tab=readme-ov-file#bonus-material) section\n",
    "- To highlight a few of my favorites:\n",
    "  1. [Direct Preference Optimization (DPO) for LLM Alignment (From Scratch)](../04_preference-tuning-with-dpo/dpo-from-scratch.ipynb) implements a popular preference tuning mechanism to align the model from this chapter more closely with human preferences\n",
    "  2. [Llama 3.2 From Scratch (A Standalone Notebook)](../../ch05/07_gpt_to_llama/standalone-llama32.ipynb), a from-scratch implementation of Meta AI's popular Llama 3.2, including loading the official pretrained weights; if you are up to some additional experiments, you can replace the `GPTModel` model in each of the chapters with the `Llama3Model` class (it should work as a 1:1 replacement)\n",
    "  3. [Converting GPT to Llama](../../ch05/07_gpt_to_llama) contains code with step-by-step guides that explain the differences between GPT-2 and the various Llama models\n",
    "  4. [Understanding the Difference Between Embedding Layers and Linear Layers](../../ch02/03_bonus_embedding-vs-matmul/embeddings-and-linear-layers.ipynb) is a conceptual explanation illustrating that the `Embedding` layer in PyTorch, which we use at the input stage of an LLM, is mathematically equivalent to a linear layer applied to one-hot encoded data\n",
    "- Happy further reading!"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "A100",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}