{ "cells": [ { "cell_type": "markdown", "id": "61bf9518", "metadata": {}, "source": [ "## CS535/EE514 - Spring 2023 - Assignment 5 - Neural Network\n", "\n", "\n", "\n", "#### Important Instructions and Submission Guidelines:\n", "- Submit your code both as notebook file (.ipynb). Naming convention for submission of this notebook is `RollNumber_PA5.ipynb` where. For example: `23100214_PA5.ipynb`\n", "- All the cells must be run once before submission. If your submission's cells are not showing the results (plots etc.), marks wil be deducted\n", "- Only the code written within this notebook's marked areas will be considered while grading. No other files will be entertained\n", "- You are advised to follow good programming practies including approriate variable naming and making use of logical comments.\n", "\n", "\n", "The university honor code should be maintained. Any violation, if found, will result in disciplinary action. " ] }, { "cell_type": "code", "execution_count": null, "id": "417e37fb", "metadata": {}, "outputs": [], "source": [ "#No other imports allowed\n", "import numpy as np\n", "from IPython.display import clear_output\n", "from keras.datasets import boston_housing\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": null, "id": "4e7883f1", "metadata": {}, "outputs": [], "source": [ "#Do not change this\n", "data = boston_housing.load_data()\n", "(x_train_valid, y_train_valid),(x_test, y_test) = (data[0][0], data[0][1]), (data[1][0], data[1][1])\n", "x_train = x_train_valid[:303]\n", "x_valid = x_train_valid[303:]\n", "y_train = y_train_valid[:303]\n", "y_valid = y_train_valid[303:]\n", "\n", "x_train = (x_train - x_train.mean())/(x_train.std())\n", "x_valid = (x_valid - x_valid.mean())/(x_valid.std())\n", "x_test = (x_test - x_test.mean())/(x_test.std())\n", "\n", "y_train = y_train.reshape((len(y_train), 1))\n", "y_valid = y_valid.reshape((len(y_valid), 1))\n", "y_test = y_test.reshape((len(y_test), 1))\n", "\n", "print(\"Number of training sample: \", len(x_train))\n", "print(\"Number of validation sample: \", len(x_valid))\n", "print(\"Number of testing sample: \", len(x_test))" ] }, { "cell_type": "markdown", "id": "ea1f2559", "metadata": {}, "source": [ "### Neural Network Implementation\n", "\n", "You are required to implement a neural network class from scratch for regression datasets and constraints. The neural network should be able to perform stochastic, batch and mini-batch methods of gradient descent. The class object receives the following **3** parameters: \n", " \n", "1. `nodes_per_layer`, it is a list of elements corresponding to number of neurons in each layer including the input and output layers.\n", "\n", "2. `method`, it is a string that can takes the values: 'sgd', 'batch' and 'minibatch'. It corresponds to the gradient descent method.\n", "\n", "3. `num_of_batches`, it is an integer corresponding to the number of batches to make for the minibatch method.\n", "\n", "The class should also have the following **8** functions:\n", "\n", "1. `__init__(self, nodes_per_layer, method = 'sgd', num_batches = 10)`. This function should initialize the variables it receives and also initialize the weights and biases for all layers except obviously the input layer. The initialization of weights should be done using the Xavier method to prevent weights from blowing up.\n", "\n", "2. `mean_squared_loss(self, y_pred, y_true)`. This function returns the mean squred loss using the arguments it receives.\n", "\n", "3. `sigmoid(self, x)`. This function applies the sigmoid function on outputs from a layer and return the output\n", "\n", "4. `forward_pass(self, input_data)`. This function receives the input layer and returns the lists: `layer_activations` and `layer_inputs`. Both the lists are identical except that `layer_activations` does not have the input layer as the first element while `layer_inputs` does. \n", "\n", "5. `backward_pass(self, targets, layer_activations, layer_inputs, lr)`. This function receives output of forward pass as This function performs backward pass first and then updates the weights corresponding to each layer. Weight update is done using lagrange multipliers according to constraints (for each node the weights should sum to 1).To do this first do the weight update normally, then calculate the lagrange multiplier using ($\\Sigma_i [w_i - 1]$) and finally subtract the lagrange multiplier from the weights of each node. It returns nothing.\n", "\n", "6. `fit(self, Xs, Ys, valid_Xs, valid_Ys, epochs, lr = 1e-3)`. This function should contain implementation for all 3 methods. For each method run forward pass to get layer outputs, find all layer inputs (including input layer itself), and use them to perfrom run backward pass. This trains the weights once. After this use the weights to find predictions for the train data and validation data. Store the training loss and validation loss. Live plot both the losses on one figure (this means the plot disappears and a new one appears as soon as an epoch is run, `clear_output(wait = True)` function is helpful for this). It returns the losses and accuracies.\n", "\n", "7. `predict(self, Xs)` This function returns the output of the final neuron layer for the data it receives as its argument. \n", "\n", "8. `def evaluate(self, Xs, Ys)` This function returns the mean_squared loss and $R^2$ based on the arguments it receives." ] }, { "cell_type": "code", "execution_count": null, "id": "77718096", "metadata": {}, "outputs": [], "source": [ "class NeuralNetwork:\n", " def __init__(self, nodes_per_layer, method='sgd', num_batches=10):\n", " pass\n", "\n", " def euclidean_loss(self, y_pred, y_true):\n", " pass\n", "\n", " def sigmoid(self, x):\n", " pass\n", "\n", " def forward_pass(self, input_data):\n", " pass\n", "\n", " def backward_pass(self, targets, layer_activations, layer_inputs, lr):\n", " pass\n", "\n", " def fit(self, Xs, Ys, valid_Xs, valid_Ys, epochs, lr=1e-3):\n", " train_losses, valid_losses = [], []\n", " for epoch in range(epochs):\n", " if self.method == 'sgd':\n", " pass\n", "\n", " elif self.method == 'batch':\n", " pass\n", "\n", " elif self.method == 'minibatch':\n", " pass\n", "\n", " # Calculate train loss and validation loss\n", " \n", "\n", " # Plot the losses\n", " pass\n", "\n", " return train_losses, valid_losses\n", "\n", " def predict(self, Xs):\n", " pass\n", "\n", " def evaluate(self, Xs, Ys):\n", " pass" ] }, { "cell_type": "code", "execution_count": null, "id": "3c3cbff8", "metadata": {}, "outputs": [], "source": [ "#Do not change anything here except the variable nodes_per_layer and the arguments method, num_batches, epochs, lr\n", "nodes_per_layer = [13, 128, 128, 1] \n", "\n", "# Create a NeuralNetwork object\n", "nn = NeuralNetwork(nodes_per_layer, method = \"sgd\", num_batches = 5)\n", "\n", "# Train the neural network\n", "train_losses, valid_losses = nn.fit(x_train, y_train, x_valid, y_valid, epochs = 100, lr = 1e-3)\n", "\n", "# Make predictions\n", "y_test_pred = nn.predict(x_test)\n", "print(\"Predictions:\", y_test_pred)\n", "\n", "# Evaluate the model\n", "mse, r2 = nn.evaluate(x_test, y_test)\n", "print(\"Mean Squared Error:\", mse)\n", "print(\"R-squared:\", r2)" ] }, { "cell_type": "code", "execution_count": null, "id": "f1139e4c", "metadata": {}, "outputs": [], "source": [ "# Do not change this. Run this cell to check whether your implementation obeys the constraints. \n", "# You should only see approximately 1 in the output of this cell.\n", "for weight_matrix in nn.weights:\n", " print(weight_matrix.sum(axis = 0))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.6" } }, "nbformat": 4, "nbformat_minor": 5 }