Submit the ipynb file as your final submission renamed as RollNumber_PA4.ipynb
\n",
"
The code MUST be implemented independently. Any plagiarism or cheating of work from others or the internet will be immediately referred to the DC.
\n",
"
10% penalty per day for 3 days after due date. No submissions will be accepted
\n",
"
Use procedural programming style and comment your code properly.
\n",
"
Deadline: (Put the deadline for the assignment here)
\n",
"
\n"
],
"metadata": {
"id": "a69MLCTxWcbh"
}
},
{
"cell_type": "code",
"source": [
"# All the necessary imports are made here if you want to include any of the other imports of your choice please do it in this cell\n",
"import numpy as np \n",
"import pandas as pd \n",
"import matplotlib.pyplot as plt \n",
"import seaborn as sns"
],
"metadata": {
"id": "wy12gAowXyxL"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"
First upload the dataset on your google drive and then run the cell below to mount your drive with the path for it.
Now after examining the dataset you have seen that the column names are too long and to cater for this run the code cell below to rename the column names to the standard column names.
Now we will see if our dataset contains null values and if it contains the null values you are to drop them. Please write appropriate code below to check for null values in the dataset and drop them if necessary.
Now we will use a suitable visualization to visualize the outliers if present in the dataset. Remember to not include the column with the name of `target_class` in the visualizations you create.
Now, we will use the SVM classifier to classify the outliers correctly and to the classes as we can see there are two unique values in target variable mapping to 0 and 1. So in this part you have to implement the linear SVM from scratch and follow the following steps.
"
],
"metadata": {
"id": "VVYNVkPyhMWY"
}
},
{
"cell_type": "markdown",
"source": [
"Step 1: First drop the target variable named as `target_class` and store it seperately from the dataset. You would have to store the rest of the dataset in another variable."
],
"metadata": {
"id": "GUsLX8hlsKSj"
}
},
{
"cell_type": "code",
"source": [
" # Store the target class column in the variable 'y' and the rest of the dataset in the variable 'X'.\n",
" y = dataset_pulstar['target_class']\n",
" X = dataset_pulstar.drop(['target_class'],axis=1)\n",
" print(y.shape,X.shape)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "6WC5naIRhLtQ",
"outputId": "1c8b758b-3969-4de8-cb12-7ca2b05d0dc8"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"(9273,) (9273, 8)\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"Step 2: Now you will split the datasets into test and train (80-20 or 70-30). So use the code cell below to make appropriate splits."
],
"metadata": {
"id": "ABfb4XGmticO"
}
},
{
"cell_type": "code",
"source": [
"#Please write your code below\n",
"rows = X.shape[0]\n",
"#Performing an 80-20 Split\n",
"X_train = X[:int(0.8*(rows))]\n",
"X_test = X[int(0.8*(rows)):]\n",
"y_train = y[:int(0.8*(rows))]\n",
"y_test = y[int(0.8*(rows)):]\n",
"\n",
"print(X_train.shape,X_test.shape,y_train.shape,y_test.shape)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "GE6qBTQhthtn",
"outputId": "537c435f-9efe-46ab-a579-16050eb58c0a"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"(7418, 8) (1855, 8) (7418,) (1855,)\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"Step 3: Now you have to standardize the columns in the dataset (X_train) and the columns in the dataset (X_test) so that the values are centered."
],
"metadata": {
"id": "68Tel-0FvQIn"
}
},
{
"cell_type": "code",
"source": [
"X_train_std = (abs(X_train-np.mean(X_train,axis=0)))/np.std(X_train,axis=0)\n",
"X_test_std = (abs(X_test-np.mean(X_test,axis=0)))/np.std(X_test,axis=0)"
],
"metadata": {
"id": "iL-cnYkLvPOw"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Step 4: You have to now implement the Linear SVM model and you are provided with the boiler plate code in the cell below:\n",
"Some formulas for the derivatives that might be useful are attached as snippets for your information."
],
"metadata": {
"id": "KUTL3jiF8WJa"
}
},
{
"cell_type": "markdown",
"source": [
"The equation that we will follow for calculating the hyperplane that best fits our model is given below:"
],
"metadata": {
"id": "PC2oAHmH7y3F"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "el_6no5q7wEz"
}
},
{
"cell_type": "markdown",
"source": [
"We can break the equation above into two steps which is potrayed by the step function below:"
],
"metadata": {
"id": "48X-bnro77yU"
}
},
{
"cell_type": "markdown",
"source": [
""
],
"metadata": {
"id": "WiT0xh7977rV"
}
},
{
"cell_type": "markdown",
"source": [
"Now the derivatives for the step function are specified as follows:"
],
"metadata": {
"id": "ejEKyklq8Qqu"
}
},
{
"cell_type": "markdown",
"source": [
"Case 1: \n",
""
],
"metadata": {
"id": "Pe82zIsX8b96"
}
},
{
"cell_type": "markdown",
"source": [
"Case 2:\n",
" \n",
""
],
"metadata": {
"id": "37hIqk_48oqI"
}
},
{
"cell_type": "code",
"source": [
"class SVM:\n",
" def __init__(self, learning_rate=1e-3, lambda_param=1e-2, n_iters=1000):\n",
" '''\n",
" This is the constructor for our SVM class and its passed 3 parameters which are defined as follows:\n",
" -> Learning_rate: This is the step_size that you would take in order for your algorithm to converge\n",
" -> lambda_param: This specifies the constant which is used for regularization as taught to you in class\n",
" -> n_iters: This specifies the epochs that you would take for the algorithm to converge.\n",
" '''\n",
" self.lr = learning_rate\n",
" self.lambda_param = lambda_param\n",
" self.n_iters = n_iters\n",
" self.w = None\n",
" self.b = None\n",
"\n",
" def initialize_weights_and_bias(self, X):\n",
" '''\n",
" In this function you are to initialize the weights initally to zero. The variables passed as arguments are as follows:\n",
" -> X: This is the original dataset X which will help you specify the n_features that you should have. \n",
" \n",
" TODO: You have to initialize the weights of the model and the bias variable to zero.\n",
" '''\n",
" n_features = X.shape[1]\n",
" self.w = np.zeros(n_features)\n",
" self.b = 0\n",
"\n",
" def get_classfication_map(self, y):\n",
" '''\n",
" In this function you will map you binary target variables to either -1 and 1 as taught \n",
" to you in class forming the two margins to your actual hyperplane. You are given with the variable y as the parameter which is defined below:\n",
" -> y: Here y, represents a single target value corresponding to a datapoint in your train dataset.\n",
" \n",
" TODO: You would have to return a value -1 or 1 if the value of the variable y is less than or equal to zero and 1 otherwise.\n",
" '''\n",
" return np.where(y <= 0, -1, 1)\n",
"\n",
" def constraint_satisfaction(self, x, idx):\n",
" '''\n",
" In this function you are given with the following arguments:\n",
" -> x: Here x, represents the datapoint from your train dataset.\n",
" -> idx: This idx variable represents an index that maps to the current datapoint x that you are processing.\n",
" \n",
" TODO: \n",
" -> You have to make up a linear model by multiplying the weights \n",
" and adding the bias term to it.\n",
" -> Make up a predicate which returns true if product of the linear_model \n",
" value with the class label is greater than or equal to 1 and otherwise false.\n",
" '''\n",
" linear_model = np.dot(x, self.w) + self.b \n",
" return self.cls_map[idx] * linear_model >= 1\n",
" \n",
" def cal_gradients(self, constrain, x, idx):\n",
" '''\n",
" In this function you are given the following parameters:\n",
" -> constrain: This is a bool which will help you calculate the derviatives \n",
" of the two cases where the datapoint either lies above the hyperplane or below the hyperplane.\n",
" -> x: Here x, represents an individual datapoint from your train dataset.\n",
" -> idx: This idx variable represents an index that maps to the current datapoint x that you are processing.\n",
"\n",
" TODO: \n",
" -> Here you will handle the two cases where your datapoint lies above \n",
" the hyperplane or below it with the help of the constrain (boolean).\n",
" -> if the datapoint lies above the hyperplane:\n",
" * Then calculate variables dw & db by using the equations specified above.\n",
" Otherwise:\n",
" * Use the formulas specified above to calcualte dw & db.\n",
" '''\n",
" if constrain:\n",
" dw = self.lambda_param * self.w\n",
" db = 0\n",
" return dw, db\n",
" dw = self.lambda_param * self.w - np.dot(self.cls_map[idx], x)\n",
" db = - self.cls_map[idx]\n",
" return dw, db\n",
" \n",
" def update_weights_and_bias(self, dw, db):\n",
" '''\n",
" In this function you are provided with the following parameters:\n",
" -> dw: This represents the derivatives for your weights as calculated by the cal_gradients function.\n",
" -> db: This represnets the derivatives for your biases as calcuated by the cal_graidents function.\n",
"\n",
" TODO:\n",
" You have to simply update the weights by multiplying it with the learning rate as you do in gradient descent.\n",
" '''\n",
" self.w -= self.lr * dw\n",
" self.b -= self.lr * db\n",
" \n",
" def fit(self, X, y):\n",
" '''\n",
" This is the fit function in which you are provided with the following parameters:\n",
" X: The complete train dataset.\n",
" y: Your target values associated with the train dataset.\n",
"\n",
" TODO:\n",
" -> First you would need to initialize your weights and biases by calling your implementation of the following functions:\n",
" * initialize_weights_and_bias()\n",
" * get_classfication_map -> store the return value in the variable self.cls_map for future reference.\n",
" -> Now you would have to run a loop for your number of epochs and \n",
" for each epoch you would have to check every datapoint for the constraint \n",
" and then calculate the gradients and update them accordingly\n",
" '''\n",
" self.initialize_weights_and_bias(X)\n",
" self.cls_map = self.get_classfication_map(y)\n",
"\n",
" for _ in range(self.n_iters):\n",
" for idx, x in enumerate(X):\n",
" constrain = self.constraint_satisfaction(x, idx)\n",
" dw, db = self.cal_gradients(constrain, x, idx)\n",
" self.update_weights_and_bias(dw, db)\n",
" \n",
" def predict(self, X):\n",
" '''\n",
" In this function you would have to predict whether your hyperplane that you have fitted on the data is correct or not.\n",
" The parameters to the function are the following:\n",
" -> X: Reperesents your train dataset.\n",
"\n",
" TODO: After running the fit function take the dot product of X with your updated weights and up the biases.\n",
" Make use of the np.sign function and np.where function to map your -1 labels to 0 and 1.\n",
"\n",
" Rember: You mapped the values to -1 and 1 in the constraint_satsifaction function so you have to map them back to 0 and 1 respectively.\n",
" '''\n",
" estimate = np.dot(X, self.w) + self.b\n",
" prediction = np.sign(estimate)\n",
" return np.where(prediction == -1, 0, 1)"
],
"metadata": {
"id": "wgZFm2Zyx59C"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Convert your datasets to numpy arrays here. Please write your code below:\n",
"X_dataset_numpy = np.array(X_train)\n",
"y_dataset_numpy = np.array(y_train)\n",
"X_test_numpy = np.array(X_test)"
],
"metadata": {
"id": "IEgXqeHNzQqw"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Make a call to the class by initliazing the n_iters or the epochs and then call the fit function passing them the appropriate arguments. \n",
"# After doing this you are to call the predict function by passing in the appropriate arguments as well\n",
"clf = SVM(n_iters=300)\n",
"clf.fit(X_dataset_numpy, y_dataset_numpy)\n",
"predictions = clf.predict(X_test_numpy)"
],
"metadata": {
"id": "lnOcahc2yK5e"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Define your manual accuracy function here. Please write the code below:\n",
"def accuracy(y_true, y_pred):\n",
" accuracy = np.sum(y_true==y_pred) / len(y_true)\n",
" return accuracy"
],
"metadata": {
"id": "dqN3Q6qW0MBd"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Print the accuracy you are getting from the linear model below:\n",
"print(\"SVM Accuracy: \", accuracy(y_test, predictions)*100)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "6GzXPen-0Q_-",
"outputId": "668b6d84-fa57-439e-de61-cb83a247d82d"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"SVM Accuracy: 97.25067385444744\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"### Problem with dispersed datasets\n",
"\n",
"\n",
"Sometimes, the sample data points are so dispersed that it is not possible to separate them using a linear hyperplane. \n",
"In such a situation, SVMs uses a `kernel trick` to transform the input space to a higher dimensional space as shown in the diagram below. It uses a mapping function to transform the 2-D input space into the 3-D input space. Now, we can easily segregate the data points using linear separation.\n",
"\n",
"\n",
"### Kernel trick - transformation of input space to higher dimensional space\n",
"\n",
""
],
"metadata": {
"id": "GqlC0jr-6aj6"
}
},
{
"cell_type": "markdown",
"source": [
"# **Kernel trick** \n",
"\n",
"[Table of Contents](#0.1)\n",
"\n",
"\n",
"In practice, SVM algorithm is implemented using a `kernel`. It uses a technique called the `kernel trick`. In simple words, a `kernel` is just a function that maps the data to a higher dimension where data is separable. A kernel transforms a low-dimensional input data space into a higher dimensional space. So, it converts non-linear separable problems to linear separable problems by adding more dimensions to it. Thus, the kernel trick helps us to build a more accurate classifier. Hence, it is useful in non-linear separation problems."
],
"metadata": {
"id": "pv4oOx0b6jzN"
}
},
{
"cell_type": "markdown",
"source": [
"In the context of SVMs, there are 4 popular kernels – `Linear kernel`,`Polynomial kernel`,`Radial Basis Function (RBF) kernel` (also called Gaussian kernel) and `Sigmoid kernel`. We will be focussing on the polynomial kernel."
],
"metadata": {
"id": "qfPvlJ4X69mO"
}
},
{
"cell_type": "markdown",
"source": [
"# **Polynomial Kernel**\n",
"\n",
"Polynomial kernel represents the similarity of vectors (training samples) in a feature space over polynomials of the original variables. The polynomial kernel looks not only at the given features of input samples to determine their similarity, but also combinations of the input samples.\n",
"\n",
"For degree-d polynomials, the polynomial kernel is defined as follows –\n",
"\n",
"**Polynomial kernel : K(xi , xj ) = (γxiT xj + r)d , γ > 0**\n",
"\n",
"Polynomial kernel is very popular in Natural Language Processing. The most common degree is d = 2 (quadratic), since larger degrees tend to overfit on NLP problems. It can be visualized with the following diagram.\n",
"\n",
"### Polynomial Kernel\n",
"\n",
""
],
"metadata": {
"id": "KVb9BxLM7HTi"
}
},
{
"cell_type": "markdown",
"source": [
"# Use a Non-linear SVM model to classify your dataset.\n",
"Write a function, which takes among its parameters, the coefficient C and generates an SVM classifier. Then it trains the classifier using the training data and then runs the classifier on the test data. \n",
"Using these results predict the accuracy and F1-score for each value of C. \n",
"Note: Remember to generalize this function so that it may run for different values of C. "
],
"metadata": {
"id": "ifB-fmTp-vd_"
}
},
{
"cell_type": "markdown",
"source": [
"## Run SVM with polynomial kernel and C=1.0"
],
"metadata": {
"id": "WakRGa6a72RG"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0GgdNabd5nlh"
},
"outputs": [],
"source": [
"### Code Here\n",
"\n",
"### ----------"
]
},
{
"cell_type": "markdown",
"source": [
"## Run SVM with polynomial kernel and C=100.0"
],
"metadata": {
"id": "DSLTPxj58VvK"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "g8LzoYlzMjh3"
},
"outputs": [],
"source": [
"### Code Here\n",
"\n",
"### ----------"
]
},
{
"cell_type": "markdown",
"source": [
"## Run SVM with polynomial kernel and C=1000.0"
],
"metadata": {
"id": "k12vJHBQ8X6z"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "UjTERVxmMjqP"
},
"outputs": [],
"source": [
"### Code Here\n",
"\n",
"### ----------"
]
},
{
"cell_type": "markdown",
"source": [
"#Results\n",
"Plot the confusion matrix for each of the above three instances. You may use any libraries you may wish to.\n"
],
"metadata": {
"id": "Kfyst51Q96GK"
}
},
{
"cell_type": "code",
"source": [
"### Code Here\n",
"\n",
"### ----------"
],
"metadata": {
"id": "_ILueeNj-my8"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Analysis\n",
"a) For which value of C did the polynomial kernel give the best results? \n",
"b) Compare the results of your linear and your best polynomial kernel model, in part (a)?"
],
"metadata": {
"id": "RzBjwGmg8aoh"
}
},
{
"cell_type": "markdown",
"source": [
"Write your answer here."
],
"metadata": {
"id": "STvNS9bI9DOI"
}
}
]
}