Peer-graded Assignment: Project Asset Submission¶
The grading criteria:¶
- Use Case
- Data Set
- Data Quality Assessment
- Data Exploration (e.g. correlation between columns)
- Data Visualization (e.g. value distribution of columns)
- At least one Feature Engineering (e.g. imputing missing values) applied
- Selection and justification of Model Performance Indicator (e.g. F1 score)
- At least one traditional Machine Learning Algorithm and one DeepLearning
- Algorithm applied and demonstrated
- Model performance between different feature engineerings and models compared and documented
Answer the following¶
Please assign one point for each item below which is properly covered in the ADD
- Why have I chosen a specific method for data quality assessment?
- Why have I chosen a specific method for feature engineering?
- Why have I chosen a specific algorithm?
- Why have I chosen a specific framework?
- Why have I chosen a specific model performance indicator?
Points to remember¶
Please review the notebooks and assign points according to how many of the following items are properly covered
- The notebooks are clearly structured and documented
- The notebooks contain visualizations in the data quality assessment steps
- The notebooks reveal the overall data processing steps ETL->Feature
- Engineering->Model Training/Evaluation/Selection->Final Result
- The notebooks contain visualizations in the model performance assessment steps
- The notebooks show the answers to the questions defined in the Usecase
Please upload a PDF of your ADD (Architectural Decision Document)
# set tensorflow to 1.x for colab (Many packages wont work/moved to different package in tensorflow 2.2)
%tensorflow_version 1.x
# importing libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# Importing the WISCONSIN Breast Cancer Dataset
data = pd.read_csv('WBCDdata.csv')
# First 5 entries of the dataset
data.head()
# Features provided in the data
data.columns
# Check if any missing or NaN values
data.info
Now we can see Unnamed:32 have 0 non null object it means the all values are null in this column so we cannot use this column for our analysis*
Clean and Normalization Data¶
As you can see there are useless columns, like “id” and “Unnamed: 32”. Let’s drop them. Also We need to change categorical data to numeric data.
data.drop("id", axis=1, inplace=True)
data.drop("Unnamed: 32",axis=1,inplace=True)
data.diagnosis = [1 if each == "M" else 0 for each in data.diagnosis]
# Division of data into Beningn and Malignant(0 and 1)
sns.countplot(data['diagnosis'],label="Count")
Principle Component Analysis¶
The purpose for doing principal component analysis on the labeled data here is to observe the variance explained by each of the components and the associated weights assigned to each feature. The resulting output will aid in deciding on which features to drop.
from sklearn.decomposition import PCA
# useful_data
useful_data = data.iloc[:,1:]
pca = PCA(n_components=3)
pca.fit(useful_data)
# Dimmension indexing
dimmensions = ['Dimension {}'.format(i) for i in range(1,len(pca.components_)+1)]
# Individual PCA Components
components = pd.DataFrame(np.round(pca.components_, 4),columns=useful_data.keys())
components.index = dimmensions
# Explained variance in PCA
ratios = pca.explained_variance_ratio_.reshape(len(pca.components_),1)
variancce_rations = pd.DataFrame(np.round(ratios,4),columns=['Explained Variance'])
variancce_rations.index = dimmensions
print(pd.concat([variancce_rations, components], axis=1))
Observations¶
It can be observed that 98.20% of the variance is explained in dimmension 1. This means that nearly all of the variance in the data can be described by one dimmension. The remaining two describe a much smaller amount of the variance
In dimmension 1, most of the feature weight is associated with the area_mean and area_worst dimmension. Due to this, in the next step, I will visualize how well each of the mean features as well as area_worst and perimeter_worst explain the resulting diagnosis.
While we will not be using PCA in the actual machine learning phase, this describes the data well and helps understand which features should be further investogated for their importance in the final prediction.
Feature Selection¶
# Observe correlation to the diagnosis
tst = data.corr()['diagnosis'].copy()
tst = tst.drop('diagnosis')
tst.sort_values(inplace=True)
tst.plot(kind='bar', alpha=0.6)
plt.show()
# Seperate out maligant and benign data for graphing
malignant = data[data['diagnosis'] == 1]
benign = data[data['diagnosis'] == 0]
# Column names to observe in following graphs - mean values only
observe = list(data.columns[1:11]) + ['area_worst'] + ['perimeter_worst']
observables = data.loc[:,observe]
plt.rcParams.update({'font.size': 8})
plot, graphs = plt.subplots(nrows=6, ncols=2, figsize=(8,10))
graphs = graphs.flatten()
for idx, graph in enumerate(graphs):
graph.figure
binwidth= (max(data[observe[idx]]) - min(data[observe[idx]]))/50
bins = np.arange(min(data[observe[idx]]), max(data[observe[idx]]) + binwidth, binwidth)
graph.hist([malignant[observe[idx]],benign[observe[idx]]], bins=bins, alpha=0.6, density=True, label=['Malignant','Benign'], color=['red','blue'])
graph.legend(loc='upper right')
graph.set_title(observe[idx])
plt.tight_layout()
Observations¶
From the graphs, we can see that radius_mean, perimeter_mean, area_mean, concavity_mean and concave_points_mean are useful in predicting cancer type due to the distinct grouping between malignant and benign cancer types in these features. We can also see that area_worst and perimeter_worst are also quite useful.
color_wheel = {0: "blue", 1: "red"}
colors = data["diagnosis"].map(lambda x: color_wheel.get(x))
pd.plotting.scatter_matrix(observables, c=colors, alpha = 0.5, figsize = (15, 15), diagonal = 'kde');
Observations¶
The scatter matrix clarifies a few more points. The perimeter_mean, area_mean and radius mean have a strong, positive, linear correlation. Most other data also has a more rough linear correlation to other features with the exception of fractal_dimension_mean, symmetry_mean and smoothness_mean.
Within these three features we can see quite a bit of mixing between malignant and benign cancer in the scatter matrix. This suggests that our assumption above, that they do not aid in predicting cancer type, is likely correct. There is less correlation and separability between the two diagnoses.
Due to the lack of clear separability and lack of variance explained I feel comfortable dropping them.
Trimming Data¶
From observing the graphs and PCA data above: fractal_dimension_mean, smoothness_mean and symmetry_mean are not very useful in predicting the type of cancer. To aid in the learning process and remove noise, these columns will be dropped.
# Drop columns that do not aid in predicting type of cancer
observables.drop(['fractal_dimension_mean', 'smoothness_mean', 'symmetry_mean'],axis=1,inplace=True)
Classification¶
Here a comparison will be made between the different types of learning algorithms. At the end a breakdown of the data and explanation of the algorithm’s performance will be made.
# Split data appropriately
X = observables
y = data['diagnosis']
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
#from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)
KNN¶
from sklearn.neighbors import KNeighborsClassifier
## from sklearn.cross_validation import cross_val_score ## ERROR
from sklearn.model_selection import cross_val_score
knn=KNeighborsClassifier()
Parameter tuning: optimizing the number of neighbors (k)
For optimization purposes it is suggested using the training data; and for performance estimation or to figure out the best model, the test data will be used.
k_range=list(range(1,50))
k_scores=[]
for k in k_range:
knn=KNeighborsClassifier(n_neighbors=k)
scores=cross_val_score(knn, X_train,y_train,cv=10,scoring='recall')
k_scores.append(scores.mean())
print(np.round(k_scores,3)) # to display scores to 3 decimal places
from matplotlib import pyplot as plt
plt.plot(k_range,k_scores,color="red")
plt.xlabel('k values')
plt.ylabel('Recall')
plt.show()
From the above plot the optimal k value is somewhere between 1 and 10. Looking at the scores, at k=5 we have the highest recall/sensitivity. We can use a more efficient parameter tuning approach called GridSearchCV as shown below which basically automates the search process and it also fits the optimal parameter to the training data.
## from sklearn.grid_search import GridSearchCV ## ERROR
from sklearn.model_selection import learning_curve, GridSearchCV
k_range=list(range(1,50))
param_grid=dict(n_neighbors=k_range)
scores = ['accuracy', 'recall'] # evaluation scores for selecting best parameter
for sc in scores:
grid_knn=GridSearchCV(knn,param_grid,cv=10,scoring=sc,n_jobs=-1)
print("# Tuning hyper-parameters for %s" % sc)
grid_knn.fit(X_train,y_train)
print(grid_knn.best_params_)
print(np.round(grid_knn.best_score_,3))
Here, the optimal n_neighbors according to recall score (sensitivity) k=7 which is same for that of accuracy score.
SVM¶
## from sklearn.cross_validation import cross_val_score ## ERROR
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
svc=SVC()
param_grid = [{'kernel': ['rbf'], 'gamma': [1e-3, 1e-4, 1e-5],'C': [1, 10, 100, 1000]},
{'kernel': ['linear'], 'C': [1, 10, 100, 1000]}]
scores = ['accuracy', 'recall']
for sc in scores:
grid=GridSearchCV(svc,param_grid,cv=10,scoring=sc,n_jobs=-1)
print("# Tuning hyper-parameters for %s" % sc)
grid.fit(X,y)
print(grid.best_params_)
print(np.round(grid.best_score_,3))
Using accuracy and recall as scoring metrics, both give the same model as optimal i.e. {‘C’: 100, ‘kernel’: ‘linear’}. Here below, we fit this model to the data and do visualization of the results.
Logistic Regression¶
from sklearn.linear_model import LogisticRegression
#from sklearn.model_selection import cross_val_score
#from sklearn.grid_search import GridSearchCV
#from sklearn.model_selection import learning_curve, GridSearchCV
lgr = LogisticRegression()
#parameter tuning
param_grid = {'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000] }
scores = ['accuracy', 'recall']
for sc in scores:
grid_lgr=GridSearchCV(lgr,param_grid,cv=10,scoring=sc,n_jobs=-1)
print("# Tuning hyper-parameters for %s" % sc)
grid_lgr.fit(X_train,y_train)
print(grid_lgr.best_params_)
print(np.round(grid_lgr.best_score_,3))
SVC and Logistic regression scored similar performance, but we should be careful since the model has not yet tested on test data or on data independent of the one used for training the models. Below, we fit the three models onto the test data (X_test) and compare the predictions (y_predict) with observations (y_test). For this purpose, various evaluation metrics are implemented.
Random Forest¶
################## Random Forest #################################
from sklearn.ensemble import RandomForestClassifier
# creating a model
model = RandomForestClassifier(n_estimators = 400, max_depth = 10)
# feeding the training set into the model
model.fit(X_train, y_train)
# predicting the test set results
y_pred = model.predict(X_test)
# Calculating the accuracies
print("Training accuracy :", model.score(X_train, y_train))
print("Testing accuarcy :", model.score(X_test, y_test))
# classification report
cr = classification_report(y_test, y_pred)
print(cr)
# confusion matrix
cm = confusion_matrix(y_test, y_pred)
print(cm)
Decision Tree¶
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
# creating a model
model = DecisionTreeClassifier()
# feeding the training set into the model
model.fit(X_train, y_train)
# predicting the test set results
y_pred = model.predict(X_test)
# Calculating the accuracies
print("Training accuracy :", model.score(X_train, y_train))
print("Testing accuarcy :", model.score(X_test, y_test))
# classification report
cr = classification_report(y_test, y_pred)
print(cr)
# confusion matrix
cm = confusion_matrix(y_test, y_pred)
print(cm)
Neural Networks – Multi Layer Perceptron¶
from sklearn.neural_network import MLPClassifier
# creating the model
model = MLPClassifier(hidden_layer_sizes = (200, 200), max_iter = 200)
# feeding the training set into the model
model.fit(X_train, y_train)
# predicting the test set results
y_pred = model.predict(X_test)
# Calculating the accuracies
print("Training accuracy :", model.score(X_train, y_train))
print("Testing accuarcy :", model.score(X_test, y_test))
# classification report
cr = classification_report(y_test, y_pred)
print(cr)
# confusion matrix
cm = confusion_matrix(y_test, y_pred)
print(cm)
Model Selection¶
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
# KNN
# fitting the knn model on the training data
knn=KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
y_pred_knn =knn.predict(X_test)
# computing and plotting confusion matrix
c_m = confusion_matrix(y_test, y_pred_knn)
print('KNN:\n confusion matrix\n', c_m,'\n\n')
ax=plt.matshow(c_m,cmap=plt.cm.Reds)
print('Confusion matrix plot of KNN classifier')
plt.colorbar(ax)
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()
# classification report
print('\n Classification report \n',classification_report(y_test, y_pred_knn))
print ('#############################################################################')
# SVC
# fitting the SVC model on the training data and predicting for test data
svc=SVC(C=100,gamma=0.001,kernel='rbf',probability=True)
svc.fit(X_train, y_train)
y_pred_svc =svc.predict(X_test)
# computing and plotting confusion matrix
c_m = confusion_matrix(y_test, y_pred_svc)
print('SVC:\n confusion matrix\n', c_m,'\n\n')
ax=plt.matshow(c_m,cmap=plt.cm.Reds)
print('Confusion matrix plot of SVC')
plt.colorbar(ax)
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()
# classification report
print('\n Classification report \n',classification_report(y_test, y_pred_svc))
print ('#############################################################################')
# Logistic Regression
# fitting the lgr model on the training data
lgr=LogisticRegression(C=1)
lgr.fit(X_train, y_train)
y_pred_lgr =lgr.predict(X_test)
# computing and plotting confusion matrix
c_m = confusion_matrix(y_test, y_pred_lgr)
print('Logistic Regression:\nconfusion matrix\n', c_m,'\n\n')
ax=plt.matshow(c_m,cmap=plt.cm.Reds)
print('Confusion matrix plot of Logistic regression')
plt.colorbar(ax)
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()
# classification report
print('\n Classification report \n',classification_report(y_test, y_pred_lgr))
Make table here for all three methods with precision recall f1score
from sklearn import metrics
# false positive rate,fpr= FP/(TN+FP) OR fpr=1-specificty, tpr=sensitivity
y_pred_knn_p =knn.predict_proba(X_test)[:,1]
y_pred_svc_p =svc.predict_proba(X_test)[:,1]
y_pred_lgr_p =lgr.predict_proba(X_test)[:,1]
models=[y_pred_knn_p,y_pred_svc_p,y_pred_lgr_p]
label=['KNN','SVC','Logistic']
# plotting ROC curves
plt.figure(figsize=(10, 8))
m=np.arange(3)
for m in m:
fpr, tpr,thresholds= metrics.roc_curve(y_test,models[m])
print('model:',label[m])
print('thresholds:',np.round(thresholds,3))
print('tpr: ',np.round(tpr,3))
print('fpr: ',np.round(fpr,3))
plt.plot(fpr,tpr,label=label[m])
plt.xlim([0.0,1.0])
plt.ylim([0.0,1.0])
plt.title('ROC curve for Cancer classifer')
plt.xlabel('False positive rate (1-specificity)')
plt.ylabel('True positive rate (sensitivity)')
plt.legend(loc=4,)
