Skip to content

Instantly share code, notes, and snippets.

@jaimeide

jaimeide/kaggle_fraud_lightgbm_catboost

Last active April 29, 2018 02:00
Show Gist options
  • Save jaimeide/f32e748e97b3c7a76f0a8fb8a8c0314a to your computer and use it in GitHub Desktop.
Save jaimeide/f32e748e97b3c7a76f0a8fb8a8c0314a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
kaggle_fraud_lightgbm_catboost
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Case Study: Fraud Detection (LightGBM and CatBoost)\n",
"\n",
"Ref: https://www.kaggle.com/mlg-ulb/creditcardfraud"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Class to create Matlab-like structure\n",
"class matlab_like():\n",
" pass"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Context\n",
"\n",
"It is important that credit card companies are able to recognize fraudulent credit card transactions so that customers are not charged for items that they did not purchase.\n",
"\n",
"## Content\n",
"\n",
"The datasets contains transactions made by credit cards in September 2013 by european cardholders. This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions.\n",
"\n",
"It contains only numerical input variables which are the result of a PCA transformation. Unfortunately, due to confidentiality issues, we cannot provide the original features and more background information about the data. Features V1, V2, ... V28 are the principal components obtained with PCA, the only features which have not been transformed with PCA are 'Time' and 'Amount'. Feature 'Time' contains the seconds elapsed between each transaction and the first transaction in the dataset. The feature 'Amount' is the transaction Amount, this feature can be used for example-dependant cost-senstive learning. Feature 'Class' is the response variable and it takes value 1 in case of fraud and 0 otherwise.\n",
"\n",
"## Inspiration\n",
"\n",
"Identify fraudulent credit card transactions.\n",
"\n",
"Given the class imbalance ratio, we recommend measuring the accuracy using the Area Under the Precision-Recall Curve (AUPRC). Confusion matrix accuracy is not meaningful for unbalanced classification.\n",
"\n",
"## Acknowledgements\n",
"\n",
"The dataset has been collected and analysed during a research collaboration of Worldline and the Machine Learning Group (http://mlg.ulb.ac.be) of ULB (Université Libre de Bruxelles) on big data mining and fraud detection. More details on current and past projects on related topics are available on http://mlg.ulb.ac.be/BruFence and http://mlg.ulb.ac.be/ARTML\n",
"\n",
"Please cite: Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Library"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import time\n",
"import scipy.stats as stats\n",
"import numpy as np\n",
"import pandas as pd\n",
"import tensorflow as tf\n",
"import seaborn as sns\n",
"import matplotlib.gridspec as gridspec\n",
"\n",
"from sklearn.preprocessing import StandardScaler # to normalize feats\n",
"from sklearn.decomposition import PCA\n",
"from sklearn.metrics import classification_report\n",
"from imblearn.over_sampling import SMOTE\n",
"from imblearn.over_sampling import ADASYN"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Define my metrics\n",
"def ji_get_metrics(C):\n",
" \"\"\"\n",
" C: ndarray, shape (2,2) as given by scikit-learn confusion_matrix function\n",
" \n",
" Good reading: http://stats.stackexchange.com/questions/49579/balanced-accuracy-vs-f-1-score\n",
" \n",
" \"\"\"\n",
" #import matplotlib.pyplot as plt\n",
" import numpy as np\n",
" \n",
" allres = matlab_like()\n",
" \n",
" assert C.shape == (2,2), \"Confusion matrix should be from binary classification only.\"\n",
" \n",
" # true negative, false positive, etc...\n",
" tn = C[0,0]; fp = C[0,1]; fn = C[1,0]; tp = C[1,1]\n",
" NP = fn+tp # Num positive examples\n",
" NN = tn+fp # Num negative examples\n",
" N = NP+NN\n",
" TPR = tp / (NP+0.) # Sensitivity or Recall_P\n",
" TNR = tn / (NN+0.) # Specificity or Recall_N\n",
" \n",
" print(' TN=%d, FP=%d'%(tn,fp))\n",
" print(' FN=%d, TP=%d'%(fn,tp))\n",
" \n",
" \n",
" precision = (tp/(tp+fp+0.)) # Precision_P\n",
" recall = TPR\n",
" \n",
" allres.C0 = TNR\n",
" allres.C1 = TPR\n",
" allres.Precision = precision\n",
" allres.Recall = recall\n",
" allres.BAcc = (TPR+TNR)/2 \n",
" allres.F1 = 2*((precision*recall)/(precision+recall))\n",
" \n",
" return allres"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Function to plot Confusion Matrix\n",
"import itertools\n",
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_confusion_matrix(cm, classes,\n",
" normalize=False,\n",
" title='Confusion matrix',\n",
" cmap=plt.cm.Blues):\n",
" \"\"\"\n",
" This function prints and plots the confusion matrix.\n",
" Normalization can be applied by setting `normalize=True`.\n",
" \"\"\"\n",
" plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
" plt.title(title)\n",
" plt.colorbar()\n",
" tick_marks = np.arange(len(classes))\n",
" plt.xticks(tick_marks, classes, rotation=0)\n",
" plt.yticks(tick_marks, classes)\n",
"\n",
" if normalize:\n",
" cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
" #print(\"Normalized confusion matrix\")\n",
" else:\n",
" 1#print('Confusion matrix, without normalization')\n",
"\n",
" #print(cm)\n",
"\n",
" thresh = cm.max() / 2.\n",
" for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
" plt.text(j, i, cm[i, j],\n",
" horizontalalignment=\"center\",\n",
" color=\"white\" if cm[i, j] > thresh else \"black\")\n",
"\n",
" plt.tight_layout()\n",
" plt.ylabel('True label')\n",
" plt.xlabel('Predicted label')"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Standardize or normalize\n",
"def ji_normalize(df,typeN,selectedCol):\n",
" # Return a normalized DF\n",
" # Use: dd = ji_normalize(df,'minmax',['colName'])\n",
"\n",
" dfx = df.copy()\n",
" if selectedCol:\n",
" if typeN == 'minmax':\n",
" dfx[selectedCol] = dfx[selectedCol].apply(lambda x: (x - x.min()) / (x.max() - x.min()))\n",
" else:\n",
" dfx[selectedCol] = StandardScaler().fit_transform(dfx[selectedCol].values.reshape(-1, 1))\n",
" else:\n",
" allcols = dfx.columns\n",
" if typeN == 'minmax':\n",
" dfx[allcols] = dfx[allcols].apply(lambda x: (x - x.min()) / (x.max() - x.min()))\n",
" else:\n",
" dfx[allcols] = StandardScaler().fit_transform(dfx[allcols].values)\n",
" \n",
" return dfx\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load the data set"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Time</th>\n",
" <th>V1</th>\n",
" <th>V2</th>\n",
" <th>V3</th>\n",
" <th>V4</th>\n",
" <th>V5</th>\n",
" <th>V6</th>\n",
" <th>V7</th>\n",
" <th>V8</th>\n",
" <th>V9</th>\n",
" <th>...</th>\n",
" <th>V21</th>\n",
" <th>V22</th>\n",
" <th>V23</th>\n",
" <th>V24</th>\n",
" <th>V25</th>\n",
" <th>V26</th>\n",
" <th>V27</th>\n",
" <th>V28</th>\n",
" <th>Amount</th>\n",
" <th>Class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0.0</td>\n",
" <td>-1.359807</td>\n",
" <td>-0.072781</td>\n",
" <td>2.536347</td>\n",
" <td>1.378155</td>\n",
" <td>-0.338321</td>\n",
" <td>0.462388</td>\n",
" <td>0.239599</td>\n",
" <td>0.098698</td>\n",
" <td>0.363787</td>\n",
" <td>...</td>\n",
" <td>-0.018307</td>\n",
" <td>0.277838</td>\n",
" <td>-0.110474</td>\n",
" <td>0.066928</td>\n",
" <td>0.128539</td>\n",
" <td>-0.189115</td>\n",
" <td>0.133558</td>\n",
" <td>-0.021053</td>\n",
" <td>149.62</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>0.0</td>\n",
" <td>1.191857</td>\n",
" <td>0.266151</td>\n",
" <td>0.166480</td>\n",
" <td>0.448154</td>\n",
" <td>0.060018</td>\n",
" <td>-0.082361</td>\n",
" <td>-0.078803</td>\n",
" <td>0.085102</td>\n",
" <td>-0.255425</td>\n",
" <td>...</td>\n",
" <td>-0.225775</td>\n",
" <td>-0.638672</td>\n",
" <td>0.101288</td>\n",
" <td>-0.339846</td>\n",
" <td>0.167170</td>\n",
" <td>0.125895</td>\n",
" <td>-0.008983</td>\n",
" <td>0.014724</td>\n",
" <td>2.69</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>1.0</td>\n",
" <td>-1.358354</td>\n",
" <td>-1.340163</td>\n",
" <td>1.773209</td>\n",
" <td>0.379780</td>\n",
" <td>-0.503198</td>\n",
" <td>1.800499</td>\n",
" <td>0.791461</td>\n",
" <td>0.247676</td>\n",
" <td>-1.514654</td>\n",
" <td>...</td>\n",
" <td>0.247998</td>\n",
" <td>0.771679</td>\n",
" <td>0.909412</td>\n",
" <td>-0.689281</td>\n",
" <td>-0.327642</td>\n",
" <td>-0.139097</td>\n",
" <td>-0.055353</td>\n",
" <td>-0.059752</td>\n",
" <td>378.66</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1.0</td>\n",
" <td>-0.966272</td>\n",
" <td>-0.185226</td>\n",
" <td>1.792993</td>\n",
" <td>-0.863291</td>\n",
" <td>-0.010309</td>\n",
" <td>1.247203</td>\n",
" <td>0.237609</td>\n",
" <td>0.377436</td>\n",
" <td>-1.387024</td>\n",
" <td>...</td>\n",
" <td>-0.108300</td>\n",
" <td>0.005274</td>\n",
" <td>-0.190321</td>\n",
" <td>-1.175575</td>\n",
" <td>0.647376</td>\n",
" <td>-0.221929</td>\n",
" <td>0.062723</td>\n",
" <td>0.061458</td>\n",
" <td>123.50</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2.0</td>\n",
" <td>-1.158233</td>\n",
" <td>0.877737</td>\n",
" <td>1.548718</td>\n",
" <td>0.403034</td>\n",
" <td>-0.407193</td>\n",
" <td>0.095921</td>\n",
" <td>0.592941</td>\n",
" <td>-0.270533</td>\n",
" <td>0.817739</td>\n",
" <td>...</td>\n",
" <td>-0.009431</td>\n",
" <td>0.798278</td>\n",
" <td>-0.137458</td>\n",
" <td>0.141267</td>\n",
" <td>-0.206010</td>\n",
" <td>0.502292</td>\n",
" <td>0.219422</td>\n",
" <td>0.215153</td>\n",
" <td>69.99</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 31 columns</p>\n",
"</div>"
],
"text/plain": [
" Time V1 V2 V3 V4 V5 V6 V7 \\\n",
"0 0.0 -1.359807 -0.072781 2.536347 1.378155 -0.338321 0.462388 0.239599 \n",
"1 0.0 1.191857 0.266151 0.166480 0.448154 0.060018 -0.082361 -0.078803 \n",
"2 1.0 -1.358354 -1.340163 1.773209 0.379780 -0.503198 1.800499 0.791461 \n",
"3 1.0 -0.966272 -0.185226 1.792993 -0.863291 -0.010309 1.247203 0.237609 \n",
"4 2.0 -1.158233 0.877737 1.548718 0.403034 -0.407193 0.095921 0.592941 \n",
"\n",
" V8 V9 ... V21 V22 V23 V24 \\\n",
"0 0.098698 0.363787 ... -0.018307 0.277838 -0.110474 0.066928 \n",
"1 0.085102 -0.255425 ... -0.225775 -0.638672 0.101288 -0.339846 \n",
"2 0.247676 -1.514654 ... 0.247998 0.771679 0.909412 -0.689281 \n",
"3 0.377436 -1.387024 ... -0.108300 0.005274 -0.190321 -1.175575 \n",
"4 -0.270533 0.817739 ... -0.009431 0.798278 -0.137458 0.141267 \n",
"\n",
" V25 V26 V27 V28 Amount Class \n",
"0 0.128539 -0.189115 0.133558 -0.021053 149.62 0 \n",
"1 0.167170 0.125895 -0.008983 0.014724 2.69 0 \n",
"2 -0.327642 -0.139097 -0.055353 -0.059752 378.66 0 \n",
"3 0.647376 -0.221929 0.062723 0.061458 123.50 0 \n",
"4 -0.206010 0.502292 0.219422 0.215153 69.99 0 \n",
"\n",
"[5 rows x 31 columns]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 1) Import data\n",
"df = pd.read_csv(\"creditcard.csv\")\n",
"df.head() # notice that the features are anonymized"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# If you want to alter/categorize or fix a particular categorical variable (not the case here...)\n",
"def val_update(val):\n",
" if val<0:\n",
" return 0\n",
" else:\n",
" return 1\n",
"#aux = df['V1'].apply(val_update)\n",
"#aux.head()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Check if there are missing values \n",
"# --> NO!\n",
"#df.isnull().sum()"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Time</th>\n",
" <th>V1</th>\n",
" <th>V2</th>\n",
" <th>V3</th>\n",
" <th>V4</th>\n",
" <th>V5</th>\n",
" <th>V6</th>\n",
" <th>V7</th>\n",
" <th>V8</th>\n",
" <th>V9</th>\n",
" <th>...</th>\n",
" <th>V21</th>\n",
" <th>V22</th>\n",
" <th>V23</th>\n",
" <th>V24</th>\n",
" <th>V25</th>\n",
" <th>V26</th>\n",
" <th>V27</th>\n",
" <th>V28</th>\n",
" <th>Amount</th>\n",
" <th>Class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>284807.000000</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>...</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>2.848070e+05</td>\n",
" <td>284807.000000</td>\n",
" <td>284807.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>94813.859575</td>\n",
" <td>3.919560e-15</td>\n",
" <td>5.688174e-16</td>\n",
" <td>-8.769071e-15</td>\n",
" <td>2.782312e-15</td>\n",
" <td>-1.552563e-15</td>\n",
" <td>2.010663e-15</td>\n",
" <td>-1.694249e-15</td>\n",
" <td>-1.927028e-16</td>\n",
" <td>-3.137024e-15</td>\n",
" <td>...</td>\n",
" <td>1.537294e-16</td>\n",
" <td>7.959909e-16</td>\n",
" <td>5.367590e-16</td>\n",
" <td>4.458112e-15</td>\n",
" <td>1.453003e-15</td>\n",
" <td>1.699104e-15</td>\n",
" <td>-3.660161e-16</td>\n",
" <td>-1.206049e-16</td>\n",
" <td>88.349619</td>\n",
" <td>0.001727</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>47488.145955</td>\n",
" <td>1.958696e+00</td>\n",
" <td>1.651309e+00</td>\n",
" <td>1.516255e+00</td>\n",
" <td>1.415869e+00</td>\n",
" <td>1.380247e+00</td>\n",
" <td>1.332271e+00</td>\n",
" <td>1.237094e+00</td>\n",
" <td>1.194353e+00</td>\n",
" <td>1.098632e+00</td>\n",
" <td>...</td>\n",
" <td>7.345240e-01</td>\n",
" <td>7.257016e-01</td>\n",
" <td>6.244603e-01</td>\n",
" <td>6.056471e-01</td>\n",
" <td>5.212781e-01</td>\n",
" <td>4.822270e-01</td>\n",
" <td>4.036325e-01</td>\n",
" <td>3.300833e-01</td>\n",
" <td>250.120109</td>\n",
" <td>0.041527</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>0.000000</td>\n",
" <td>-5.640751e+01</td>\n",
" <td>-7.271573e+01</td>\n",
" <td>-4.832559e+01</td>\n",
" <td>-5.683171e+00</td>\n",
" <td>-1.137433e+02</td>\n",
" <td>-2.616051e+01</td>\n",
" <td>-4.355724e+01</td>\n",
" <td>-7.321672e+01</td>\n",
" <td>-1.343407e+01</td>\n",
" <td>...</td>\n",
" <td>-3.483038e+01</td>\n",
" <td>-1.093314e+01</td>\n",
" <td>-4.480774e+01</td>\n",
" <td>-2.836627e+00</td>\n",
" <td>-1.029540e+01</td>\n",
" <td>-2.604551e+00</td>\n",
" <td>-2.256568e+01</td>\n",
" <td>-1.543008e+01</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>54201.500000</td>\n",
" <td>-9.203734e-01</td>\n",
" <td>-5.985499e-01</td>\n",
" <td>-8.903648e-01</td>\n",
" <td>-8.486401e-01</td>\n",
" <td>-6.915971e-01</td>\n",
" <td>-7.682956e-01</td>\n",
" <td>-5.540759e-01</td>\n",
" <td>-2.086297e-01</td>\n",
" <td>-6.430976e-01</td>\n",
" <td>...</td>\n",
" <td>-2.283949e-01</td>\n",
" <td>-5.423504e-01</td>\n",
" <td>-1.618463e-01</td>\n",
" <td>-3.545861e-01</td>\n",
" <td>-3.171451e-01</td>\n",
" <td>-3.269839e-01</td>\n",
" <td>-7.083953e-02</td>\n",
" <td>-5.295979e-02</td>\n",
" <td>5.600000</td>\n",
" <td>0.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>84692.000000</td>\n",
" <td>1.810880e-02</td>\n",
" <td>6.548556e-02</td>\n",
" <td>1.798463e-01</td>\n",
" <td>-1.984653e-02</td>\n",
" <td>-5.433583e-02</td>\n",
" <td>-2.741871e-01</td>\n",
" <td>4.010308e-02</td>\n",
" <td>2.235804e-02</td>\n",
" <td>-5.142873e-02</td>\n",
" <td>...</td>\n",
" <td>-2.945017e-02</td>\n",
" <td>6.781943e-03</td>\n",
" <td>-1.119293e-02</td>\n",
" <td>4.097606e-02</td>\n",
" <td>1.659350e-02</td>\n",
" <td>-5.213911e-02</td>\n",
" <td>1.342146e-03</td>\n",
" <td>1.124383e-02</td>\n",
" <td>22.000000</td>\n",
" <td>0.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>139320.500000</td>\n",
" <td>1.315642e+00</td>\n",
" <td>8.037239e-01</td>\n",
" <td>1.027196e+00</td>\n",
" <td>7.433413e-01</td>\n",
" <td>6.119264e-01</td>\n",
" <td>3.985649e-01</td>\n",
" <td>5.704361e-01</td>\n",
" <td>3.273459e-01</td>\n",
" <td>5.971390e-01</td>\n",
" <td>...</td>\n",
" <td>1.863772e-01</td>\n",
" <td>5.285536e-01</td>\n",
" <td>1.476421e-01</td>\n",
" <td>4.395266e-01</td>\n",
" <td>3.507156e-01</td>\n",
" <td>2.409522e-01</td>\n",
" <td>9.104512e-02</td>\n",
" <td>7.827995e-02</td>\n",
" <td>77.165000</td>\n",
" <td>0.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>172792.000000</td>\n",
" <td>2.454930e+00</td>\n",
" <td>2.205773e+01</td>\n",
" <td>9.382558e+00</td>\n",
" <td>1.687534e+01</td>\n",
" <td>3.480167e+01</td>\n",
" <td>7.330163e+01</td>\n",
" <td>1.205895e+02</td>\n",
" <td>2.000721e+01</td>\n",
" <td>1.559499e+01</td>\n",
" <td>...</td>\n",
" <td>2.720284e+01</td>\n",
" <td>1.050309e+01</td>\n",
" <td>2.252841e+01</td>\n",
" <td>4.584549e+00</td>\n",
" <td>7.519589e+00</td>\n",
" <td>3.517346e+00</td>\n",
" <td>3.161220e+01</td>\n",
" <td>3.384781e+01</td>\n",
" <td>25691.160000</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>8 rows × 31 columns</p>\n",
"</div>"
],
"text/plain": [
" Time V1 V2 V3 V4 \\\n",
"count 284807.000000 2.848070e+05 2.848070e+05 2.848070e+05 2.848070e+05 \n",
"mean 94813.859575 3.919560e-15 5.688174e-16 -8.769071e-15 2.782312e-15 \n",
"std 47488.145955 1.958696e+00 1.651309e+00 1.516255e+00 1.415869e+00 \n",
"min 0.000000 -5.640751e+01 -7.271573e+01 -4.832559e+01 -5.683171e+00 \n",
"25% 54201.500000 -9.203734e-01 -5.985499e-01 -8.903648e-01 -8.486401e-01 \n",
"50% 84692.000000 1.810880e-02 6.548556e-02 1.798463e-01 -1.984653e-02 \n",
"75% 139320.500000 1.315642e+00 8.037239e-01 1.027196e+00 7.433413e-01 \n",
"max 172792.000000 2.454930e+00 2.205773e+01 9.382558e+00 1.687534e+01 \n",
"\n",
" V5 V6 V7 V8 V9 \\\n",
"count 2.848070e+05 2.848070e+05 2.848070e+05 2.848070e+05 2.848070e+05 \n",
"mean -1.552563e-15 2.010663e-15 -1.694249e-15 -1.927028e-16 -3.137024e-15 \n",
"std 1.380247e+00 1.332271e+00 1.237094e+00 1.194353e+00 1.098632e+00 \n",
"min -1.137433e+02 -2.616051e+01 -4.355724e+01 -7.321672e+01 -1.343407e+01 \n",
"25% -6.915971e-01 -7.682956e-01 -5.540759e-01 -2.086297e-01 -6.430976e-01 \n",
"50% -5.433583e-02 -2.741871e-01 4.010308e-02 2.235804e-02 -5.142873e-02 \n",
"75% 6.119264e-01 3.985649e-01 5.704361e-01 3.273459e-01 5.971390e-01 \n",
"max 3.480167e+01 7.330163e+01 1.205895e+02 2.000721e+01 1.559499e+01 \n",
"\n",
" ... V21 V22 V23 V24 \\\n",
"count ... 2.848070e+05 2.848070e+05 2.848070e+05 2.848070e+05 \n",
"mean ... 1.537294e-16 7.959909e-16 5.367590e-16 4.458112e-15 \n",
"std ... 7.345240e-01 7.257016e-01 6.244603e-01 6.056471e-01 \n",
"min ... -3.483038e+01 -1.093314e+01 -4.480774e+01 -2.836627e+00 \n",
"25% ... -2.283949e-01 -5.423504e-01 -1.618463e-01 -3.545861e-01 \n",
"50% ... -2.945017e-02 6.781943e-03 -1.119293e-02 4.097606e-02 \n",
"75% ... 1.863772e-01 5.285536e-01 1.476421e-01 4.395266e-01 \n",
"max ... 2.720284e+01 1.050309e+01 2.252841e+01 4.584549e+00 \n",
"\n",
" V25 V26 V27 V28 Amount \\\n",
"count 2.848070e+05 2.848070e+05 2.848070e+05 2.848070e+05 284807.000000 \n",
"mean 1.453003e-15 1.699104e-15 -3.660161e-16 -1.206049e-16 88.349619 \n",
"std 5.212781e-01 4.822270e-01 4.036325e-01 3.300833e-01 250.120109 \n",
"min -1.029540e+01 -2.604551e+00 -2.256568e+01 -1.543008e+01 0.000000 \n",
"25% -3.171451e-01 -3.269839e-01 -7.083953e-02 -5.295979e-02 5.600000 \n",
"50% 1.659350e-02 -5.213911e-02 1.342146e-03 1.124383e-02 22.000000 \n",
"75% 3.507156e-01 2.409522e-01 9.104512e-02 7.827995e-02 77.165000 \n",
"max 7.519589e+00 3.517346e+00 3.161220e+01 3.384781e+01 25691.160000 \n",
"\n",
" Class \n",
"count 284807.000000 \n",
"mean 0.001727 \n",
"std 0.041527 \n",
"min 0.000000 \n",
"25% 0.000000 \n",
"50% 0.000000 \n",
"75% 0.000000 \n",
"max 1.000000 \n",
"\n",
"[8 rows x 31 columns]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Original data to keep:\n",
"df.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prepare training & validation & testing"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Train+Validation:(199364, 30), Test:(85443, 30)\n",
"Index,Train: [ 39873 39874 39875 ..., 199361 199362 199363]|Valid: [ 0 1 2 ..., 39870 39871 39872]\n",
"Index,Train: [ 0 1 2 ..., 199361 199362 199363]|Valid: [39873 39874 39875 ..., 79743 79744 79745]\n",
"Index,Train: [ 0 1 2 ..., 199361 199362 199363]|Valid: [ 79746 79747 79748 ..., 119616 119617 119618]\n",
"Index,Train: [ 0 1 2 ..., 199361 199362 199363]|Valid: [119619 119620 119621 ..., 159489 159490 159491]\n",
"Index,Train: [ 0 1 2 ..., 159489 159490 159491]|Valid: [159492 159493 159494 ..., 199361 199362 199363]\n",
"\n",
"** Train+Validation Data **\n",
"Percentage of normal transactions: 0.9983 (n=199016)\n",
"Percentage of fraud transactions: 0.0017 (n=348)\n",
"Total number of transactions in resampled data: 199364\n",
"\n",
"** Test Data **\n",
"Percentage of normal transactions: 0.9983 (n=85299)\n",
"Percentage of fraud transactions: 0.0017 (n=144)\n",
"Total number of transactions in resampled data: 85443\n"
]
}
],
"source": [
"# ORIGINAL DATA\n",
"# - df\n",
"# - x_data\n",
"# - y_labels\n",
"\n",
"# 2) Prepare the learning\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.model_selection import KFold\n",
"from sklearn.model_selection import StratifiedKFold\n",
"\n",
"x_data = df.drop('Class',axis=1)\n",
"y_labels = df['Class']\n",
"y_names = ['Normal','Fraud']\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(x_data,y_labels,test_size=0.3,random_state=101)\n",
"\n",
"print('Train+Validation:%s, Test:%s'%(X_train.shape,X_test.shape))\n",
"\n",
"# 2.a) Random Cross Validation (10-folds)\n",
"k_3folds = KFold(n_splits=3,random_state=101)\n",
"k_5folds = KFold(n_splits=5,random_state=101)\n",
"k_10folds = KFold(n_splits=10,random_state=101)\n",
"for train_indices, validation_indices in k_5folds.split(X_train):\n",
" print('Index,Train: %s|Valid: %s' % (train_indices, validation_indices))\n",
"\n",
"# 2.b) Stratified Cross Validation (10-folds) - distributed based on labels\n",
"sk_3folds = StratifiedKFold(n_splits=3, random_state=101)\n",
"sk_5folds = StratifiedKFold(n_splits=5, random_state=101)\n",
"sk_10folds = StratifiedKFold(n_splits=10, random_state=101)\n",
"\n",
"# Showing ratio\n",
"dfaux = y_train\n",
"print(\"\")\n",
"print(\"** Train+Validation Data **\")\n",
"print(\"Percentage of normal transactions: %1.4f (n=%d)\"%(len(dfaux[dfaux == 0])/float(len(dfaux)),len(dfaux[dfaux == 0])))\n",
"print(\"Percentage of fraud transactions: %1.4f (n=%d)\"%(len(dfaux[dfaux == 1])/float(len(dfaux)),len(dfaux[dfaux == 1])))\n",
"print(\"Total number of transactions in resampled data: %d\"%(len(dfaux)))\n",
"# Showing ratio\n",
"dfaux = y_test\n",
"print(\"\")\n",
"print(\"** Test Data **\")\n",
"print(\"Percentage of normal transactions: %1.4f (n=%d)\"%(len(dfaux[dfaux == 0])/float(len(dfaux)),len(dfaux[dfaux == 0])))\n",
"print(\"Percentage of fraud transactions: %1.4f (n=%d)\"%(len(dfaux[dfaux == 1])/float(len(dfaux)),len(dfaux[dfaux == 1])))\n",
"print(\"Total number of transactions in resampled data: %d\"%(len(dfaux)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Dealing with Imbalanced Data: Sampling Approach\n",
"\n",
"**OBS**: Here, we have to be careful to not use the test for the sampling purpose... otherwise, we will already use the class information. Thus, the correct thing to do is a Sampling on the training data solely, leaving the test data completely apart"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1) Undersampling: create dataset with proportional train/test"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Percentage of normal transactions: 0.50 (n=348)\n",
"Percentage of fraud transactions: 0.50 (n=348)\n",
"Total number of transactions in resampled data: 696\n"
]
}
],
"source": [
"# Set data to undersample. The example used the whole data, but maybe this is not correct...\n",
"#datax = pd.concat([x_data,y_labels],axis=1) # This would use the whole data including Test... (maybe it is OK)\n",
"datax0 = pd.concat([x_data,y_labels],axis=1) # ORIGINAL DATA\n",
"datax = pd.concat([X_train,y_train],axis=1) #\n",
"\n",
"# Number of data points in the minority class\n",
"number_records_fraud = len(datax[datax.Class == 1])\n",
"fraud_indices = np.array(datax[datax.Class == 1].index)\n",
"\n",
"# Picking the indices of the normal classes\n",
"normal_indices = datax[datax.Class == 0].index\n",
"\n",
"# Out of the indices we picked, randomly select \"x\" number (number_records_fraud)\n",
"random_normal_indices = np.random.choice(normal_indices, number_records_fraud, replace = False)\n",
"random_normal_indices = np.array(random_normal_indices)\n",
"\n",
"# Appending the 2 indices\n",
"under_sample_indices = np.concatenate([fraud_indices,random_normal_indices])\n",
"\n",
"# Under sample dataset\n",
"under_sampled_data = datax0.iloc[under_sample_indices,:]\n",
"under_sampled_data_shuffled = under_sampled_data.sample(frac=1) # shuffle data to use for train/test\n",
"\n",
"X_undersampled = under_sampled_data_shuffled.loc[:, under_sampled_data_shuffled.columns != 'Class'] # Do not use direct to train/test...\n",
"y_undersampled = under_sampled_data_shuffled.loc[:, under_sampled_data_shuffled.columns == 'Class']\n",
"# Convert to Series (JI)\n",
"y_undersampled = y_undersampled.iloc[:,0]\n",
"\n",
"# Showing ratio\n",
"aux0 = len(under_sampled_data[under_sampled_data.Class == 0])\n",
"aux1 = len(under_sampled_data[under_sampled_data.Class == 1])\n",
"print(\"Percentage of normal transactions: %1.2f (n=%d)\"%(aux0/float(len(under_sampled_data)),aux0))\n",
"print(\"Percentage of fraud transactions: %1.2f (n=%d)\"%(aux1/float(len(under_sampled_data)),aux1))\n",
"print(\"Total number of transactions in resampled data: %d\"%(len(under_sampled_data)))\n",
"\n",
"# # Split into Train and Validation data sets\n",
"# #from sklearn.cross_validation import train_test_split\n",
"# from sklearn.model_selection import train_test_split\n",
"\n",
"# # Undersampled dataset\n",
"# X_train_undersampled, X_test_undersampled, y_train_undersampled, y_test_undersampled = \\\n",
"# train_test_split(X_undersampled,y_undersampled,test_size = 0.3,random_state = 0)\n",
"# print(\"\")\n",
"# print(\"Number transactions train dataset: \", len(X_train_undersampled))\n",
"# print(\"Number transactions validation dataset: \", len(X_test_undersampled))\n",
"# print(\"Total number of transactions: \", len(X_train_undersampled)+len(X_test_undersampled))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2) SMOTE: Synthetic Minority Oversampling"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Applied SMOTE: y_train old_size=199364 (#fraud=348), new_size=398032 (#fraud=199016)\n",
"Applied SMOTE (borderline): y_train old_size=199364 (#fraud=348), new_size=398032 (#fraud=199016)\n"
]
}
],
"source": [
"# SMOTE (Ide, ICPR 2016)\n",
"# Check: - ji_Example_Nice_ML_Framework.ipynb\n",
"# - ji_example_SMOTE.ipynb\n",
"\n",
"from imblearn.over_sampling import SMOTE\n",
"from imblearn.over_sampling import ADASYN\n",
"\n",
"# SMOTE and others..\n",
"\n",
"# a) Standard SMOTE\n",
"sm = SMOTE(kind='regular',random_state=101)\n",
"X_train_SMOTE, y_train_SMOTE = sm.fit_sample(X_train, y_train)\n",
"print('Applied SMOTE: y_train old_size=%d (#fraud=%d), new_size=%d (#fraud=%d)'%(len(y_train),len(y_train[y_train==1]),len(y_train_SMOTE),len(y_train_SMOTE[y_train_SMOTE==1])))\n",
"X_train_SMOTE = pd.DataFrame(X_train_SMOTE,columns=X_train.columns)\n",
"y_train_SMOTE = pd.Series(y_train_SMOTE)\n",
"# Shuffle data\n",
"iprm = np.random.permutation(len(y_train_SMOTE))\n",
"X_train_SMOTE = X_train_SMOTE.iloc[iprm]\n",
"y_train_SMOTE = y_train_SMOTE.iloc[iprm]\n",
"\n",
"# b) Borderline SMOTE\n",
"sm = SMOTE(kind='borderline2',random_state=101) # \n",
"X_train_SMOTE2, y_train_SMOTE2 = sm.fit_sample(X_train, y_train)\n",
"print('Applied SMOTE (borderline): y_train old_size=%d (#fraud=%d), new_size=%d (#fraud=%d)'%(len(y_train),len(y_train[y_train==1]),len(y_train_SMOTE2),len(y_train_SMOTE2[y_train_SMOTE2==1])))\n",
"X_train_SMOTE2 = pd.DataFrame(X_train_SMOTE2,columns=X_train.columns)\n",
"y_train_SMOTE2 = pd.Series(y_train_SMOTE2)\n",
"# Shuffle data\n",
"iprm = np.random.permutation(len(y_train_SMOTE2))\n",
"X_train_SMOTE2 = X_train_SMOTE2.iloc[iprm]\n",
"y_train_SMOTE2 = y_train_SMOTE2.iloc[iprm]\n",
"\n",
"#sm = SMOTE(kind='svm')\n",
"# c) ADASYN\n",
"#sm = ADASYN()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Feature Selection\n",
"\n",
"Ref: "
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# 1) Check correlation among feats \n",
"# --> you can consider removing correlated feats, since it creates colinearity in regression-based approaches\n",
"\n",
"# Cross-correlation map of the features\n",
"f,ax = plt.subplots(figsize=(20, 20))\n",
"#sns.heatmap(X_train.corr(), annot=True, linewidths=.5, fmt= '.1f',ax=ax)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Index(['V1', 'V2', 'V3', 'V4', 'V5', 'V7', 'V9', 'V10', 'V11', 'V12', 'V14',\n",
" 'V16', 'V17', 'V18', 'V21'],\n",
" dtype='object')\n",
"Index(['Time', 'V1', 'V2', 'V3', 'V4', 'V5', 'V7', 'V8', 'V10', 'V11', 'V12',\n",
" 'V14', 'V16', 'V17', 'Amount'],\n",
" dtype='object')\n"
]
}
],
"source": [
"# 2) Univariate feature selection\n",
"# - In univariate feature selection, we will use SelectKBest that removes all but the k highest scoring features. \n",
"# - http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest\n",
"\n",
"from sklearn.feature_selection import SelectKBest,chi2,f_classif,mutual_info_classif\n",
"\n",
"# Some options: f_classif (ANOVA), chi2, mutual_info_classif\n",
"# find k-best scored 10 features\n",
"select_feature1 = SelectKBest(f_classif, k=15).fit(X_train, y_train)\n",
"print(X_train.columns[select_feature1.get_support()])\n",
"select_feature2 = SelectKBest(chi2, k=15).fit(X_train.abs(), y_train)\n",
"print(X_train.columns[select_feature2.get_support()])\n",
"#select_feature3 = SelectKBest(mutual_info_classif, k=10).fit(X_train, y_train)\n",
"#print(X_train.columns[select_feature3.get_support()])\n",
"\n",
"# Transform data\n",
"X_train_UniF = select_feature1.transform(X_train)\n",
"X_test_UniF = select_feature1.transform(X_test)\n",
"X_train_Chi2 = select_feature2.transform(X_train)\n",
"X_test_Chi2 = select_feature2.transform(X_test)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# 3) Recursive feature elimination (RFE) using Random Forest¶ (could be any other...)\n",
"# - http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html \n",
"# - Basically, it uses one of the classification methods (random forest in our example), \n",
"# assign weights to each of features. Whose absolute weights are the smallest are pruned \n",
"# from the current set features. That procedure is recursively repeated on the pruned set until \n",
"# the desired number of features\n",
"tic = time.time()\n",
"\n",
"from sklearn.feature_selection import RFE\n",
"from sklearn.ensemble import RandomForestClassifier # Random forest classifier\n",
"\n",
"# Here, we will apply RFE on the undersampled data so that it will not be affected by the data imbalance\n",
"\n",
"# Create the RFE object and rank each pixel\n",
"clf_rf_3 = RandomForestClassifier(n_estimators=10) \n",
"rfe = RFE(estimator=clf_rf_3, n_features_to_select=15, step=1)\n",
"# Normalize\n",
"XX_train = X_undersampled.copy()\n",
"yy_train = y_undersampled.copy()\n",
"cols_to_norm = ['Time','Amount']\n",
"#XX_train[cols_to_norm] = XX_train[cols_to_norm].apply(lambda x: (x - x.mean()) / x.std())\n",
"XX_train[cols_to_norm] = XX_train[cols_to_norm].apply(lambda x: (x - x.min()) / (x.max() - x.min()))\n",
"\n",
"rfe = rfe.fit(XX_train, yy_train)\n",
"print('Chosen best 15 features by Recursive Feature Elimination:',XX_train.columns[rfe.support_])\n",
"\n",
"X_train_RFE = rfe.transform(X_train)\n",
"X_test_RFE = rfe.transform(X_test)\n",
"\n",
"toc = time.time()\n",
"print(\"Computation time: %d seconds\"%(toc-tic))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Boosting methods\n",
"- XGBoost\n",
"- LightGBM [x]\n",
"- CatBoost [x]"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from sklearn.metrics import confusion_matrix\n",
"from sklearn.model_selection import GridSearchCV\n",
"\n",
"# Useful functions v1 (XGBoost)\n",
"def get_auc(m, Xtrain,ytrain, Xtest,ytest): \n",
" return (metrics.roc_auc_score(ytrain,m.predict_proba(Xtrain)[:,1]),\n",
" metrics.roc_auc_score(ytest,m.predict_proba(Xtest)[:,1]))\n",
"\n",
"def get_metrics(model,XX,yy):\n",
" cnf_matrix = confusion_matrix(yy,model.predict(XX))\n",
" # Get scores\n",
" res = ji_get_metrics(cnf_matrix)\n",
" print(\" Precision: %1.2f\"%(res.Precision))\n",
" print(\" Recall(TPR): %1.2f\"%(res.Recall))\n",
" print(\" BAcc=(TPR+TNR)/2: %1.2f\"%(res.BAcc))\n",
" return res\n",
"\n",
"# Useful functions v2 (LightGBM)\n",
"def get_auc2(m, Xtrain,ytrain, Xtest,ytest):\n",
" return (metrics.roc_auc_score(ytrain,m.predict(Xtrain)),\n",
" metrics.roc_auc_score(ytest,m.predict(Xtest)))\n",
"\n",
"def get_metrics2(model,XX,yy):\n",
" pred = model.predict(XX)\n",
" pred[pred>0.5] = 1\n",
" pred[pred<=0.5] = 0\n",
" cnf_matrix = confusion_matrix(yy,pred)\n",
" # Get scores\n",
" res = ji_get_metrics(cnf_matrix)\n",
" print(\" Precision: %1.2f\"%(res.Precision))\n",
" print(\" Recall(TPR): %1.2f\"%(res.Recall))\n",
" print(\" BAcc=(TPR+TNR)/2: %1.2f\"%(res.BAcc))\n",
" return res"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# LightGBM"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Light GBM\n",
"# install with conda: conda install -c conda-forge lightgbm \n",
"\n",
"import lightgbm as lgb\n",
"from sklearn import metrics\n",
"\n",
"# Wraper function: Tune + Train + Test\n",
"def run_LightGBM(XX_train,yy_train,XX_test,yy_test,explabel,flag):\n",
" tic = time.time()\n",
"\n",
" # 1) Parameter Tuning \n",
" model = lgb.LGBMClassifier(silent=False)\n",
" if flag.set2use==0:\n",
" param_dist = {\"max_depth\": [10,30,50], # 27 combinations\n",
" \"learning_rate\" : [0.01,0.05, 0.1],\n",
" \"num_leaves\": [300,900,1200],\n",
" \"n_estimators\": [200]}\n",
" else: \n",
" param_dist = {\"max_depth\": [10,30,50,70,90], # 125 combinations\n",
" \"learning_rate\" : [0.01,0.05, 0.1,0.15,0.2],\n",
" \"num_leaves\": [100,300,600,900,1200],\n",
" \"n_estimators\": [200]}\n",
"\n",
" if flag.pars2use:\n",
" best = flag.pars2use\n",
" print('Using parameters:', best)\n",
" else:\n",
" # Grid search\n",
" grid_search = GridSearchCV(model, n_jobs=-1, param_grid=param_dist, cv = 3, scoring=\"roc_auc\", verbose=1)\n",
" grid_search.fit(XX_train, yy_train)\n",
" toc = time.time()\n",
" print(\"Computation time (parameter tunning): %d seconds...\"%(toc-tic)) \n",
" best = grid_search.best_params_\n",
" print('Best parameters:', best)\n",
" \n",
" # 3) Run with best parameters\n",
" tic = time.time()\n",
" d_train = lgb.Dataset(XX_train, label=yy_train,free_raw_data=False)\n",
" model2 = lgb.train(best, d_train)\n",
"\n",
" # Evaluate\n",
" print('********** SUMMARY: LightGBM (%s) **************'%(explabel))\n",
" print('- Parameters used: ',best)\n",
" print('- AUC(Train|Test): ',get_auc2(model2,XX_train,yy_train,XX_test,yy_test))\n",
" print('- Metrics: Train data (n=%d) **'%len(yy_train))\n",
" res_train = get_metrics2(model2,XX_train,yy_train)\n",
" print('- Metrics: Test data (n=%d) **'%len(yy_test))\n",
" res_test = get_metrics2(model2,XX_test,yy_test)\n",
"\n",
" toc = time.time()\n",
" print(\"Computation time (Total): %d seconds\"%(toc-tic))\n",
" \n",
" return best"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Fitting 3 folds for each of 125 candidates, totalling 375 fits\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=-1)]: Done 18 tasks | elapsed: 22.0s\n",
"[Parallel(n_jobs=-1)]: Done 168 tasks | elapsed: 2.3min\n",
"[Parallel(n_jobs=-1)]: Done 375 out of 375 | elapsed: 4.2min finished\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Computation time (parameter tunning): 252 seconds...\n",
"Best parameters: {'learning_rate': 0.15, 'max_depth': 30, 'n_estimators': 200, 'num_leaves': 100}\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jaime/anaconda3/lib/python3.6/site-packages/lightgbm/engine.py:99: UserWarning: Found `n_estimators` in params. Will use it instead of argument\n",
" warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"********** SUMMARY: LightGBM (Exp1-Undersampled) **************\n",
"- Parameters used: {'learning_rate': 0.15, 'max_depth': 30, 'num_leaves': 100, 'verbose': 1}\n",
"- AUC(Train|Test): (1.0, 0.98343742794952649)\n",
"- Metrics: Train data (n=696) **\n",
" TN=348, FP=0\n",
" FN=0, TP=348\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=82003, FP=3296\n",
" FN=9, TP=135\n",
" Precision: 0.04\n",
" Recall(TPR): 0.94\n",
" BAcc=(TPR+TNR)/2: 0.95\n",
"Computation time (Total): 2 seconds\n"
]
}
],
"source": [
"## EXPERIMENT 1 - Train data: undersampled, Test data: all\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp1-Undersampled'\n",
"XX_train = X_undersampled.copy()\n",
"yy_train = y_undersampled.copy()\n",
"\n",
"XX_test = X_test.copy()\n",
"yy_test = y_test.copy()\n",
"# 2) Run wrapper\n",
"flag = matlab_like()\n",
"flag.set2use = 1 # 3*125\n",
"flag.pars2use = {} # search\n",
"#flag.pars2use = {'learning_rate': 0.1, 'max_depth': 10, 'n_estimators': 200, 'num_leaves': 300}\n",
"best1 = run_LightGBM(XX_train,yy_train,XX_test,yy_test,mylabel,flag)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Using parameters: {'learning_rate': 0.15, 'max_depth': 30, 'num_leaves': 100, 'verbose': 1}\n",
"********** SUMMARY: LightGBM (Exp3-SMOTE (best parameters:Exp1)) **************\n",
"- Parameters used: {'learning_rate': 0.15, 'max_depth': 30, 'num_leaves': 100, 'verbose': 1}\n",
"- AUC(Train|Test): (0.99999584327342994, 0.9282734280459195)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=198956, FP=60\n",
" FN=0, TP=199016\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85235, FP=64\n",
" FN=21, TP=123\n",
" Precision: 0.66\n",
" Recall(TPR): 0.85\n",
" BAcc=(TPR+TNR)/2: 0.93\n",
"Computation time (Total): 5 seconds\n",
"Using parameters: {'learning_rate': 0.15, 'max_depth': 30, 'num_leaves': 100, 'verbose': 1}\n",
"********** SUMMARY: LightGBM (Exp4-SMOTE2 (best parameters:Exp1)) **************\n",
"- Parameters used: {'learning_rate': 0.15, 'max_depth': 30, 'num_leaves': 100, 'verbose': 1}\n",
"- AUC(Train|Test): (0.9999992787074411, 0.92872457798775798)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=198997, FP=19\n",
" FN=36, TP=198980\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85268, FP=31\n",
" FN=25, TP=119\n",
" Precision: 0.79\n",
" Recall(TPR): 0.83\n",
" BAcc=(TPR+TNR)/2: 0.91\n",
"Computation time (Total): 6 seconds\n"
]
}
],
"source": [
"## EXPERIMENT 3 - Train data: SMOTE, Test data: all (best parameters from Exp1)\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp3-SMOTE (best parameters:Exp1)'\n",
"XX_train = X_train_SMOTE.copy()\n",
"yy_train = y_train_SMOTE.copy()\n",
"XX_test = X_test.copy()\n",
"yy_test = y_test.copy()\n",
"# 2) Run wrapper\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = best1 # will not run GridSearch if selected\n",
"tmp = run_LightGBM(XX_train,yy_train,XX_test,yy_test,mylabel,flag)\n",
"print('\\n\\n')\n",
"\n",
"## EXPERIMENT 4 - Train data: SMOTE2, Test data: all (best parameters from Exp1)\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp4-SMOTE2 (best parameters:Exp1)'\n",
"XX_train = X_train_SMOTE2.copy()\n",
"yy_train = y_train_SMOTE2.copy()\n",
"XX_test = X_test.copy()\n",
"yy_test = y_test.copy()\n",
"# 2) Run wrapper\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = best1 # will not run GridSearch if selected\n",
"tmp = run_LightGBM(XX_train,yy_train,XX_test,yy_test,mylabel,flag)\n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Fitting 3 folds for each of 125 candidates, totalling 375 fits\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=-1)]: Done 18 tasks | elapsed: 22.1s\n",
"[Parallel(n_jobs=-1)]: Done 168 tasks | elapsed: 2.3min\n",
"[Parallel(n_jobs=-1)]: Done 375 out of 375 | elapsed: 4.2min finished\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Computation time (parameter tunning): 254 seconds...\n",
"Best parameters: {'learning_rate': 0.2, 'max_depth': 10, 'n_estimators': 200, 'num_leaves': 100}\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jaime/anaconda3/lib/python3.6/site-packages/lightgbm/engine.py:99: UserWarning: Found `n_estimators` in params. Will use it instead of argument\n",
" warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"********** SUMMARY: LightGBM (Exp5-Undersampled) **************\n",
"- Parameters used: {'learning_rate': 0.2, 'max_depth': 10, 'num_leaves': 100, 'verbose': 1}\n",
"- AUC(Train|Test): (1.0, 0.9804275906582206)\n",
"- Metrics: Train data (n=696) **\n",
" TN=348, FP=0\n",
" FN=0, TP=348\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=81721, FP=3578\n",
" FN=10, TP=134\n",
" Precision: 0.04\n",
" Recall(TPR): 0.93\n",
" BAcc=(TPR+TNR)/2: 0.94\n",
"Computation time (Total): 1 seconds\n",
"Using parameters: {'learning_rate': 0.2, 'max_depth': 10, 'num_leaves': 100, 'verbose': 1}\n",
"********** SUMMARY: LightGBM (Exp5-SMOTE (best parameters from Undersampled)) **************\n",
"- Parameters used: {'learning_rate': 0.2, 'max_depth': 10, 'num_leaves': 100, 'verbose': 1}\n",
"- AUC(Train|Test): (0.99993273632868551, 0.93552561349553398)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=198935, FP=81\n",
" FN=0, TP=199016\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85209, FP=90\n",
" FN=21, TP=123\n",
" Precision: 0.58\n",
" Recall(TPR): 0.85\n",
" BAcc=(TPR+TNR)/2: 0.93\n",
"Computation time (Total): 4 seconds\n",
"\n",
"\n",
"\n",
"Fitting 3 folds for each of 125 candidates, totalling 375 fits\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=-1)]: Done 18 tasks | elapsed: 20.5s\n",
"[Parallel(n_jobs=-1)]: Done 168 tasks | elapsed: 2.1min\n",
"[Parallel(n_jobs=-1)]: Done 375 out of 375 | elapsed: 4.1min finished\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Computation time (parameter tunning): 245 seconds...\n",
"Best parameters: {'learning_rate': 0.2, 'max_depth': 10, 'n_estimators': 200, 'num_leaves': 100}\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jaime/anaconda3/lib/python3.6/site-packages/lightgbm/engine.py:99: UserWarning: Found `n_estimators` in params. Will use it instead of argument\n",
" warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"********** SUMMARY: LightGBM (Exp6-Undersampled) **************\n",
"- Parameters used: {'learning_rate': 0.2, 'max_depth': 10, 'num_leaves': 100, 'verbose': 1}\n",
"- AUC(Train|Test): (1.0, 0.9804275906582206)\n",
"- Metrics: Train data (n=696) **\n",
" TN=348, FP=0\n",
" FN=0, TP=348\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=81721, FP=3578\n",
" FN=10, TP=134\n",
" Precision: 0.04\n",
" Recall(TPR): 0.93\n",
" BAcc=(TPR+TNR)/2: 0.94\n",
"Computation time (Total): 1 seconds\n",
"Using parameters: {'learning_rate': 0.2, 'max_depth': 10, 'num_leaves': 100, 'verbose': 1}\n",
"********** SUMMARY: LightGBM (Exp6-SMOTE2 (best parameters from Undersampled)) **************\n",
"- Parameters used: {'learning_rate': 0.2, 'max_depth': 10, 'num_leaves': 100, 'verbose': 1}\n",
"- AUC(Train|Test): (0.99999945223575926, 0.94372332911288526)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=198985, FP=31\n",
" FN=40, TP=198976\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85252, FP=47\n",
" FN=26, TP=118\n",
" Precision: 0.72\n",
" Recall(TPR): 0.82\n",
" BAcc=(TPR+TNR)/2: 0.91\n",
"Computation time (Total): 4 seconds\n",
"\n",
"\n",
"\n",
"Selected feats: Index(['Time', 'V1', 'V2', 'V3', 'V4', 'V5', 'V7', 'V8', 'V10', 'V11', 'V12',\n",
" 'V14', 'V16', 'V17', 'Amount'],\n",
" dtype='object')\n",
"Fitting 3 folds for each of 125 candidates, totalling 375 fits\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=-1)]: Done 18 tasks | elapsed: 20.0s\n",
"[Parallel(n_jobs=-1)]: Done 168 tasks | elapsed: 2.2min\n",
"[Parallel(n_jobs=-1)]: Done 375 out of 375 | elapsed: 4.2min finished\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Computation time (parameter tunning): 250 seconds...\n",
"Best parameters: {'learning_rate': 0.2, 'max_depth': 10, 'n_estimators': 200, 'num_leaves': 100}\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jaime/anaconda3/lib/python3.6/site-packages/lightgbm/engine.py:99: UserWarning: Found `n_estimators` in params. Will use it instead of argument\n",
" warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"********** SUMMARY: LightGBM (Exp7-Undersampled) **************\n",
"- Parameters used: {'learning_rate': 0.2, 'max_depth': 10, 'num_leaves': 100, 'verbose': 1}\n",
"- AUC(Train|Test): (1.0, 0.98224611204247536)\n",
"- Metrics: Train data (n=696) **\n",
" TN=348, FP=0\n",
" FN=0, TP=348\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=81982, FP=3317\n",
" FN=11, TP=133\n",
" Precision: 0.04\n",
" Recall(TPR): 0.92\n",
" BAcc=(TPR+TNR)/2: 0.94\n",
"Computation time (Total): 1 seconds\n",
"Using parameters: {'learning_rate': 0.2, 'max_depth': 10, 'num_leaves': 100, 'verbose': 1}\n",
"********** SUMMARY: LightGBM (Exp7-SMOTE2 (best parameters from Undersampled)) **************\n",
"- Parameters used: {'learning_rate': 0.2, 'max_depth': 10, 'num_leaves': 100, 'verbose': 1}\n",
"- AUC(Train|Test): (0.99999879507268219, 0.94818252070168851)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=198991, FP=25\n",
" FN=38, TP=198978\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85266, FP=33\n",
" FN=25, TP=119\n",
" Precision: 0.78\n",
" Recall(TPR): 0.83\n",
" BAcc=(TPR+TNR)/2: 0.91\n",
"Computation time (Total): 4 seconds\n"
]
}
],
"source": [
"## EXPERIMENT 5 - Train data: SMOTE + Selected Feats, Test data: all (best parameters from Undersampled data)\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp5-SMOTE (best parameters from Undersampled)'\n",
"iselected = X_train.columns[select_feature1.get_support()] # Univariate ANOVA\n",
"XX_train = X_train_SMOTE[iselected].copy()\n",
"yy_train = y_train_SMOTE.copy()\n",
"XX_test = X_test[iselected].copy()\n",
"yy_test = y_test.copy()\n",
"# 2) Get best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 1 # smaller set\n",
"flag.pars2use = {} # will not run GridSearch if selected\n",
"tmp = run_LightGBM(X_undersampled[iselected],y_undersampled,XX_test,yy_test,'Exp5-Undersampled',flag)\n",
"# 3) Run on whole data with the best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = tmp # will not run GridSearch if selected\n",
"best5 = run_LightGBM(XX_train,yy_train,XX_test,yy_test,mylabel,flag)\n",
"print('\\n\\n')\n",
"\n",
"## EXPERIMENT 6 - Train data: SMOTE2 + Selected Feats, Test data: all (best parameters from Undersampled data)\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp6-SMOTE2 (best parameters from Undersampled)'\n",
"iselected = X_train.columns[select_feature1.get_support()] # Univariate ANOVA\n",
"XX_train = X_train_SMOTE2[iselected].copy()\n",
"yy_train = y_train_SMOTE2.copy()\n",
"XX_test = X_test[iselected].copy()\n",
"yy_test = y_test.copy()\n",
"# 2) Get best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 1 # smaller set\n",
"flag.pars2use = {} # will not run GridSearch if selected\n",
"tmp = run_LightGBM(X_undersampled[iselected],y_undersampled,XX_test,yy_test,'Exp6-Undersampled',flag)\n",
"# 3) Run on whole data with the best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = tmp # will not run GridSearch if selected\n",
"best6 = run_LightGBM(XX_train,yy_train,XX_test,yy_test,mylabel,flag)\n",
"print('\\n\\n')\n",
"\n",
"## EXPERIMENT 7 - Train data: SMOTE2 + Selected Feats (Chi2), Test data: all (best parameters from Undersampled data)\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp7-SMOTE2 (best parameters from Undersampled)'\n",
"iselected = X_train.columns[select_feature2.get_support()] # Chi2\n",
"print('Selected feats: ',iselected)\n",
"XX_train = X_train_SMOTE2[iselected].copy()\n",
"yy_train = y_train_SMOTE2.copy()\n",
"XX_test = X_test[iselected].copy() \n",
"yy_test = y_test.copy()\n",
"# 2) Get best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 1 # smaller set\n",
"flag.pars2use = {} # will not run GridSearch if selected\n",
"tmp = run_LightGBM(X_undersampled[iselected],y_undersampled,XX_test,yy_test,'Exp7-Undersampled',flag)\n",
"# 3) Run on whole data with the best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = tmp # will not run GridSearch if selected\n",
"best7 = run_LightGBM(XX_train,yy_train,XX_test,yy_test,mylabel,flag)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# CatBoost"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# CatBoost\n",
"# - Install: pip install catboost --no-cache-dir\n",
"import catboost as cb\n",
"from sklearn import metrics\n",
"\n",
"# Wraper function: Tune + Train + Test\n",
"def run_CatBoost(XX_train,yy_train,XX_test,yy_test,explabel,flag):\n",
" tic = time.time()\n",
" \n",
" # 1) Parameter Tuning \n",
" model = cb.CatBoostClassifier(verbose=300)\n",
" if flag.set2use==0: # 27 combinations\n",
"# param_dist = {'depth': [4],\n",
"# 'learning_rate' : [0.03],\n",
"# 'l2_leaf_reg': [1,4],\n",
"# 'iterations': [300]}\n",
" param_dist = {'depth': [4, 7, 10],\n",
" 'learning_rate' : [0.03, 0.1, 0.15],\n",
" 'l2_leaf_reg': [1,4,9],\n",
" 'iterations': [300]}\n",
" else: # 125 combinations\n",
" param_dist = {'depth': [2, 4, 6, 8, 10],\n",
" 'learning_rate' : [0.01, 0.03, 0.1, 0.15, 0.2],\n",
" 'l2_leaf_reg': [1,3,5,7,9],\n",
" 'iterations': [300]}\n",
" if flag.pars2use:\n",
" best = flag.pars2use\n",
" print('Using parameters:', best)\n",
" else:\n",
" # Grid search\n",
" grid_search = GridSearchCV(model, param_grid=param_dist, scoring=\"roc_auc\", cv = 3,verbose=1)\n",
" grid_search.fit(XX_train, yy_train)\n",
" toc = time.time()\n",
" print(\"Computation time (parameter tunning): %d seconds...\"%(toc-tic)) \n",
" best = grid_search.best_params_\n",
" print('Best parameters:', best)\n",
" \n",
" # 3) Run with best parameters\n",
" tic = time.time()\n",
"\n",
" model = cb.CatBoostClassifier(eval_metric=\"AUC\", depth=best['depth'], iterations= best['iterations'],\\\n",
" l2_leaf_reg= best['l2_leaf_reg'], learning_rate= best['learning_rate'],verbose=30)\n",
" model.fit(XX_train,yy_train)\n",
"\n",
" # Evaluate\n",
" print('********** SUMMARY: CatBoost (%s) **************'%(explabel))\n",
" print('- Parameters used: ',best)\n",
" print('- AUC(Train|Test): ',get_auc(model,XX_train,yy_train,XX_test,yy_test))\n",
" print('- Metrics: Train data (n=%d) **'%len(yy_train))\n",
" res_train = get_metrics(model,XX_train,yy_train)\n",
" print('- Metrics: Test data (n=%d) **'%len(yy_test))\n",
" res_test = get_metrics(model,XX_test,yy_test)\n",
"\n",
" toc = time.time()\n",
" print(\"Computation time (Total): %d seconds\"%(toc-tic))\n",
" \n",
" return best"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Fitting 3 folds for each of 125 candidates, totalling 375 fits\n",
"0:\tlearn: 0.6813019\ttotal: 7.77ms\tremaining: 2.32s\n",
"0:\tlearn: 0.6807556\ttotal: 2.85ms\tremaining: 851ms\n",
"0:\tlearn: 0.6814685\ttotal: 3.67ms\tremaining: 1.1s\n",
"0:\tlearn: 0.6530555\ttotal: 3.09ms\tremaining: 925ms\n",
"0:\tlearn: 0.6514411\ttotal: 2.79ms\tremaining: 834ms\n",
"0:\tlearn: 0.6525051\ttotal: 2.96ms\tremaining: 886ms\n",
"0:\tlearn: 0.5857945\ttotal: 3.24ms\tremaining: 969ms\n",
"0:\tlearn: 0.5785231\ttotal: 2.43ms\tremaining: 726ms\n",
"0:\tlearn: 0.5922107\ttotal: 5.37ms\tremaining: 1.6s\n",
"0:\tlearn: 0.4934768\ttotal: 5.51ms\tremaining: 1.65s\n",
"0:\tlearn: 0.5274067\ttotal: 3.17ms\tremaining: 947ms\n",
"0:\tlearn: 0.5483641\ttotal: 3.88ms\tremaining: 1.16s\n",
"0:\tlearn: 0.5146773\ttotal: 3.79ms\tremaining: 1.13s\n",
"0:\tlearn: 0.4861008\ttotal: 4.44ms\tremaining: 1.33s\n",
"0:\tlearn: 0.4919630\ttotal: 4.95ms\tremaining: 1.48s\n",
"0:\tlearn: 0.6833408\ttotal: 3.46ms\tremaining: 1.03s\n",
"0:\tlearn: 0.6816158\ttotal: 2.78ms\tremaining: 830ms\n",
"0:\tlearn: 0.6794890\ttotal: 3.08ms\tremaining: 922ms\n",
"0:\tlearn: 0.6533085\ttotal: 2.33ms\tremaining: 697ms\n",
"0:\tlearn: 0.6652302\ttotal: 2.73ms\tremaining: 817ms\n",
"0:\tlearn: 0.6547760\ttotal: 3.61ms\tremaining: 1.08s\n",
"0:\tlearn: 0.5833221\ttotal: 3.47ms\tremaining: 1.04s\n",
"0:\tlearn: 0.5838764\ttotal: 2.46ms\tremaining: 735ms\n",
"0:\tlearn: 0.5736553\ttotal: 4.03ms\tremaining: 1.21s\n",
"0:\tlearn: 0.4964053\ttotal: 2.44ms\tremaining: 731ms\n",
"0:\tlearn: 0.5105519\ttotal: 3.54ms\tremaining: 1.06s\n",
"0:\tlearn: 0.5242174\ttotal: 3.97ms\tremaining: 1.19s\n",
"0:\tlearn: 0.4308241\ttotal: 3.26ms\tremaining: 974ms\n",
"0:\tlearn: 0.4971967\ttotal: 4.92ms\tremaining: 1.47s\n",
"0:\tlearn: 0.4614730\ttotal: 1.75ms\tremaining: 525ms\n",
"0:\tlearn: 0.6854944\ttotal: 3.5ms\tremaining: 1.05s\n",
"0:\tlearn: 0.6778102\ttotal: 3.41ms\tremaining: 1.02s\n",
"0:\tlearn: 0.6825427\ttotal: 2.66ms\tremaining: 795ms\n",
"0:\tlearn: 0.6528072\ttotal: 2.2ms\tremaining: 657ms\n",
"0:\tlearn: 0.6639703\ttotal: 4.62ms\tremaining: 1.38s\n",
"0:\tlearn: 0.6539612\ttotal: 4.29ms\tremaining: 1.28s\n",
"0:\tlearn: 0.5833337\ttotal: 3.82ms\tremaining: 1.14s\n",
"0:\tlearn: 0.5928229\ttotal: 4.62ms\tremaining: 1.38s\n",
"0:\tlearn: 0.5681818\ttotal: 4.34ms\tremaining: 1.3s\n",
"0:\tlearn: 0.5587382\ttotal: 4.72ms\tremaining: 1.41s\n",
"0:\tlearn: 0.5255628\ttotal: 3.74ms\tremaining: 1.12s\n",
"0:\tlearn: 0.5320330\ttotal: 3.91ms\tremaining: 1.17s\n",
"0:\tlearn: 0.4991931\ttotal: 4.05ms\tremaining: 1.21s\n",
"0:\tlearn: 0.5301314\ttotal: 3.22ms\tremaining: 963ms\n",
"0:\tlearn: 0.4922353\ttotal: 3.68ms\tremaining: 1.1s\n",
"0:\tlearn: 0.6786844\ttotal: 3.85ms\tremaining: 1.15s\n",
"0:\tlearn: 0.6840565\ttotal: 3.4ms\tremaining: 1.01s\n",
"0:\tlearn: 0.6824330\ttotal: 4.27ms\tremaining: 1.28s\n",
"0:\tlearn: 0.6623803\ttotal: 2.71ms\tremaining: 810ms\n",
"0:\tlearn: 0.6489919\ttotal: 2.31ms\tremaining: 689ms\n",
"0:\tlearn: 0.6598859\ttotal: 4.07ms\tremaining: 1.22s\n",
"0:\tlearn: 0.5951843\ttotal: 4.95ms\tremaining: 1.48s\n",
"0:\tlearn: 0.5631466\ttotal: 5.02ms\tremaining: 1.5s\n",
"0:\tlearn: 0.5745403\ttotal: 2.95ms\tremaining: 881ms\n",
"0:\tlearn: 0.5400210\ttotal: 4.73ms\tremaining: 1.41s\n",
"0:\tlearn: 0.5081807\ttotal: 3.63ms\tremaining: 1.09s\n",
"0:\tlearn: 0.5339605\ttotal: 6.28ms\tremaining: 1.88s\n",
"0:\tlearn: 0.4830802\ttotal: 5.18ms\tremaining: 1.55s\n",
"0:\tlearn: 0.4768061\ttotal: 2.92ms\tremaining: 873ms\n",
"0:\tlearn: 0.5156923\ttotal: 3.5ms\tremaining: 1.04s\n",
"0:\tlearn: 0.6780669\ttotal: 5.86ms\tremaining: 1.75s\n",
"0:\tlearn: 0.6811313\ttotal: 2.56ms\tremaining: 767ms\n",
"0:\tlearn: 0.6806700\ttotal: 3.13ms\tremaining: 935ms\n",
"0:\tlearn: 0.6572111\ttotal: 3.67ms\tremaining: 1.1s\n",
"0:\tlearn: 0.6497997\ttotal: 1.97ms\tremaining: 588ms\n",
"0:\tlearn: 0.6574673\ttotal: 2.84ms\tremaining: 850ms\n",
"0:\tlearn: 0.5660853\ttotal: 3.08ms\tremaining: 920ms\n",
"0:\tlearn: 0.5888499\ttotal: 4.32ms\tremaining: 1.29s\n",
"0:\tlearn: 0.5804811\ttotal: 3.56ms\tremaining: 1.06s\n",
"0:\tlearn: 0.5622689\ttotal: 3.94ms\tremaining: 1.18s\n",
"0:\tlearn: 0.5437219\ttotal: 3.72ms\tremaining: 1.11s\n",
"0:\tlearn: 0.5503123\ttotal: 3.33ms\tremaining: 997ms\n",
"0:\tlearn: 0.5124816\ttotal: 2.7ms\tremaining: 807ms\n",
"0:\tlearn: 0.5554585\ttotal: 1.99ms\tremaining: 594ms\n",
"0:\tlearn: 0.4740115\ttotal: 5.83ms\tremaining: 1.74s\n",
"0:\tlearn: 0.6779435\ttotal: 4.61ms\tremaining: 1.38s\n",
"0:\tlearn: 0.6762567\ttotal: 6.79ms\tremaining: 2.03s\n",
"0:\tlearn: 0.6811856\ttotal: 8.24ms\tremaining: 2.46s\n",
"0:\tlearn: 0.6441574\ttotal: 2.73ms\tremaining: 816ms\n",
"0:\tlearn: 0.6453864\ttotal: 5.76ms\tremaining: 1.72s\n",
"0:\tlearn: 0.6444228\ttotal: 4.41ms\tremaining: 1.32s\n",
"0:\tlearn: 0.5426073\ttotal: 4.05ms\tremaining: 1.21s\n",
"0:\tlearn: 0.5787804\ttotal: 6.23ms\tremaining: 1.86s\n",
"0:\tlearn: 0.5827517\ttotal: 3.66ms\tremaining: 1.09s\n",
"0:\tlearn: 0.4734915\ttotal: 5.83ms\tremaining: 1.74s\n",
"0:\tlearn: 0.4846873\ttotal: 4.37ms\tremaining: 1.31s\n",
"0:\tlearn: 0.4806220\ttotal: 3.93ms\tremaining: 1.18s\n",
"0:\tlearn: 0.4628260\ttotal: 6.42ms\tremaining: 1.92s\n",
"0:\tlearn: 0.4884660\ttotal: 5.88ms\tremaining: 1.76s\n",
"0:\tlearn: 0.4564705\ttotal: 4.28ms\tremaining: 1.28s\n",
"0:\tlearn: 0.6788312\ttotal: 8.91ms\tremaining: 2.66s\n",
"0:\tlearn: 0.6790821\ttotal: 5.51ms\tremaining: 1.65s\n",
"0:\tlearn: 0.6794857\ttotal: 3.49ms\tremaining: 1.04s\n",
"0:\tlearn: 0.6581456\ttotal: 2.96ms\tremaining: 886ms\n",
"0:\tlearn: 0.6566574\ttotal: 3.84ms\tremaining: 1.15s\n",
"0:\tlearn: 0.6467796\ttotal: 5.54ms\tremaining: 1.66s\n",
"0:\tlearn: 0.5812314\ttotal: 3.33ms\tremaining: 996ms\n",
"0:\tlearn: 0.5770790\ttotal: 2.93ms\tremaining: 875ms\n",
"0:\tlearn: 0.5832409\ttotal: 5.51ms\tremaining: 1.65s\n",
"0:\tlearn: 0.4525078\ttotal: 5.29ms\tremaining: 1.58s\n",
"0:\tlearn: 0.5270898\ttotal: 5.58ms\tremaining: 1.67s\n",
"0:\tlearn: 0.5188263\ttotal: 4.64ms\tremaining: 1.39s\n",
"0:\tlearn: 0.4983376\ttotal: 4.31ms\tremaining: 1.29s\n",
"0:\tlearn: 0.4622215\ttotal: 3.98ms\tremaining: 1.19s\n",
"0:\tlearn: 0.4786458\ttotal: 3.65ms\tremaining: 1.09s\n",
"0:\tlearn: 0.6778867\ttotal: 3.7ms\tremaining: 1.11s\n",
"0:\tlearn: 0.6795281\ttotal: 4.71ms\tremaining: 1.41s\n",
"0:\tlearn: 0.6775620\ttotal: 3.52ms\tremaining: 1.05s\n",
"0:\tlearn: 0.6522421\ttotal: 3.76ms\tremaining: 1.12s\n",
"0:\tlearn: 0.6550402\ttotal: 4.83ms\tremaining: 1.44s\n",
"0:\tlearn: 0.6536056\ttotal: 3.49ms\tremaining: 1.04s\n",
"0:\tlearn: 0.5576692\ttotal: 3.6ms\tremaining: 1.08s\n",
"0:\tlearn: 0.5648121\ttotal: 3.58ms\tremaining: 1.07s\n",
"0:\tlearn: 0.5662229\ttotal: 5.38ms\tremaining: 1.61s\n",
"0:\tlearn: 0.5166958\ttotal: 6.62ms\tremaining: 1.98s\n",
"0:\tlearn: 0.5266873\ttotal: 4.32ms\tremaining: 1.29s\n",
"0:\tlearn: 0.5189711\ttotal: 5.05ms\tremaining: 1.51s\n",
"0:\tlearn: 0.4761727\ttotal: 5.84ms\tremaining: 1.75s\n",
"0:\tlearn: 0.4860604\ttotal: 5.39ms\tremaining: 1.61s\n",
"0:\tlearn: 0.4706394\ttotal: 8.31ms\tremaining: 2.48s\n",
"0:\tlearn: 0.6790016\ttotal: 4.58ms\tremaining: 1.37s\n",
"0:\tlearn: 0.6791076\ttotal: 3.01ms\tremaining: 899ms\n",
"0:\tlearn: 0.6785722\ttotal: 4.33ms\tremaining: 1.29s\n",
"0:\tlearn: 0.6575018\ttotal: 4.92ms\tremaining: 1.47s\n",
"0:\tlearn: 0.6574009\ttotal: 5.39ms\tremaining: 1.61s\n",
"0:\tlearn: 0.6549890\ttotal: 4.49ms\tremaining: 1.34s\n",
"0:\tlearn: 0.5698768\ttotal: 4.06ms\tremaining: 1.21s\n",
"0:\tlearn: 0.5730014\ttotal: 4.14ms\tremaining: 1.24s\n",
"0:\tlearn: 0.5618981\ttotal: 5.38ms\tremaining: 1.61s\n",
"0:\tlearn: 0.5278057\ttotal: 6.12ms\tremaining: 1.83s\n",
"0:\tlearn: 0.5266303\ttotal: 2.83ms\tremaining: 847ms\n",
"0:\tlearn: 0.5384740\ttotal: 3.21ms\tremaining: 959ms\n",
"0:\tlearn: 0.5088481\ttotal: 6.75ms\tremaining: 2.02s\n",
"0:\tlearn: 0.4802518\ttotal: 4.37ms\tremaining: 1.31s\n",
"0:\tlearn: 0.4772028\ttotal: 5.81ms\tremaining: 1.74s\n",
"0:\tlearn: 0.6768407\ttotal: 3.86ms\tremaining: 1.15s\n",
"0:\tlearn: 0.6823523\ttotal: 6.21ms\tremaining: 1.86s\n",
"0:\tlearn: 0.6802335\ttotal: 4.49ms\tremaining: 1.34s\n",
"0:\tlearn: 0.6515911\ttotal: 5.7ms\tremaining: 1.7s\n",
"0:\tlearn: 0.6590137\ttotal: 6.1ms\tremaining: 1.82s\n",
"0:\tlearn: 0.6576490\ttotal: 8.02ms\tremaining: 2.4s\n",
"0:\tlearn: 0.5779230\ttotal: 5.14ms\tremaining: 1.54s\n",
"0:\tlearn: 0.5807853\ttotal: 4.54ms\tremaining: 1.36s\n",
"0:\tlearn: 0.5897245\ttotal: 4.81ms\tremaining: 1.44s\n",
"0:\tlearn: 0.5151954\ttotal: 6.92ms\tremaining: 2.07s\n",
"0:\tlearn: 0.5463128\ttotal: 6.83ms\tremaining: 2.04s\n",
"0:\tlearn: 0.5305259\ttotal: 4.44ms\tremaining: 1.33s\n",
"0:\tlearn: 0.4541982\ttotal: 3.06ms\tremaining: 914ms\n",
"0:\tlearn: 0.4709993\ttotal: 5.5ms\tremaining: 1.64s\n",
"0:\tlearn: 0.5087564\ttotal: 2.85ms\tremaining: 852ms\n",
"0:\tlearn: 0.6772419\ttotal: 7.45ms\tremaining: 2.23s\n",
"0:\tlearn: 0.6796002\ttotal: 9.49ms\tremaining: 2.84s\n",
"0:\tlearn: 0.6770937\ttotal: 6.67ms\tremaining: 1.99s\n",
"0:\tlearn: 0.6531528\ttotal: 7.61ms\tremaining: 2.27s\n",
"0:\tlearn: 0.6530965\ttotal: 9.67ms\tremaining: 2.89s\n",
"0:\tlearn: 0.6507628\ttotal: 6.4ms\tremaining: 1.91s\n",
"0:\tlearn: 0.5402150\ttotal: 6.83ms\tremaining: 2.04s\n",
"0:\tlearn: 0.5549966\ttotal: 6.79ms\tremaining: 2.03s\n",
"0:\tlearn: 0.5307944\ttotal: 6.13ms\tremaining: 1.83s\n",
"0:\tlearn: 0.4952264\ttotal: 6.56ms\tremaining: 1.96s\n",
"0:\tlearn: 0.4730618\ttotal: 6.66ms\tremaining: 1.99s\n",
"0:\tlearn: 0.5061578\ttotal: 6.72ms\tremaining: 2.01s\n",
"0:\tlearn: 0.4422251\ttotal: 5.83ms\tremaining: 1.74s\n",
"0:\tlearn: 0.4295642\ttotal: 8.29ms\tremaining: 2.48s\n",
"0:\tlearn: 0.4292252\ttotal: 7.65ms\tremaining: 2.29s\n",
"0:\tlearn: 0.6776251\ttotal: 8.78ms\tremaining: 2.63s\n",
"0:\tlearn: 0.6800739\ttotal: 8.14ms\tremaining: 2.43s\n",
"0:\tlearn: 0.6786988\ttotal: 8.51ms\tremaining: 2.54s\n",
"0:\tlearn: 0.6496086\ttotal: 7.9ms\tremaining: 2.36s\n",
"0:\tlearn: 0.6545137\ttotal: 6.75ms\tremaining: 2.02s\n",
"0:\tlearn: 0.6527396\ttotal: 6.93ms\tremaining: 2.07s\n",
"0:\tlearn: 0.5682055\ttotal: 7.34ms\tremaining: 2.19s\n",
"0:\tlearn: 0.5463319\ttotal: 7.37ms\tremaining: 2.2s\n",
"0:\tlearn: 0.5633239\ttotal: 6.01ms\tremaining: 1.8s\n",
"0:\tlearn: 0.5133816\ttotal: 7.55ms\tremaining: 2.26s\n",
"0:\tlearn: 0.4906654\ttotal: 8.85ms\tremaining: 2.65s\n",
"0:\tlearn: 0.5218724\ttotal: 9.9ms\tremaining: 2.96s\n",
"0:\tlearn: 0.4570834\ttotal: 8.88ms\tremaining: 2.66s\n",
"0:\tlearn: 0.4367231\ttotal: 8.13ms\tremaining: 2.43s\n",
"0:\tlearn: 0.4507043\ttotal: 10.6ms\tremaining: 3.18s\n",
"0:\tlearn: 0.6787022\ttotal: 7.38ms\tremaining: 2.21s\n",
"0:\tlearn: 0.6796556\ttotal: 7.17ms\tremaining: 2.15s\n",
"0:\tlearn: 0.6802835\ttotal: 9.94ms\tremaining: 2.97s\n",
"0:\tlearn: 0.6483132\ttotal: 6.4ms\tremaining: 1.91s\n",
"0:\tlearn: 0.6571252\ttotal: 6.65ms\tremaining: 1.99s\n",
"0:\tlearn: 0.6582895\ttotal: 8.1ms\tremaining: 2.42s\n",
"0:\tlearn: 0.5663648\ttotal: 6.61ms\tremaining: 1.98s\n",
"0:\tlearn: 0.5616587\ttotal: 8.02ms\tremaining: 2.4s\n",
"0:\tlearn: 0.5631064\ttotal: 9.96ms\tremaining: 2.98s\n",
"0:\tlearn: 0.5022197\ttotal: 6.45ms\tremaining: 1.93s\n",
"0:\tlearn: 0.5274027\ttotal: 6.44ms\tremaining: 1.93s\n",
"0:\tlearn: 0.5029847\ttotal: 8.19ms\tremaining: 2.45s\n",
"0:\tlearn: 0.4619394\ttotal: 8.74ms\tremaining: 2.62s\n",
"0:\tlearn: 0.4947912\ttotal: 7.22ms\tremaining: 2.16s\n",
"0:\tlearn: 0.4786790\ttotal: 10ms\tremaining: 3s\n",
"0:\tlearn: 0.6801312\ttotal: 6.96ms\tremaining: 2.08s\n",
"0:\tlearn: 0.6797344\ttotal: 6.26ms\tremaining: 1.87s\n",
"0:\tlearn: 0.6825872\ttotal: 7.43ms\tremaining: 2.22s\n",
"0:\tlearn: 0.6565465\ttotal: 9.37ms\tremaining: 2.8s\n",
"0:\tlearn: 0.6562187\ttotal: 6.29ms\tremaining: 1.88s\n",
"0:\tlearn: 0.6576552\ttotal: 8.86ms\tremaining: 2.65s\n",
"0:\tlearn: 0.5796811\ttotal: 4.81ms\tremaining: 1.44s\n",
"0:\tlearn: 0.5708441\ttotal: 7.06ms\tremaining: 2.11s\n",
"0:\tlearn: 0.5779496\ttotal: 6.05ms\tremaining: 1.81s\n",
"0:\tlearn: 0.5189771\ttotal: 7.99ms\tremaining: 2.39s\n",
"0:\tlearn: 0.5237320\ttotal: 6.07ms\tremaining: 1.82s\n",
"0:\tlearn: 0.5304453\ttotal: 8.43ms\tremaining: 2.52s\n",
"0:\tlearn: 0.4578489\ttotal: 8.75ms\tremaining: 2.62s\n",
"0:\tlearn: 0.4788079\ttotal: 7.79ms\tremaining: 2.33s\n",
"0:\tlearn: 0.5026990\ttotal: 7.24ms\tremaining: 2.17s\n",
"0:\tlearn: 0.6781502\ttotal: 7.61ms\tremaining: 2.27s\n",
"0:\tlearn: 0.6813798\ttotal: 7.75ms\tremaining: 2.31s\n",
"0:\tlearn: 0.6807016\ttotal: 7.41ms\tremaining: 2.22s\n",
"0:\tlearn: 0.6553633\ttotal: 5.99ms\tremaining: 1.79s\n",
"0:\tlearn: 0.6577753\ttotal: 8.02ms\tremaining: 2.4s\n",
"0:\tlearn: 0.6520666\ttotal: 8.09ms\tremaining: 2.42s\n",
"0:\tlearn: 0.5786650\ttotal: 6.13ms\tremaining: 1.83s\n",
"0:\tlearn: 0.5723111\ttotal: 9.35ms\tremaining: 2.79s\n",
"0:\tlearn: 0.5807038\ttotal: 6ms\tremaining: 1.79s\n",
"0:\tlearn: 0.5278627\ttotal: 9.8ms\tremaining: 2.93s\n",
"0:\tlearn: 0.5300084\ttotal: 7.84ms\tremaining: 2.34s\n",
"0:\tlearn: 0.5229891\ttotal: 7.63ms\tremaining: 2.28s\n",
"0:\tlearn: 0.4881911\ttotal: 7.04ms\tremaining: 2.1s\n",
"0:\tlearn: 0.5012449\ttotal: 7.89ms\tremaining: 2.36s\n",
"0:\tlearn: 0.4790684\ttotal: 9.76ms\tremaining: 2.92s\n",
"0:\tlearn: 0.6780886\ttotal: 16.8ms\tremaining: 5.03s\n",
"0:\tlearn: 0.6768475\ttotal: 15.8ms\tremaining: 4.72s\n",
"0:\tlearn: 0.6767277\ttotal: 18.8ms\tremaining: 5.63s\n",
"0:\tlearn: 0.6454693\ttotal: 17.5ms\tremaining: 5.22s\n",
"0:\tlearn: 0.6457107\ttotal: 20.9ms\tremaining: 6.25s\n",
"0:\tlearn: 0.6521434\ttotal: 15.9ms\tremaining: 4.76s\n",
"0:\tlearn: 0.5576528\ttotal: 19.3ms\tremaining: 5.76s\n",
"0:\tlearn: 0.5361132\ttotal: 17.1ms\tremaining: 5.11s\n",
"0:\tlearn: 0.5504446\ttotal: 14.1ms\tremaining: 4.21s\n",
"0:\tlearn: 0.4409020\ttotal: 15.3ms\tremaining: 4.58s\n",
"0:\tlearn: 0.4863649\ttotal: 13.9ms\tremaining: 4.17s\n",
"0:\tlearn: 0.5110774\ttotal: 17.1ms\tremaining: 5.11s\n",
"0:\tlearn: 0.4375592\ttotal: 21.8ms\tremaining: 6.5s\n",
"0:\tlearn: 0.4269613\ttotal: 17.3ms\tremaining: 5.17s\n",
"0:\tlearn: 0.4379969\ttotal: 15ms\tremaining: 4.48s\n",
"0:\tlearn: 0.6770481\ttotal: 13.5ms\tremaining: 4.03s\n",
"0:\tlearn: 0.6795200\ttotal: 21.4ms\tremaining: 6.39s\n",
"0:\tlearn: 0.6820363\ttotal: 16.5ms\tremaining: 4.94s\n",
"0:\tlearn: 0.6484635\ttotal: 15ms\tremaining: 4.48s\n",
"0:\tlearn: 0.6537390\ttotal: 19.6ms\tremaining: 5.85s\n",
"0:\tlearn: 0.6556304\ttotal: 16.7ms\tremaining: 4.99s\n",
"0:\tlearn: 0.5519368\ttotal: 20.2ms\tremaining: 6.04s\n",
"0:\tlearn: 0.5909617\ttotal: 16.2ms\tremaining: 4.84s\n",
"0:\tlearn: 0.5580428\ttotal: 20.3ms\tremaining: 6.08s\n",
"0:\tlearn: 0.5358982\ttotal: 19.5ms\tremaining: 5.83s\n",
"0:\tlearn: 0.5075729\ttotal: 22.2ms\tremaining: 6.65s\n",
"0:\tlearn: 0.5087355\ttotal: 20.9ms\tremaining: 6.26s\n",
"0:\tlearn: 0.4403853\ttotal: 16.1ms\tremaining: 4.81s\n",
"0:\tlearn: 0.4367675\ttotal: 16.8ms\tremaining: 5.02s\n",
"0:\tlearn: 0.4735834\ttotal: 16.5ms\tremaining: 4.93s\n",
"0:\tlearn: 0.6809523\ttotal: 15.3ms\tremaining: 4.58s\n",
"0:\tlearn: 0.6819357\ttotal: 16.7ms\tremaining: 5.01s\n",
"0:\tlearn: 0.6812558\ttotal: 22ms\tremaining: 6.57s\n",
"0:\tlearn: 0.6559010\ttotal: 18.1ms\tremaining: 5.4s\n",
"0:\tlearn: 0.6558215\ttotal: 14.7ms\tremaining: 4.39s\n",
"0:\tlearn: 0.6579529\ttotal: 18.6ms\tremaining: 5.57s\n",
"0:\tlearn: 0.5707957\ttotal: 17.6ms\tremaining: 5.25s\n",
"0:\tlearn: 0.5642434\ttotal: 15.3ms\tremaining: 4.57s\n",
"0:\tlearn: 0.5734427\ttotal: 19.8ms\tremaining: 5.91s\n",
"0:\tlearn: 0.5016866\ttotal: 17.9ms\tremaining: 5.34s\n",
"0:\tlearn: 0.5153625\ttotal: 23.7ms\tremaining: 7.08s\n",
"0:\tlearn: 0.5303178\ttotal: 20.2ms\tremaining: 6.03s\n",
"0:\tlearn: 0.4503330\ttotal: 15.6ms\tremaining: 4.67s\n",
"0:\tlearn: 0.4966209\ttotal: 14.6ms\tremaining: 4.36s\n",
"0:\tlearn: 0.4792043\ttotal: 18ms\tremaining: 5.39s\n",
"0:\tlearn: 0.6813159\ttotal: 18ms\tremaining: 5.37s\n",
"0:\tlearn: 0.6823085\ttotal: 15.2ms\tremaining: 4.53s\n",
"0:\tlearn: 0.6819107\ttotal: 15.8ms\tremaining: 4.71s\n",
"0:\tlearn: 0.6534696\ttotal: 20ms\tremaining: 5.98s\n",
"0:\tlearn: 0.6608044\ttotal: 20.3ms\tremaining: 6.08s\n",
"0:\tlearn: 0.6591916\ttotal: 20.9ms\tremaining: 6.25s\n",
"0:\tlearn: 0.5900049\ttotal: 21.6ms\tremaining: 6.46s\n",
"0:\tlearn: 0.5774045\ttotal: 21.3ms\tremaining: 6.36s\n",
"0:\tlearn: 0.5790334\ttotal: 19.4ms\tremaining: 5.79s\n",
"0:\tlearn: 0.5149778\ttotal: 14.8ms\tremaining: 4.43s\n",
"0:\tlearn: 0.5268448\ttotal: 8.39ms\tremaining: 2.51s\n",
"0:\tlearn: 0.5336916\ttotal: 15.1ms\tremaining: 4.51s\n",
"0:\tlearn: 0.4867199\ttotal: 20.2ms\tremaining: 6.03s\n",
"0:\tlearn: 0.4897595\ttotal: 16.7ms\tremaining: 5s\n",
"0:\tlearn: 0.4542394\ttotal: 19.4ms\tremaining: 5.8s\n",
"0:\tlearn: 0.6825141\ttotal: 17.7ms\tremaining: 5.29s\n",
"0:\tlearn: 0.6815209\ttotal: 15.3ms\tremaining: 4.57s\n",
"0:\tlearn: 0.6814706\ttotal: 15.5ms\tremaining: 4.62s\n",
"0:\tlearn: 0.6637281\ttotal: 21.9ms\tremaining: 6.54s\n",
"0:\tlearn: 0.6582870\ttotal: 15.5ms\tremaining: 4.65s\n",
"0:\tlearn: 0.6600557\ttotal: 17.1ms\tremaining: 5.12s\n",
"0:\tlearn: 0.5903187\ttotal: 13.4ms\tremaining: 4s\n",
"0:\tlearn: 0.5881031\ttotal: 14.4ms\tremaining: 4.3s\n",
"0:\tlearn: 0.5887720\ttotal: 22.6ms\tremaining: 6.75s\n",
"0:\tlearn: 0.5292795\ttotal: 15.1ms\tremaining: 4.52s\n",
"0:\tlearn: 0.5537161\ttotal: 16.6ms\tremaining: 4.96s\n",
"0:\tlearn: 0.5267198\ttotal: 24.4ms\tremaining: 7.28s\n",
"0:\tlearn: 0.5055331\ttotal: 17.4ms\tremaining: 5.21s\n",
"0:\tlearn: 0.5035923\ttotal: 22.9ms\tremaining: 6.83s\n",
"0:\tlearn: 0.4838883\ttotal: 16ms\tremaining: 4.79s\n",
"0:\tlearn: 0.6766299\ttotal: 56.3ms\tremaining: 16.8s\n",
"0:\tlearn: 0.6787849\ttotal: 67.7ms\tremaining: 20.3s\n",
"0:\tlearn: 0.6768967\ttotal: 53.8ms\tremaining: 16.1s\n",
"0:\tlearn: 0.6396526\ttotal: 57.8ms\tremaining: 17.3s\n",
"0:\tlearn: 0.6455622\ttotal: 58.6ms\tremaining: 17.5s\n",
"0:\tlearn: 0.6460594\ttotal: 61.2ms\tremaining: 18.3s\n",
"0:\tlearn: 0.5601185\ttotal: 52.2ms\tremaining: 15.6s\n",
"0:\tlearn: 0.5447255\ttotal: 51.9ms\tremaining: 15.5s\n",
"0:\tlearn: 0.5606173\ttotal: 53.6ms\tremaining: 16s\n",
"0:\tlearn: 0.5014927\ttotal: 59.3ms\tremaining: 17.7s\n",
"0:\tlearn: 0.5019297\ttotal: 53.9ms\tremaining: 16.1s\n",
"0:\tlearn: 0.4684576\ttotal: 53.7ms\tremaining: 16.1s\n",
"0:\tlearn: 0.4242576\ttotal: 51.4ms\tremaining: 15.4s\n",
"0:\tlearn: 0.4288064\ttotal: 74ms\tremaining: 22.1s\n",
"0:\tlearn: 0.4199392\ttotal: 64.8ms\tremaining: 19.4s\n",
"0:\tlearn: 0.6788534\ttotal: 48.3ms\tremaining: 14.4s\n",
"0:\tlearn: 0.6812609\ttotal: 52.6ms\tremaining: 15.7s\n",
"0:\tlearn: 0.6796333\ttotal: 52.7ms\tremaining: 15.8s\n",
"0:\tlearn: 0.6541848\ttotal: 65.9ms\tremaining: 19.7s\n",
"0:\tlearn: 0.6536978\ttotal: 21.1ms\tremaining: 6.31s\n",
"0:\tlearn: 0.6590265\ttotal: 55.4ms\tremaining: 16.6s\n",
"0:\tlearn: 0.5627339\ttotal: 57.3ms\tremaining: 17.1s\n",
"0:\tlearn: 0.5790008\ttotal: 51.9ms\tremaining: 15.5s\n",
"0:\tlearn: 0.5654847\ttotal: 51.6ms\tremaining: 15.4s\n",
"0:\tlearn: 0.5096104\ttotal: 50.5ms\tremaining: 15.1s\n",
"0:\tlearn: 0.5296614\ttotal: 54ms\tremaining: 16.1s\n",
"0:\tlearn: 0.5191129\ttotal: 52.8ms\tremaining: 15.8s\n",
"0:\tlearn: 0.4542304\ttotal: 69.1ms\tremaining: 20.7s\n",
"0:\tlearn: 0.4689849\ttotal: 49ms\tremaining: 14.6s\n",
"0:\tlearn: 0.4805282\ttotal: 69.3ms\tremaining: 20.7s\n",
"0:\tlearn: 0.6805136\ttotal: 68.8ms\tremaining: 20.6s\n",
"0:\tlearn: 0.6812610\ttotal: 47.3ms\tremaining: 14.1s\n",
"0:\tlearn: 0.6812344\ttotal: 50.5ms\tremaining: 15.1s\n",
"0:\tlearn: 0.6556025\ttotal: 53.7ms\tremaining: 16s\n",
"0:\tlearn: 0.6584101\ttotal: 46ms\tremaining: 13.7s\n",
"0:\tlearn: 0.6553724\ttotal: 56.1ms\tremaining: 16.8s\n",
"0:\tlearn: 0.5764340\ttotal: 48.4ms\tremaining: 14.5s\n",
"0:\tlearn: 0.5875414\ttotal: 67.1ms\tremaining: 20.1s\n",
"0:\tlearn: 0.5792510\ttotal: 46ms\tremaining: 13.8s\n",
"0:\tlearn: 0.5501599\ttotal: 54.9ms\tremaining: 16.4s\n",
"0:\tlearn: 0.5205637\ttotal: 56.6ms\tremaining: 16.9s\n",
"0:\tlearn: 0.5253998\ttotal: 54.2ms\tremaining: 16.2s\n",
"0:\tlearn: 0.4478873\ttotal: 61.2ms\tremaining: 18.3s\n",
"0:\tlearn: 0.4922146\ttotal: 50.5ms\tremaining: 15.1s\n",
"0:\tlearn: 0.4881003\ttotal: 51.7ms\tremaining: 15.5s\n",
"0:\tlearn: 0.6809382\ttotal: 55.2ms\tremaining: 16.5s\n",
"0:\tlearn: 0.6838455\ttotal: 54.2ms\tremaining: 16.2s\n",
"0:\tlearn: 0.6815734\ttotal: 50.4ms\tremaining: 15.1s\n",
"0:\tlearn: 0.6615390\ttotal: 53.3ms\tremaining: 15.9s\n",
"0:\tlearn: 0.6617555\ttotal: 55.3ms\tremaining: 16.5s\n",
"0:\tlearn: 0.6622349\ttotal: 55.1ms\tremaining: 16.5s\n",
"0:\tlearn: 0.5763964\ttotal: 54.7ms\tremaining: 16.4s\n",
"0:\tlearn: 0.6034771\ttotal: 48.9ms\tremaining: 14.6s\n",
"0:\tlearn: 0.5971667\ttotal: 67.8ms\tremaining: 20.3s\n",
"0:\tlearn: 0.5701615\ttotal: 50.3ms\tremaining: 15s\n",
"0:\tlearn: 0.5359567\ttotal: 62.4ms\tremaining: 18.7s\n",
"0:\tlearn: 0.5503117\ttotal: 51.5ms\tremaining: 15.4s\n",
"0:\tlearn: 0.5064307\ttotal: 44.2ms\tremaining: 13.2s\n",
"0:\tlearn: 0.5145313\ttotal: 68.1ms\tremaining: 20.4s\n",
"0:\tlearn: 0.4689965\ttotal: 58.8ms\tremaining: 17.6s\n",
"0:\tlearn: 0.6830417\ttotal: 51.3ms\tremaining: 15.3s\n",
"0:\tlearn: 0.6830601\ttotal: 53.5ms\tremaining: 16s\n",
"0:\tlearn: 0.6830268\ttotal: 60.3ms\tremaining: 18s\n",
"0:\tlearn: 0.6602923\ttotal: 65.6ms\tremaining: 19.6s\n",
"0:\tlearn: 0.6569699\ttotal: 58.8ms\tremaining: 17.6s\n",
"0:\tlearn: 0.6650876\ttotal: 53.5ms\tremaining: 16s\n",
"0:\tlearn: 0.5929283\ttotal: 50.8ms\tremaining: 15.2s\n",
"0:\tlearn: 0.6190265\ttotal: 61.2ms\tremaining: 18.3s\n",
"0:\tlearn: 0.6059873\ttotal: 59.5ms\tremaining: 17.8s\n",
"0:\tlearn: 0.5461550\ttotal: 68.4ms\tremaining: 20.4s\n",
"0:\tlearn: 0.5564542\ttotal: 57.4ms\tremaining: 17.1s\n",
"0:\tlearn: 0.5580309\ttotal: 53.3ms\tremaining: 15.9s\n",
"0:\tlearn: 0.5182608\ttotal: 52.3ms\tremaining: 15.6s\n",
"0:\tlearn: 0.5161463\ttotal: 53.9ms\tremaining: 16.1s\n",
"0:\tlearn: 0.5442647\ttotal: 49ms\tremaining: 14.7s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=1)]: Done 375 out of 375 | elapsed: 30.0min finished\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"0:\tlearn: 0.6803440\ttotal: 16.1ms\tremaining: 4.83s\n",
"Computation time (parameter tunning): 1802 seconds...\n",
"Best parameters: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"0:\tlearn: 0.9755665\ttotal: 12.6ms\tremaining: 3.77s\n",
"30:\tlearn: 0.9899921\ttotal: 376ms\tremaining: 3.26s\n",
"60:\tlearn: 0.9910986\ttotal: 713ms\tremaining: 2.79s\n",
"90:\tlearn: 0.9924197\ttotal: 1.05s\tremaining: 2.4s\n",
"120:\tlearn: 0.9929400\ttotal: 1.41s\tremaining: 2.08s\n",
"150:\tlearn: 0.9939474\ttotal: 1.75s\tremaining: 1.73s\n",
"180:\tlearn: 0.9946657\ttotal: 2.1s\tremaining: 1.38s\n",
"210:\tlearn: 0.9955328\ttotal: 2.47s\tremaining: 1.04s\n",
"240:\tlearn: 0.9964658\ttotal: 2.83s\tremaining: 692ms\n",
"270:\tlearn: 0.9971512\ttotal: 3.16s\tremaining: 339ms\n",
"********** SUMMARY: XGBoost (Exp1-Undersampled) **************\n",
"- Parameters used: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"- AUC(Train|Test): (0.99759710661910417, 0.98088350325847262)\n",
"- Metrics: Train data (n=696) **\n",
" TN=345, FP=3\n",
" FN=19, TP=329\n",
" Precision: 0.99\n",
" Recall(TPR): 0.95\n",
" BAcc=(TPR+TNR)/2: 0.97\n",
"- Metrics: Test data (n=85443) **\n",
" TN=83613, FP=1686\n",
" FN=16, TP=128\n",
" Precision: 0.07\n",
" Recall(TPR): 0.89\n",
" BAcc=(TPR+TNR)/2: 0.93\n",
"Computation time (Total): 4 seconds\n"
]
}
],
"source": [
"## EXPERIMENT 1 - Train data: undersampled, Test data: all\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp1-Undersampled'\n",
"XX_train = X_undersampled.copy()\n",
"yy_train = y_undersampled.copy()\n",
"\n",
"XX_test = X_test.copy()\n",
"yy_test = y_test.copy()\n",
"# 2) Run wrapper\n",
"flag = matlab_like()\n",
"flag.set2use = 1 # 3*125\n",
"flag.pars2use = {} # search\n",
"#flag.pars2use = {'depth': 4,'learning_rate' : 0.03,'l2_leaf_reg': 1,'iterations': 300}\n",
"best1 = run_CatBoost(XX_train,yy_train,XX_test,yy_test,mylabel,flag)"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Using parameters: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"0:\tlearn: 0.9956213\ttotal: 182ms\tremaining: 54.4s\n",
"30:\tlearn: 0.9979998\ttotal: 5.28s\tremaining: 45.9s\n",
"60:\tlearn: 0.9983776\ttotal: 10.7s\tremaining: 41.9s\n",
"90:\tlearn: 0.9988216\ttotal: 16.2s\tremaining: 37.2s\n",
"120:\tlearn: 0.9991379\ttotal: 21.5s\tremaining: 31.8s\n",
"150:\tlearn: 0.9993833\ttotal: 26.8s\tremaining: 26.5s\n",
"180:\tlearn: 0.9995442\ttotal: 32.2s\tremaining: 21.2s\n",
"210:\tlearn: 0.9996588\ttotal: 37.8s\tremaining: 15.9s\n",
"240:\tlearn: 0.9997403\ttotal: 43.2s\tremaining: 10.6s\n",
"270:\tlearn: 0.9997981\ttotal: 48.7s\tremaining: 5.21s\n",
"********** SUMMARY: CatBoost (Exp3-SMOTE (best parameters:Exp1)) **************\n",
"- Parameters used: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"- AUC(Train|Test): (0.99983772633015, 0.97994766123349097)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=198473, FP=543\n",
" FN=1393, TP=197623\n",
" Precision: 1.00\n",
" Recall(TPR): 0.99\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85054, FP=245\n",
" FN=21, TP=123\n",
" Precision: 0.33\n",
" Recall(TPR): 0.85\n",
" BAcc=(TPR+TNR)/2: 0.93\n",
"Computation time (Total): 61 seconds\n"
]
}
],
"source": [
"## EXPERIMENT 3 - Train data: SMOTE, Test data: all (best parameters from Exp1)\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp3-SMOTE (best parameters:Exp1)'\n",
"XX_train = X_train_SMOTE.copy()\n",
"yy_train = y_train_SMOTE.copy()\n",
"XX_test = X_test.copy()\n",
"yy_test = y_test.copy()\n",
"# 2) Run wrapper\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = best1 # will not run GridSearch if selected\n",
"tmp=run_CatBoost(XX_train,yy_train,XX_test,yy_test,mylabel,flag)"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Using parameters: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"0:\tlearn: 0.9990728\ttotal: 204ms\tremaining: 1m\n",
"30:\tlearn: 0.9998273\ttotal: 5.49s\tremaining: 47.7s\n",
"60:\tlearn: 0.9998476\ttotal: 10.8s\tremaining: 42.1s\n",
"90:\tlearn: 0.9998707\ttotal: 16s\tremaining: 36.7s\n",
"120:\tlearn: 0.9998869\ttotal: 21.3s\tremaining: 31.4s\n",
"150:\tlearn: 0.9998951\ttotal: 26.6s\tremaining: 26.2s\n",
"180:\tlearn: 0.9999142\ttotal: 31.9s\tremaining: 21s\n",
"210:\tlearn: 0.9999239\ttotal: 37.2s\tremaining: 15.7s\n",
"240:\tlearn: 0.9999280\ttotal: 42.6s\tremaining: 10.4s\n",
"270:\tlearn: 0.9999363\ttotal: 47.9s\tremaining: 5.12s\n",
"********** SUMMARY: CatBoost (Exp4-SMOTE2 (best parameters:Exp1)) **************\n",
"- Parameters used: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"- AUC(Train|Test): (0.99994226887317472, 0.98169242247206234)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=198912, FP=104\n",
" FN=158, TP=198858\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85247, FP=52\n",
" FN=22, TP=122\n",
" Precision: 0.70\n",
" Recall(TPR): 0.85\n",
" BAcc=(TPR+TNR)/2: 0.92\n",
"Computation time (Total): 61 seconds\n"
]
}
],
"source": [
"## EXPERIMENT 4 - Train data: SMOTE2, Test data: all (best parameters from Exp1)\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp4-SMOTE2 (best parameters:Exp1)'\n",
"XX_train = X_train_SMOTE2.copy()\n",
"yy_train = y_train_SMOTE2.copy()\n",
"XX_test = X_test.copy()\n",
"yy_test = y_test.copy()\n",
"# 2) Run wrapper\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = best1 # will not run GridSearch if selected\n",
"tmp = run_CatBoost(XX_train,yy_train,XX_test,yy_test,mylabel,flag)"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Fitting 3 folds for each of 125 candidates, totalling 375 fits\n",
"0:\tlearn: 0.6782282\ttotal: 5.55ms\tremaining: 1.66s\n",
"0:\tlearn: 0.6768638\ttotal: 4.5ms\tremaining: 1.35s\n",
"0:\tlearn: 0.6788042\ttotal: 3.4ms\tremaining: 1.02s\n",
"0:\tlearn: 0.6470119\ttotal: 1.67ms\tremaining: 499ms\n",
"0:\tlearn: 0.6529075\ttotal: 3.78ms\tremaining: 1.13s\n",
"0:\tlearn: 0.6460511\ttotal: 3.45ms\tremaining: 1.03s\n",
"0:\tlearn: 0.5767201\ttotal: 2.9ms\tremaining: 868ms\n",
"0:\tlearn: 0.5766367\ttotal: 3.67ms\tremaining: 1.1s\n",
"0:\tlearn: 0.5626467\ttotal: 3.67ms\tremaining: 1.1s\n",
"0:\tlearn: 0.5125459\ttotal: 4.34ms\tremaining: 1.3s\n",
"0:\tlearn: 0.5457981\ttotal: 2.09ms\tremaining: 626ms\n",
"0:\tlearn: 0.5497705\ttotal: 2.84ms\tremaining: 848ms\n",
"0:\tlearn: 0.4316385\ttotal: 3.91ms\tremaining: 1.17s\n",
"0:\tlearn: 0.4231215\ttotal: 3.06ms\tremaining: 914ms\n",
"0:\tlearn: 0.4679216\ttotal: 2.99ms\tremaining: 895ms\n",
"0:\tlearn: 0.6812262\ttotal: 3.57ms\tremaining: 1.07s\n",
"0:\tlearn: 0.6819004\ttotal: 5.51ms\tremaining: 1.65s\n",
"0:\tlearn: 0.6778015\ttotal: 4.93ms\tremaining: 1.48s\n",
"0:\tlearn: 0.6517278\ttotal: 3.97ms\tremaining: 1.19s\n",
"0:\tlearn: 0.6550727\ttotal: 3.78ms\tremaining: 1.13s\n",
"0:\tlearn: 0.6576069\ttotal: 1.7ms\tremaining: 509ms\n",
"0:\tlearn: 0.5623467\ttotal: 4.28ms\tremaining: 1.28s\n",
"0:\tlearn: 0.5549505\ttotal: 1.63ms\tremaining: 487ms\n",
"0:\tlearn: 0.5667544\ttotal: 4.08ms\tremaining: 1.22s\n",
"0:\tlearn: 0.5246947\ttotal: 2.13ms\tremaining: 636ms\n",
"0:\tlearn: 0.5036556\ttotal: 1.9ms\tremaining: 567ms\n",
"0:\tlearn: 0.5337054\ttotal: 4.58ms\tremaining: 1.37s\n",
"0:\tlearn: 0.5250580\ttotal: 2.58ms\tremaining: 772ms\n",
"0:\tlearn: 0.4635236\ttotal: 1.9ms\tremaining: 569ms\n",
"0:\tlearn: 0.4933276\ttotal: 4.22ms\tremaining: 1.26s\n",
"0:\tlearn: 0.6805902\ttotal: 4.01ms\tremaining: 1.2s\n",
"0:\tlearn: 0.6809312\ttotal: 3.32ms\tremaining: 992ms\n",
"0:\tlearn: 0.6822600\ttotal: 2.51ms\tremaining: 751ms\n",
"0:\tlearn: 0.6517583\ttotal: 1.93ms\tremaining: 577ms\n",
"0:\tlearn: 0.6663978\ttotal: 4.33ms\tremaining: 1.29s\n",
"0:\tlearn: 0.6540472\ttotal: 2ms\tremaining: 597ms\n",
"0:\tlearn: 0.5613944\ttotal: 2.93ms\tremaining: 876ms\n",
"0:\tlearn: 0.5628106\ttotal: 5.96ms\tremaining: 1.78s\n",
"0:\tlearn: 0.5540760\ttotal: 3.82ms\tremaining: 1.14s\n",
"0:\tlearn: 0.4966759\ttotal: 4.85ms\tremaining: 1.45s\n",
"0:\tlearn: 0.5244175\ttotal: 1.68ms\tremaining: 501ms\n",
"0:\tlearn: 0.5425779\ttotal: 3.6ms\tremaining: 1.08s\n",
"0:\tlearn: 0.4802076\ttotal: 2.31ms\tremaining: 690ms\n",
"0:\tlearn: 0.4982887\ttotal: 2.62ms\tremaining: 784ms\n",
"0:\tlearn: 0.4933666\ttotal: 2.54ms\tremaining: 761ms\n",
"0:\tlearn: 0.6788715\ttotal: 5.53ms\tremaining: 1.65s\n",
"0:\tlearn: 0.6835402\ttotal: 4.97ms\tremaining: 1.49s\n",
"0:\tlearn: 0.6814965\ttotal: 3.33ms\tremaining: 996ms\n",
"0:\tlearn: 0.6588327\ttotal: 4.06ms\tremaining: 1.21s\n",
"0:\tlearn: 0.6568276\ttotal: 2.67ms\tremaining: 798ms\n",
"0:\tlearn: 0.6559828\ttotal: 3.28ms\tremaining: 980ms\n",
"0:\tlearn: 0.5714939\ttotal: 3.33ms\tremaining: 997ms\n",
"0:\tlearn: 0.5775452\ttotal: 1.87ms\tremaining: 559ms\n",
"0:\tlearn: 0.5966192\ttotal: 5.91ms\tremaining: 1.77s\n",
"0:\tlearn: 0.5070442\ttotal: 4.34ms\tremaining: 1.3s\n",
"0:\tlearn: 0.5021608\ttotal: 2.64ms\tremaining: 789ms\n",
"0:\tlearn: 0.5035187\ttotal: 2.69ms\tremaining: 805ms\n",
"0:\tlearn: 0.4459619\ttotal: 2.45ms\tremaining: 732ms\n",
"0:\tlearn: 0.4574967\ttotal: 3.29ms\tremaining: 983ms\n",
"0:\tlearn: 0.4617752\ttotal: 3.71ms\tremaining: 1.11s\n",
"0:\tlearn: 0.6812721\ttotal: 3.07ms\tremaining: 919ms\n",
"0:\tlearn: 0.6808104\ttotal: 4.83ms\tremaining: 1.44s\n",
"0:\tlearn: 0.6827044\ttotal: 2.37ms\tremaining: 710ms\n",
"0:\tlearn: 0.6523513\ttotal: 3.75ms\tremaining: 1.12s\n",
"0:\tlearn: 0.6533735\ttotal: 7.02ms\tremaining: 2.1s\n",
"0:\tlearn: 0.6641291\ttotal: 3.93ms\tremaining: 1.18s\n",
"0:\tlearn: 0.5722754\ttotal: 2.26ms\tremaining: 677ms\n",
"0:\tlearn: 0.5630016\ttotal: 4.62ms\tremaining: 1.38s\n",
"0:\tlearn: 0.5749636\ttotal: 2.5ms\tremaining: 747ms\n",
"0:\tlearn: 0.5448886\ttotal: 5.47ms\tremaining: 1.63s\n",
"0:\tlearn: 0.5240179\ttotal: 2.44ms\tremaining: 729ms\n",
"0:\tlearn: 0.5265722\ttotal: 1.8ms\tremaining: 538ms\n",
"0:\tlearn: 0.4979202\ttotal: 2.49ms\tremaining: 744ms\n",
"0:\tlearn: 0.4780331\ttotal: 2.81ms\tremaining: 839ms\n",
"0:\tlearn: 0.4837085\ttotal: 2.43ms\tremaining: 726ms\n",
"0:\tlearn: 0.6755397\ttotal: 3.31ms\tremaining: 991ms\n",
"0:\tlearn: 0.6775787\ttotal: 6.16ms\tremaining: 1.84s\n",
"0:\tlearn: 0.6747756\ttotal: 3.33ms\tremaining: 995ms\n",
"0:\tlearn: 0.6529347\ttotal: 3.58ms\tremaining: 1.07s\n",
"0:\tlearn: 0.6394422\ttotal: 3.44ms\tremaining: 1.03s\n",
"0:\tlearn: 0.6510305\ttotal: 4.85ms\tremaining: 1.45s\n",
"0:\tlearn: 0.5553942\ttotal: 5.25ms\tremaining: 1.57s\n",
"0:\tlearn: 0.5463883\ttotal: 3.25ms\tremaining: 972ms\n",
"0:\tlearn: 0.5408561\ttotal: 8.36ms\tremaining: 2.5s\n",
"0:\tlearn: 0.4949652\ttotal: 4.64ms\tremaining: 1.39s\n",
"0:\tlearn: 0.5101659\ttotal: 4ms\tremaining: 1.2s\n",
"0:\tlearn: 0.4566089\ttotal: 4.38ms\tremaining: 1.31s\n",
"0:\tlearn: 0.4610889\ttotal: 3.03ms\tremaining: 906ms\n",
"0:\tlearn: 0.4703945\ttotal: 5.1ms\tremaining: 1.53s\n",
"0:\tlearn: 0.4267787\ttotal: 5.4ms\tremaining: 1.61s\n",
"0:\tlearn: 0.6755084\ttotal: 2.78ms\tremaining: 833ms\n",
"0:\tlearn: 0.6773947\ttotal: 4.78ms\tremaining: 1.43s\n",
"0:\tlearn: 0.6781341\ttotal: 7.19ms\tremaining: 2.15s\n",
"0:\tlearn: 0.6474001\ttotal: 5.4ms\tremaining: 1.61s\n",
"0:\tlearn: 0.6575429\ttotal: 4.42ms\tremaining: 1.32s\n",
"0:\tlearn: 0.6457112\ttotal: 3.99ms\tremaining: 1.19s\n",
"0:\tlearn: 0.5880805\ttotal: 4.2ms\tremaining: 1.25s\n",
"0:\tlearn: 0.5510624\ttotal: 5.58ms\tremaining: 1.67s\n",
"0:\tlearn: 0.5528950\ttotal: 5.62ms\tremaining: 1.68s\n",
"0:\tlearn: 0.5299722\ttotal: 5.81ms\tremaining: 1.74s\n",
"0:\tlearn: 0.5121420\ttotal: 2.88ms\tremaining: 860ms\n",
"0:\tlearn: 0.5164399\ttotal: 3.78ms\tremaining: 1.13s\n",
"0:\tlearn: 0.4479937\ttotal: 3.36ms\tremaining: 1s\n",
"0:\tlearn: 0.4862367\ttotal: 4.39ms\tremaining: 1.31s\n",
"0:\tlearn: 0.4370215\ttotal: 4.42ms\tremaining: 1.32s\n",
"0:\tlearn: 0.6763191\ttotal: 5.12ms\tremaining: 1.53s\n",
"0:\tlearn: 0.6795027\ttotal: 4.42ms\tremaining: 1.32s\n",
"0:\tlearn: 0.6788951\ttotal: 4.98ms\tremaining: 1.49s\n",
"0:\tlearn: 0.6569673\ttotal: 5.41ms\tremaining: 1.62s\n",
"0:\tlearn: 0.6497692\ttotal: 3.34ms\tremaining: 999ms\n",
"0:\tlearn: 0.6504118\ttotal: 4.53ms\tremaining: 1.35s\n",
"0:\tlearn: 0.5685806\ttotal: 4.16ms\tremaining: 1.24s\n",
"0:\tlearn: 0.5789459\ttotal: 5.8ms\tremaining: 1.74s\n",
"0:\tlearn: 0.5724911\ttotal: 4.53ms\tremaining: 1.35s\n",
"0:\tlearn: 0.5122118\ttotal: 4.1ms\tremaining: 1.23s\n",
"0:\tlearn: 0.5230818\ttotal: 2.97ms\tremaining: 889ms\n",
"0:\tlearn: 0.5126159\ttotal: 3.5ms\tremaining: 1.05s\n",
"0:\tlearn: 0.4913266\ttotal: 4.49ms\tremaining: 1.34s\n",
"0:\tlearn: 0.4551784\ttotal: 4.78ms\tremaining: 1.43s\n",
"0:\tlearn: 0.4727632\ttotal: 5.01ms\tremaining: 1.5s\n",
"0:\tlearn: 0.6771353\ttotal: 2.67ms\tremaining: 798ms\n",
"0:\tlearn: 0.6791263\ttotal: 6.1ms\tremaining: 1.82s\n",
"0:\tlearn: 0.6779145\ttotal: 5.08ms\tremaining: 1.52s\n",
"0:\tlearn: 0.6510000\ttotal: 4.25ms\tremaining: 1.27s\n",
"0:\tlearn: 0.6525793\ttotal: 7.06ms\tremaining: 2.11s\n",
"0:\tlearn: 0.6557962\ttotal: 3.67ms\tremaining: 1.1s\n",
"0:\tlearn: 0.5635795\ttotal: 6.24ms\tremaining: 1.86s\n",
"0:\tlearn: 0.5643913\ttotal: 5.7ms\tremaining: 1.7s\n",
"0:\tlearn: 0.5820207\ttotal: 4.87ms\tremaining: 1.46s\n",
"0:\tlearn: 0.5064923\ttotal: 3.27ms\tremaining: 978ms\n",
"0:\tlearn: 0.5167101\ttotal: 6.53ms\tremaining: 1.95s\n",
"0:\tlearn: 0.5161201\ttotal: 4.39ms\tremaining: 1.31s\n",
"0:\tlearn: 0.4506035\ttotal: 4.17ms\tremaining: 1.25s\n",
"0:\tlearn: 0.4830090\ttotal: 6.91ms\tremaining: 2.06s\n",
"0:\tlearn: 0.4832417\ttotal: 2.43ms\tremaining: 728ms\n",
"0:\tlearn: 0.6787261\ttotal: 2.27ms\tremaining: 678ms\n",
"0:\tlearn: 0.6800767\ttotal: 7.25ms\tremaining: 2.17s\n",
"0:\tlearn: 0.6799179\ttotal: 4.71ms\tremaining: 1.41s\n",
"0:\tlearn: 0.6563123\ttotal: 3.98ms\tremaining: 1.19s\n",
"0:\tlearn: 0.6539297\ttotal: 4.43ms\tremaining: 1.32s\n",
"0:\tlearn: 0.6562254\ttotal: 3.79ms\tremaining: 1.13s\n",
"0:\tlearn: 0.5736937\ttotal: 2.65ms\tremaining: 793ms\n",
"0:\tlearn: 0.5670417\ttotal: 4.74ms\tremaining: 1.42s\n",
"0:\tlearn: 0.5796575\ttotal: 4.76ms\tremaining: 1.42s\n",
"0:\tlearn: 0.5349756\ttotal: 5.37ms\tremaining: 1.6s\n",
"0:\tlearn: 0.5227944\ttotal: 4.14ms\tremaining: 1.24s\n",
"0:\tlearn: 0.5379618\ttotal: 3.57ms\tremaining: 1.07s\n",
"0:\tlearn: 0.4831227\ttotal: 3.23ms\tremaining: 966ms\n",
"0:\tlearn: 0.4929648\ttotal: 5.13ms\tremaining: 1.53s\n",
"0:\tlearn: 0.4804120\ttotal: 4.63ms\tremaining: 1.39s\n",
"0:\tlearn: 0.6743106\ttotal: 5.78ms\tremaining: 1.73s\n",
"0:\tlearn: 0.6776812\ttotal: 4.4ms\tremaining: 1.31s\n",
"0:\tlearn: 0.6778146\ttotal: 6.07ms\tremaining: 1.81s\n",
"0:\tlearn: 0.6449497\ttotal: 6.15ms\tremaining: 1.84s\n",
"0:\tlearn: 0.6489001\ttotal: 6.78ms\tremaining: 2.03s\n",
"0:\tlearn: 0.6445126\ttotal: 11.1ms\tremaining: 3.33s\n",
"0:\tlearn: 0.5423006\ttotal: 5.71ms\tremaining: 1.71s\n",
"0:\tlearn: 0.5288251\ttotal: 5.29ms\tremaining: 1.58s\n",
"0:\tlearn: 0.5391324\ttotal: 6.3ms\tremaining: 1.88s\n",
"0:\tlearn: 0.4571305\ttotal: 8.81ms\tremaining: 2.63s\n",
"0:\tlearn: 0.4841953\ttotal: 5.67ms\tremaining: 1.69s\n",
"0:\tlearn: 0.4966344\ttotal: 6.89ms\tremaining: 2.06s\n",
"0:\tlearn: 0.4013280\ttotal: 6.76ms\tremaining: 2.02s\n",
"0:\tlearn: 0.4664245\ttotal: 7.62ms\tremaining: 2.28s\n",
"0:\tlearn: 0.4515069\ttotal: 5.52ms\tremaining: 1.65s\n",
"0:\tlearn: 0.6806751\ttotal: 4.46ms\tremaining: 1.33s\n",
"0:\tlearn: 0.6780997\ttotal: 6.88ms\tremaining: 2.06s\n",
"0:\tlearn: 0.6801288\ttotal: 5.55ms\tremaining: 1.66s\n",
"0:\tlearn: 0.6448890\ttotal: 6.56ms\tremaining: 1.96s\n",
"0:\tlearn: 0.6488364\ttotal: 7.7ms\tremaining: 2.3s\n",
"0:\tlearn: 0.6498428\ttotal: 6.87ms\tremaining: 2.06s\n",
"0:\tlearn: 0.5441466\ttotal: 6.76ms\tremaining: 2.02s\n",
"0:\tlearn: 0.5723313\ttotal: 6.04ms\tremaining: 1.81s\n",
"0:\tlearn: 0.5657202\ttotal: 6.81ms\tremaining: 2.04s\n",
"0:\tlearn: 0.5206599\ttotal: 7.34ms\tremaining: 2.19s\n",
"0:\tlearn: 0.4903534\ttotal: 5.72ms\tremaining: 1.71s\n",
"0:\tlearn: 0.5167145\ttotal: 8.04ms\tremaining: 2.4s\n",
"0:\tlearn: 0.4559572\ttotal: 5.83ms\tremaining: 1.74s\n",
"0:\tlearn: 0.4535328\ttotal: 7.58ms\tremaining: 2.27s\n",
"0:\tlearn: 0.4823002\ttotal: 6.11ms\tremaining: 1.82s\n",
"0:\tlearn: 0.6799016\ttotal: 5.27ms\tremaining: 1.58s\n",
"0:\tlearn: 0.6807964\ttotal: 7.13ms\tremaining: 2.13s\n",
"0:\tlearn: 0.6811215\ttotal: 5.12ms\tremaining: 1.53s\n",
"0:\tlearn: 0.6495007\ttotal: 7.76ms\tremaining: 2.32s\n",
"0:\tlearn: 0.6586012\ttotal: 6.98ms\tremaining: 2.09s\n",
"0:\tlearn: 0.6493873\ttotal: 6.09ms\tremaining: 1.82s\n",
"0:\tlearn: 0.5547726\ttotal: 7.63ms\tremaining: 2.28s\n",
"0:\tlearn: 0.5709955\ttotal: 5.63ms\tremaining: 1.68s\n",
"0:\tlearn: 0.5705141\ttotal: 5.19ms\tremaining: 1.55s\n",
"0:\tlearn: 0.4983985\ttotal: 4.7ms\tremaining: 1.4s\n",
"0:\tlearn: 0.5193663\ttotal: 7.6ms\tremaining: 2.27s\n",
"0:\tlearn: 0.5257400\ttotal: 8.13ms\tremaining: 2.43s\n",
"0:\tlearn: 0.4771618\ttotal: 6.63ms\tremaining: 1.98s\n",
"0:\tlearn: 0.4653248\ttotal: 5.35ms\tremaining: 1.6s\n",
"0:\tlearn: 0.4808129\ttotal: 10.6ms\tremaining: 3.17s\n",
"0:\tlearn: 0.6773537\ttotal: 6.88ms\tremaining: 2.06s\n",
"0:\tlearn: 0.6796119\ttotal: 4.71ms\tremaining: 1.41s\n",
"0:\tlearn: 0.6805703\ttotal: 8.07ms\tremaining: 2.41s\n",
"0:\tlearn: 0.6511389\ttotal: 5ms\tremaining: 1.5s\n",
"0:\tlearn: 0.6584294\ttotal: 7.78ms\tremaining: 2.33s\n",
"0:\tlearn: 0.6571252\ttotal: 8.87ms\tremaining: 2.65s\n",
"0:\tlearn: 0.5558355\ttotal: 7.42ms\tremaining: 2.22s\n",
"0:\tlearn: 0.5782454\ttotal: 4.71ms\tremaining: 1.41s\n",
"0:\tlearn: 0.5626353\ttotal: 5.34ms\tremaining: 1.6s\n",
"0:\tlearn: 0.5220926\ttotal: 5.87ms\tremaining: 1.75s\n",
"0:\tlearn: 0.5228394\ttotal: 5.69ms\tremaining: 1.7s\n",
"0:\tlearn: 0.5242810\ttotal: 5.62ms\tremaining: 1.68s\n",
"0:\tlearn: 0.4640638\ttotal: 6.05ms\tremaining: 1.81s\n",
"0:\tlearn: 0.4832789\ttotal: 4.28ms\tremaining: 1.28s\n",
"0:\tlearn: 0.5094620\ttotal: 4.48ms\tremaining: 1.34s\n",
"0:\tlearn: 0.6794259\ttotal: 8.17ms\tremaining: 2.44s\n",
"0:\tlearn: 0.6819624\ttotal: 6.46ms\tremaining: 1.93s\n",
"0:\tlearn: 0.6810833\ttotal: 6.11ms\tremaining: 1.83s\n",
"0:\tlearn: 0.6532264\ttotal: 7.01ms\tremaining: 2.1s\n",
"0:\tlearn: 0.6578880\ttotal: 5.47ms\tremaining: 1.64s\n",
"0:\tlearn: 0.6560282\ttotal: 6.6ms\tremaining: 1.97s\n",
"0:\tlearn: 0.5793249\ttotal: 6.14ms\tremaining: 1.83s\n",
"0:\tlearn: 0.5858695\ttotal: 5.21ms\tremaining: 1.56s\n",
"0:\tlearn: 0.5721219\ttotal: 6.44ms\tremaining: 1.93s\n",
"0:\tlearn: 0.5299483\ttotal: 4.89ms\tremaining: 1.46s\n",
"0:\tlearn: 0.5384012\ttotal: 6.86ms\tremaining: 2.05s\n",
"0:\tlearn: 0.5214187\ttotal: 5.67ms\tremaining: 1.7s\n",
"0:\tlearn: 0.4876877\ttotal: 6.53ms\tremaining: 1.95s\n",
"0:\tlearn: 0.4768380\ttotal: 6.36ms\tremaining: 1.9s\n",
"0:\tlearn: 0.4976312\ttotal: 8.1ms\tremaining: 2.42s\n",
"0:\tlearn: 0.6766882\ttotal: 12.8ms\tremaining: 3.82s\n",
"0:\tlearn: 0.6756084\ttotal: 9.56ms\tremaining: 2.86s\n",
"0:\tlearn: 0.6778394\ttotal: 12.4ms\tremaining: 3.71s\n",
"0:\tlearn: 0.6380815\ttotal: 10.4ms\tremaining: 3.11s\n",
"0:\tlearn: 0.6459530\ttotal: 13ms\tremaining: 3.88s\n",
"0:\tlearn: 0.6369186\ttotal: 11.1ms\tremaining: 3.32s\n",
"0:\tlearn: 0.5323338\ttotal: 13.2ms\tremaining: 3.95s\n",
"0:\tlearn: 0.5336079\ttotal: 9.58ms\tremaining: 2.86s\n",
"0:\tlearn: 0.5397687\ttotal: 10.8ms\tremaining: 3.23s\n",
"0:\tlearn: 0.4798964\ttotal: 10.2ms\tremaining: 3.04s\n",
"0:\tlearn: 0.4979005\ttotal: 13.6ms\tremaining: 4.06s\n",
"0:\tlearn: 0.4959239\ttotal: 13.4ms\tremaining: 4.01s\n",
"0:\tlearn: 0.3923967\ttotal: 13.2ms\tremaining: 3.95s\n",
"0:\tlearn: 0.4216806\ttotal: 12.3ms\tremaining: 3.67s\n",
"0:\tlearn: 0.4700379\ttotal: 11.2ms\tremaining: 3.33s\n",
"0:\tlearn: 0.6786077\ttotal: 10.4ms\tremaining: 3.11s\n",
"0:\tlearn: 0.6777055\ttotal: 11.6ms\tremaining: 3.46s\n",
"0:\tlearn: 0.6790016\ttotal: 13.3ms\tremaining: 3.99s\n",
"0:\tlearn: 0.6524699\ttotal: 7.64ms\tremaining: 2.28s\n",
"0:\tlearn: 0.6527967\ttotal: 10ms\tremaining: 2.99s\n",
"0:\tlearn: 0.6448447\ttotal: 11.1ms\tremaining: 3.32s\n",
"0:\tlearn: 0.5535797\ttotal: 12.4ms\tremaining: 3.72s\n",
"0:\tlearn: 0.5701482\ttotal: 11.8ms\tremaining: 3.53s\n",
"0:\tlearn: 0.5714067\ttotal: 9.92ms\tremaining: 2.96s\n",
"0:\tlearn: 0.5225831\ttotal: 13.4ms\tremaining: 4s\n",
"0:\tlearn: 0.4939300\ttotal: 12.3ms\tremaining: 3.68s\n",
"0:\tlearn: 0.5038585\ttotal: 12.5ms\tremaining: 3.74s\n",
"0:\tlearn: 0.4837578\ttotal: 11.1ms\tremaining: 3.31s\n",
"0:\tlearn: 0.4682088\ttotal: 14.9ms\tremaining: 4.45s\n",
"0:\tlearn: 0.4545791\ttotal: 12ms\tremaining: 3.59s\n",
"0:\tlearn: 0.6789956\ttotal: 16.7ms\tremaining: 4.98s\n",
"0:\tlearn: 0.6794206\ttotal: 10.2ms\tremaining: 3.05s\n",
"0:\tlearn: 0.6790119\ttotal: 11ms\tremaining: 3.29s\n",
"0:\tlearn: 0.6507991\ttotal: 10.3ms\tremaining: 3.09s\n",
"0:\tlearn: 0.6563977\ttotal: 11.1ms\tremaining: 3.31s\n",
"0:\tlearn: 0.6530569\ttotal: 11.8ms\tremaining: 3.52s\n",
"0:\tlearn: 0.5652976\ttotal: 11.1ms\tremaining: 3.31s\n",
"0:\tlearn: 0.5745746\ttotal: 11.8ms\tremaining: 3.53s\n",
"0:\tlearn: 0.5746444\ttotal: 9.63ms\tremaining: 2.88s\n",
"0:\tlearn: 0.5009619\ttotal: 12.3ms\tremaining: 3.69s\n",
"0:\tlearn: 0.5076188\ttotal: 11.7ms\tremaining: 3.49s\n",
"0:\tlearn: 0.5080836\ttotal: 12.4ms\tremaining: 3.7s\n",
"0:\tlearn: 0.4620329\ttotal: 14.3ms\tremaining: 4.28s\n",
"0:\tlearn: 0.4815555\ttotal: 15.6ms\tremaining: 4.67s\n",
"0:\tlearn: 0.4440580\ttotal: 12.4ms\tremaining: 3.69s\n",
"0:\tlearn: 0.6798617\ttotal: 10.8ms\tremaining: 3.23s\n",
"0:\tlearn: 0.6803465\ttotal: 10.7ms\tremaining: 3.21s\n",
"0:\tlearn: 0.6800037\ttotal: 14.6ms\tremaining: 4.36s\n",
"0:\tlearn: 0.6492530\ttotal: 11.3ms\tremaining: 3.38s\n",
"0:\tlearn: 0.6564213\ttotal: 10.9ms\tremaining: 3.25s\n",
"0:\tlearn: 0.6586911\ttotal: 9.4ms\tremaining: 2.81s\n",
"0:\tlearn: 0.5662428\ttotal: 10.2ms\tremaining: 3.06s\n",
"0:\tlearn: 0.5915932\ttotal: 18.6ms\tremaining: 5.55s\n",
"0:\tlearn: 0.5898168\ttotal: 11.9ms\tremaining: 3.54s\n",
"0:\tlearn: 0.5101613\ttotal: 13.3ms\tremaining: 3.98s\n",
"0:\tlearn: 0.5287853\ttotal: 13.8ms\tremaining: 4.12s\n",
"0:\tlearn: 0.5162068\ttotal: 15.6ms\tremaining: 4.67s\n",
"0:\tlearn: 0.4709647\ttotal: 12.2ms\tremaining: 3.64s\n",
"0:\tlearn: 0.4845199\ttotal: 10.5ms\tremaining: 3.13s\n",
"0:\tlearn: 0.4905388\ttotal: 14.4ms\tremaining: 4.29s\n",
"0:\tlearn: 0.6821250\ttotal: 12ms\tremaining: 3.58s\n",
"0:\tlearn: 0.6816714\ttotal: 12ms\tremaining: 3.6s\n",
"0:\tlearn: 0.6814836\ttotal: 13ms\tremaining: 3.88s\n",
"0:\tlearn: 0.6573797\ttotal: 10.1ms\tremaining: 3.02s\n",
"0:\tlearn: 0.6570641\ttotal: 11.5ms\tremaining: 3.43s\n",
"0:\tlearn: 0.6553556\ttotal: 11ms\tremaining: 3.29s\n",
"0:\tlearn: 0.5749051\ttotal: 14ms\tremaining: 4.19s\n",
"0:\tlearn: 0.5768753\ttotal: 11ms\tremaining: 3.29s\n",
"0:\tlearn: 0.5852401\ttotal: 13.3ms\tremaining: 3.97s\n",
"0:\tlearn: 0.5350025\ttotal: 12.3ms\tremaining: 3.69s\n",
"0:\tlearn: 0.5535985\ttotal: 12ms\tremaining: 3.59s\n",
"0:\tlearn: 0.5212525\ttotal: 12.3ms\tremaining: 3.69s\n",
"0:\tlearn: 0.4924604\ttotal: 16ms\tremaining: 4.77s\n",
"0:\tlearn: 0.4903541\ttotal: 12.5ms\tremaining: 3.74s\n",
"0:\tlearn: 0.4979451\ttotal: 10.1ms\tremaining: 3.02s\n",
"0:\tlearn: 0.6744949\ttotal: 43.7ms\tremaining: 13.1s\n",
"0:\tlearn: 0.6787579\ttotal: 40.1ms\tremaining: 12s\n",
"0:\tlearn: 0.6777435\ttotal: 34ms\tremaining: 10.2s\n",
"0:\tlearn: 0.6375237\ttotal: 35.3ms\tremaining: 10.5s\n",
"0:\tlearn: 0.6462363\ttotal: 42.4ms\tremaining: 12.7s\n",
"0:\tlearn: 0.6422612\ttotal: 35.5ms\tremaining: 10.6s\n",
"0:\tlearn: 0.5582762\ttotal: 42.3ms\tremaining: 12.7s\n",
"0:\tlearn: 0.5505489\ttotal: 40.6ms\tremaining: 12.1s\n",
"0:\tlearn: 0.5433797\ttotal: 40ms\tremaining: 12s\n",
"0:\tlearn: 0.4729717\ttotal: 33.6ms\tremaining: 10.1s\n",
"0:\tlearn: 0.5075606\ttotal: 49.4ms\tremaining: 14.8s\n",
"0:\tlearn: 0.4624069\ttotal: 36.7ms\tremaining: 11s\n",
"0:\tlearn: 0.4051759\ttotal: 35.7ms\tremaining: 10.7s\n",
"0:\tlearn: 0.4451289\ttotal: 42.6ms\tremaining: 12.7s\n",
"0:\tlearn: 0.3921217\ttotal: 38.5ms\tremaining: 11.5s\n",
"0:\tlearn: 0.6784394\ttotal: 40.9ms\tremaining: 12.2s\n",
"0:\tlearn: 0.6765293\ttotal: 31ms\tremaining: 9.28s\n",
"0:\tlearn: 0.6795951\ttotal: 34.6ms\tremaining: 10.3s\n",
"0:\tlearn: 0.6494188\ttotal: 32.4ms\tremaining: 9.7s\n",
"0:\tlearn: 0.6501507\ttotal: 37.7ms\tremaining: 11.3s\n",
"0:\tlearn: 0.6504714\ttotal: 39.5ms\tremaining: 11.8s\n",
"0:\tlearn: 0.5623603\ttotal: 35.8ms\tremaining: 10.7s\n",
"0:\tlearn: 0.5806660\ttotal: 46.7ms\tremaining: 13.9s\n",
"0:\tlearn: 0.5621579\ttotal: 42ms\tremaining: 12.6s\n",
"0:\tlearn: 0.4841510\ttotal: 35.2ms\tremaining: 10.5s\n",
"0:\tlearn: 0.5120376\ttotal: 38.7ms\tremaining: 11.6s\n",
"0:\tlearn: 0.5122134\ttotal: 39.2ms\tremaining: 11.7s\n",
"0:\tlearn: 0.4375847\ttotal: 48.5ms\tremaining: 14.5s\n",
"0:\tlearn: 0.4727207\ttotal: 32.2ms\tremaining: 9.62s\n",
"0:\tlearn: 0.4658269\ttotal: 8.64ms\tremaining: 2.58s\n",
"0:\tlearn: 0.6786432\ttotal: 40.5ms\tremaining: 12.1s\n",
"0:\tlearn: 0.6812723\ttotal: 40.2ms\tremaining: 12s\n",
"0:\tlearn: 0.6811281\ttotal: 36.8ms\tremaining: 11s\n",
"0:\tlearn: 0.6496256\ttotal: 41.5ms\tremaining: 12.4s\n",
"0:\tlearn: 0.6574481\ttotal: 47ms\tremaining: 14.1s\n",
"0:\tlearn: 0.6521655\ttotal: 58.5ms\tremaining: 17.5s\n",
"0:\tlearn: 0.5627419\ttotal: 41.4ms\tremaining: 12.4s\n",
"0:\tlearn: 0.5766052\ttotal: 44.3ms\tremaining: 13.2s\n",
"0:\tlearn: 0.5664168\ttotal: 38.2ms\tremaining: 11.4s\n",
"0:\tlearn: 0.5320755\ttotal: 36.6ms\tremaining: 10.9s\n",
"0:\tlearn: 0.5335885\ttotal: 32.5ms\tremaining: 9.72s\n",
"0:\tlearn: 0.5343398\ttotal: 41.3ms\tremaining: 12.3s\n",
"0:\tlearn: 0.4807229\ttotal: 42.1ms\tremaining: 12.6s\n",
"0:\tlearn: 0.4704637\ttotal: 47.5ms\tremaining: 14.2s\n",
"0:\tlearn: 0.4808865\ttotal: 45.1ms\tremaining: 13.5s\n",
"0:\tlearn: 0.6808239\ttotal: 39.8ms\tremaining: 11.9s\n",
"0:\tlearn: 0.6821663\ttotal: 44.4ms\tremaining: 13.3s\n",
"0:\tlearn: 0.6802639\ttotal: 44.3ms\tremaining: 13.3s\n",
"0:\tlearn: 0.6577968\ttotal: 38.5ms\tremaining: 11.5s\n",
"0:\tlearn: 0.6580819\ttotal: 38.4ms\tremaining: 11.5s\n",
"0:\tlearn: 0.6614477\ttotal: 38.9ms\tremaining: 11.6s\n",
"0:\tlearn: 0.5653852\ttotal: 28.4ms\tremaining: 8.49s\n",
"0:\tlearn: 0.5694080\ttotal: 43.2ms\tremaining: 12.9s\n",
"0:\tlearn: 0.5774342\ttotal: 39.8ms\tremaining: 11.9s\n",
"0:\tlearn: 0.5071954\ttotal: 46.6ms\tremaining: 13.9s\n",
"0:\tlearn: 0.5176690\ttotal: 4.55ms\tremaining: 1.36s\n",
"0:\tlearn: 0.5349606\ttotal: 37.6ms\tremaining: 11.2s\n",
"0:\tlearn: 0.4798021\ttotal: 36.6ms\tremaining: 11s\n",
"0:\tlearn: 0.4808540\ttotal: 36.8ms\tremaining: 11s\n",
"0:\tlearn: 0.4810487\ttotal: 47.2ms\tremaining: 14.1s\n",
"0:\tlearn: 0.6816982\ttotal: 44.1ms\tremaining: 13.2s\n",
"0:\tlearn: 0.6834005\ttotal: 50.8ms\tremaining: 15.2s\n",
"0:\tlearn: 0.6804206\ttotal: 46.4ms\tremaining: 13.9s\n",
"0:\tlearn: 0.6525155\ttotal: 37.6ms\tremaining: 11.2s\n",
"0:\tlearn: 0.6610592\ttotal: 34.4ms\tremaining: 10.3s\n",
"0:\tlearn: 0.6611896\ttotal: 47.2ms\tremaining: 14.1s\n",
"0:\tlearn: 0.5942750\ttotal: 44.8ms\tremaining: 13.4s\n",
"0:\tlearn: 0.5868268\ttotal: 32.3ms\tremaining: 9.66s\n",
"0:\tlearn: 0.5784653\ttotal: 39.9ms\tremaining: 11.9s\n",
"0:\tlearn: 0.5427078\ttotal: 39.9ms\tremaining: 11.9s\n",
"0:\tlearn: 0.5587203\ttotal: 37.8ms\tremaining: 11.3s\n",
"0:\tlearn: 0.5354851\ttotal: 32.4ms\tremaining: 9.69s\n",
"0:\tlearn: 0.4972968\ttotal: 32.6ms\tremaining: 9.75s\n",
"0:\tlearn: 0.5165868\ttotal: 45.7ms\tremaining: 13.7s\n",
"0:\tlearn: 0.5084522\ttotal: 47ms\tremaining: 14s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=1)]: Done 375 out of 375 | elapsed: 23.1min finished\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"0:\tlearn: 0.6805422\ttotal: 36.2ms\tremaining: 10.8s\n",
"Computation time (parameter tunning): 1394 seconds...\n",
"Best parameters: {'depth': 10, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"0:\tlearn: 0.9757522\ttotal: 35.6ms\tremaining: 10.7s\n",
"30:\tlearn: 0.9876800\ttotal: 840ms\tremaining: 7.29s\n",
"60:\tlearn: 0.9899921\ttotal: 1.67s\tremaining: 6.54s\n",
"90:\tlearn: 0.9915032\ttotal: 2.53s\tremaining: 5.82s\n",
"120:\tlearn: 0.9927831\ttotal: 3.38s\tremaining: 5.01s\n",
"150:\tlearn: 0.9936088\ttotal: 4.15s\tremaining: 4.1s\n",
"180:\tlearn: 0.9944758\ttotal: 5.03s\tremaining: 3.31s\n",
"210:\tlearn: 0.9952107\ttotal: 5.88s\tremaining: 2.48s\n",
"240:\tlearn: 0.9960695\ttotal: 6.7s\tremaining: 1.64s\n",
"270:\tlearn: 0.9968374\ttotal: 7.53s\tremaining: 806ms\n",
"********** SUMMARY: XGBoost (Exp5-Undersampled) **************\n",
"- Parameters used: {'depth': 10, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"- AUC(Train|Test): (0.99737415774871185, 0.97999414803612384)\n",
"- Metrics: Train data (n=696) **\n",
" TN=345, FP=3\n",
" FN=20, TP=328\n",
" Precision: 0.99\n",
" Recall(TPR): 0.94\n",
" BAcc=(TPR+TNR)/2: 0.97\n",
"- Metrics: Test data (n=85443) **\n",
" TN=83571, FP=1728\n",
" FN=16, TP=128\n",
" Precision: 0.07\n",
" Recall(TPR): 0.89\n",
" BAcc=(TPR+TNR)/2: 0.93\n",
"Computation time (Total): 9 seconds\n",
"Using parameters: {'depth': 10, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"0:\tlearn: 0.9966839\ttotal: 158ms\tremaining: 47.3s\n",
"30:\tlearn: 0.9989378\ttotal: 5.47s\tremaining: 47.5s\n",
"60:\tlearn: 0.9991746\ttotal: 11.1s\tremaining: 43.4s\n",
"90:\tlearn: 0.9994173\ttotal: 16.6s\tremaining: 38.1s\n",
"120:\tlearn: 0.9995694\ttotal: 22.2s\tremaining: 32.8s\n",
"150:\tlearn: 0.9996831\ttotal: 27.7s\tremaining: 27.3s\n",
"180:\tlearn: 0.9997494\ttotal: 33.3s\tremaining: 21.9s\n",
"210:\tlearn: 0.9997967\ttotal: 38.9s\tremaining: 16.4s\n",
"240:\tlearn: 0.9998361\ttotal: 44.4s\tremaining: 10.9s\n",
"270:\tlearn: 0.9998584\ttotal: 49.9s\tremaining: 5.34s\n",
"********** SUMMARY: XGBoost (Exp5-SMOTE (best parameters from Undersampled)) **************\n",
"- Parameters used: {'depth': 10, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"- AUC(Train|Test): (0.99987530067718511, 0.98224391389243848)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=198643, FP=373\n",
" FN=602, TP=198414\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85123, FP=176\n",
" FN=21, TP=123\n",
" Precision: 0.41\n",
" Recall(TPR): 0.85\n",
" BAcc=(TPR+TNR)/2: 0.93\n",
"Computation time (Total): 61 seconds\n",
"\n",
"\n",
"\n",
"Fitting 3 folds for each of 125 candidates, totalling 375 fits\n",
"0:\tlearn: 0.6806815\ttotal: 8.16ms\tremaining: 2.44s\n",
"0:\tlearn: 0.6781766\ttotal: 2.15ms\tremaining: 642ms\n",
"0:\tlearn: 0.6786779\ttotal: 1.62ms\tremaining: 486ms\n",
"0:\tlearn: 0.6564682\ttotal: 4.19ms\tremaining: 1.25s\n",
"0:\tlearn: 0.6557293\ttotal: 3.8ms\tremaining: 1.14s\n",
"0:\tlearn: 0.6455712\ttotal: 3.79ms\tremaining: 1.13s\n",
"0:\tlearn: 0.5743258\ttotal: 2.63ms\tremaining: 788ms\n",
"0:\tlearn: 0.5911110\ttotal: 3.07ms\tremaining: 918ms\n",
"0:\tlearn: 0.5772984\ttotal: 3.71ms\tremaining: 1.11s\n",
"0:\tlearn: 0.5374510\ttotal: 4.1ms\tremaining: 1.22s\n",
"0:\tlearn: 0.5247147\ttotal: 4.5ms\tremaining: 1.34s\n",
"0:\tlearn: 0.5112121\ttotal: 4.08ms\tremaining: 1.22s\n",
"0:\tlearn: 0.4839175\ttotal: 3.46ms\tremaining: 1.03s\n",
"0:\tlearn: 0.4892244\ttotal: 2.28ms\tremaining: 683ms\n",
"0:\tlearn: 0.5261726\ttotal: 2.69ms\tremaining: 803ms\n",
"0:\tlearn: 0.6785660\ttotal: 5.14ms\tremaining: 1.54s\n",
"0:\tlearn: 0.6790792\ttotal: 2.88ms\tremaining: 861ms\n",
"0:\tlearn: 0.6798327\ttotal: 3.72ms\tremaining: 1.11s\n",
"0:\tlearn: 0.6588773\ttotal: 4.79ms\tremaining: 1.43s\n",
"0:\tlearn: 0.6620213\ttotal: 4.2ms\tremaining: 1.25s\n",
"0:\tlearn: 0.6558154\ttotal: 1.52ms\tremaining: 454ms\n",
"0:\tlearn: 0.5600259\ttotal: 2.23ms\tremaining: 666ms\n",
"0:\tlearn: 0.5894426\ttotal: 3.87ms\tremaining: 1.16s\n",
"0:\tlearn: 0.5805324\ttotal: 3.9ms\tremaining: 1.17s\n",
"0:\tlearn: 0.5030250\ttotal: 3.39ms\tremaining: 1.01s\n",
"0:\tlearn: 0.5153704\ttotal: 3.41ms\tremaining: 1.02s\n",
"0:\tlearn: 0.5276363\ttotal: 2.88ms\tremaining: 862ms\n",
"0:\tlearn: 0.4509759\ttotal: 2.82ms\tremaining: 844ms\n",
"0:\tlearn: 0.5444975\ttotal: 5.51ms\tremaining: 1.65s\n",
"0:\tlearn: 0.4996852\ttotal: 2.9ms\tremaining: 867ms\n",
"0:\tlearn: 0.6799945\ttotal: 2.25ms\tremaining: 672ms\n",
"0:\tlearn: 0.6797339\ttotal: 3.44ms\tremaining: 1.03s\n",
"0:\tlearn: 0.6801737\ttotal: 4.43ms\tremaining: 1.33s\n",
"0:\tlearn: 0.6497087\ttotal: 2.55ms\tremaining: 762ms\n",
"0:\tlearn: 0.6609664\ttotal: 4.14ms\tremaining: 1.24s\n",
"0:\tlearn: 0.6534890\ttotal: 3.96ms\tremaining: 1.18s\n",
"0:\tlearn: 0.5466920\ttotal: 2.23ms\tremaining: 666ms\n",
"0:\tlearn: 0.5882933\ttotal: 4.42ms\tremaining: 1.32s\n",
"0:\tlearn: 0.5760560\ttotal: 2.22ms\tremaining: 663ms\n",
"0:\tlearn: 0.5550171\ttotal: 6.39ms\tremaining: 1.91s\n",
"0:\tlearn: 0.5342596\ttotal: 5.22ms\tremaining: 1.56s\n",
"0:\tlearn: 0.4946068\ttotal: 4.37ms\tremaining: 1.31s\n",
"0:\tlearn: 0.4371225\ttotal: 5.21ms\tremaining: 1.56s\n",
"0:\tlearn: 0.4911728\ttotal: 2.04ms\tremaining: 611ms\n",
"0:\tlearn: 0.4724947\ttotal: 3.88ms\tremaining: 1.16s\n",
"0:\tlearn: 0.6806473\ttotal: 4.87ms\tremaining: 1.45s\n",
"0:\tlearn: 0.6809358\ttotal: 3.32ms\tremaining: 992ms\n",
"0:\tlearn: 0.6796018\ttotal: 3.75ms\tremaining: 1.12s\n",
"0:\tlearn: 0.6554948\ttotal: 4.33ms\tremaining: 1.29s\n",
"0:\tlearn: 0.6542348\ttotal: 4.33ms\tremaining: 1.29s\n",
"0:\tlearn: 0.6529540\ttotal: 3.2ms\tremaining: 957ms\n",
"0:\tlearn: 0.5763279\ttotal: 4.13ms\tremaining: 1.23s\n",
"0:\tlearn: 0.5950889\ttotal: 3.05ms\tremaining: 912ms\n",
"0:\tlearn: 0.5807446\ttotal: 2.46ms\tremaining: 734ms\n",
"0:\tlearn: 0.5321385\ttotal: 3.44ms\tremaining: 1.03s\n",
"0:\tlearn: 0.5424769\ttotal: 2.98ms\tremaining: 892ms\n",
"0:\tlearn: 0.5110818\ttotal: 5.09ms\tremaining: 1.52s\n",
"0:\tlearn: 0.4721951\ttotal: 3.31ms\tremaining: 991ms\n",
"0:\tlearn: 0.4919216\ttotal: 2.38ms\tremaining: 712ms\n",
"0:\tlearn: 0.4739656\ttotal: 2.36ms\tremaining: 706ms\n",
"0:\tlearn: 0.6776103\ttotal: 3.48ms\tremaining: 1.04s\n",
"0:\tlearn: 0.6811576\ttotal: 4.28ms\tremaining: 1.28s\n",
"0:\tlearn: 0.6792764\ttotal: 5.51ms\tremaining: 1.65s\n",
"0:\tlearn: 0.6536534\ttotal: 3.64ms\tremaining: 1.09s\n",
"0:\tlearn: 0.6590176\ttotal: 4.68ms\tremaining: 1.4s\n",
"0:\tlearn: 0.6622037\ttotal: 5.35ms\tremaining: 1.6s\n",
"0:\tlearn: 0.5664288\ttotal: 4.84ms\tremaining: 1.45s\n",
"0:\tlearn: 0.5883072\ttotal: 3.33ms\tremaining: 995ms\n",
"0:\tlearn: 0.5872451\ttotal: 2.76ms\tremaining: 827ms\n",
"0:\tlearn: 0.5205922\ttotal: 4.62ms\tremaining: 1.38s\n",
"0:\tlearn: 0.5357832\ttotal: 3.12ms\tremaining: 934ms\n",
"0:\tlearn: 0.5112989\ttotal: 3.09ms\tremaining: 925ms\n",
"0:\tlearn: 0.4704743\ttotal: 4.42ms\tremaining: 1.32s\n",
"0:\tlearn: 0.4965879\ttotal: 3.81ms\tremaining: 1.14s\n",
"0:\tlearn: 0.4573103\ttotal: 2.73ms\tremaining: 816ms\n",
"0:\tlearn: 0.6745641\ttotal: 4.64ms\tremaining: 1.39s\n",
"0:\tlearn: 0.6752296\ttotal: 2.5ms\tremaining: 747ms\n",
"0:\tlearn: 0.6774655\ttotal: 2.91ms\tremaining: 870ms\n",
"0:\tlearn: 0.6380563\ttotal: 3.4ms\tremaining: 1.02s\n",
"0:\tlearn: 0.6479365\ttotal: 3.43ms\tremaining: 1.03s\n",
"0:\tlearn: 0.6407864\ttotal: 2.06ms\tremaining: 617ms\n",
"0:\tlearn: 0.5120140\ttotal: 5.41ms\tremaining: 1.62s\n",
"0:\tlearn: 0.5543730\ttotal: 4.04ms\tremaining: 1.21s\n",
"0:\tlearn: 0.5312355\ttotal: 5.96ms\tremaining: 1.78s\n",
"0:\tlearn: 0.4699393\ttotal: 6.3ms\tremaining: 1.88s\n",
"0:\tlearn: 0.4879585\ttotal: 4.42ms\tremaining: 1.32s\n",
"0:\tlearn: 0.4746054\ttotal: 5.63ms\tremaining: 1.68s\n",
"0:\tlearn: 0.4015263\ttotal: 4.93ms\tremaining: 1.48s\n",
"0:\tlearn: 0.4259083\ttotal: 5.83ms\tremaining: 1.74s\n",
"0:\tlearn: 0.4265997\ttotal: 4.33ms\tremaining: 1.29s\n",
"0:\tlearn: 0.6749810\ttotal: 6.07ms\tremaining: 1.81s\n",
"0:\tlearn: 0.6790600\ttotal: 3.77ms\tremaining: 1.13s\n",
"0:\tlearn: 0.6800456\ttotal: 3.15ms\tremaining: 941ms\n",
"0:\tlearn: 0.6561975\ttotal: 2.91ms\tremaining: 869ms\n",
"0:\tlearn: 0.6586196\ttotal: 5.64ms\tremaining: 1.69s\n",
"0:\tlearn: 0.6493304\ttotal: 5.17ms\tremaining: 1.54s\n",
"0:\tlearn: 0.5635160\ttotal: 4.58ms\tremaining: 1.37s\n",
"0:\tlearn: 0.5700935\ttotal: 3.3ms\tremaining: 988ms\n",
"0:\tlearn: 0.5561022\ttotal: 4.83ms\tremaining: 1.44s\n",
"0:\tlearn: 0.4830522\ttotal: 4.95ms\tremaining: 1.48s\n",
"0:\tlearn: 0.4984296\ttotal: 4.87ms\tremaining: 1.45s\n",
"0:\tlearn: 0.5087164\ttotal: 3.27ms\tremaining: 979ms\n",
"0:\tlearn: 0.4673614\ttotal: 4.04ms\tremaining: 1.21s\n",
"0:\tlearn: 0.4683908\ttotal: 4.99ms\tremaining: 1.49s\n",
"0:\tlearn: 0.4692782\ttotal: 4.38ms\tremaining: 1.31s\n",
"0:\tlearn: 0.6807625\ttotal: 2.98ms\tremaining: 892ms\n",
"0:\tlearn: 0.6783109\ttotal: 5.09ms\tremaining: 1.52s\n",
"0:\tlearn: 0.6785684\ttotal: 2.5ms\tremaining: 749ms\n",
"0:\tlearn: 0.6492098\ttotal: 7.3ms\tremaining: 2.18s\n",
"0:\tlearn: 0.6480563\ttotal: 4.79ms\tremaining: 1.43s\n",
"0:\tlearn: 0.6529972\ttotal: 5.12ms\tremaining: 1.53s\n",
"0:\tlearn: 0.5713224\ttotal: 4.43ms\tremaining: 1.32s\n",
"0:\tlearn: 0.5772736\ttotal: 3.18ms\tremaining: 952ms\n",
"0:\tlearn: 0.5609679\ttotal: 4.46ms\tremaining: 1.33s\n",
"0:\tlearn: 0.5024992\ttotal: 2.52ms\tremaining: 755ms\n",
"0:\tlearn: 0.5236048\ttotal: 6.26ms\tremaining: 1.87s\n",
"0:\tlearn: 0.5116883\ttotal: 4.6ms\tremaining: 1.38s\n",
"0:\tlearn: 0.4809106\ttotal: 5.72ms\tremaining: 1.71s\n",
"0:\tlearn: 0.4547855\ttotal: 3.58ms\tremaining: 1.07s\n",
"0:\tlearn: 0.4836719\ttotal: 4.78ms\tremaining: 1.43s\n",
"0:\tlearn: 0.6812309\ttotal: 4.15ms\tremaining: 1.24s\n",
"0:\tlearn: 0.6787945\ttotal: 2.81ms\tremaining: 840ms\n",
"0:\tlearn: 0.6796612\ttotal: 5.79ms\tremaining: 1.73s\n",
"0:\tlearn: 0.6534378\ttotal: 4.72ms\tremaining: 1.41s\n",
"0:\tlearn: 0.6541970\ttotal: 3ms\tremaining: 897ms\n",
"0:\tlearn: 0.6498622\ttotal: 2.78ms\tremaining: 831ms\n",
"0:\tlearn: 0.5668001\ttotal: 3.43ms\tremaining: 1.02s\n",
"0:\tlearn: 0.5597445\ttotal: 2.59ms\tremaining: 775ms\n",
"0:\tlearn: 0.5769490\ttotal: 5.29ms\tremaining: 1.58s\n",
"0:\tlearn: 0.5350513\ttotal: 4.5ms\tremaining: 1.34s\n",
"0:\tlearn: 0.5142208\ttotal: 4.68ms\tremaining: 1.4s\n",
"0:\tlearn: 0.5148426\ttotal: 5.28ms\tremaining: 1.58s\n",
"0:\tlearn: 0.4462993\ttotal: 5.28ms\tremaining: 1.58s\n",
"0:\tlearn: 0.4857340\ttotal: 4.93ms\tremaining: 1.47s\n",
"0:\tlearn: 0.4819538\ttotal: 3.05ms\tremaining: 913ms\n",
"0:\tlearn: 0.6790030\ttotal: 4.18ms\tremaining: 1.25s\n",
"0:\tlearn: 0.6815175\ttotal: 4.78ms\tremaining: 1.43s\n",
"0:\tlearn: 0.6785220\ttotal: 3.67ms\tremaining: 1.09s\n",
"0:\tlearn: 0.6528557\ttotal: 4.59ms\tremaining: 1.37s\n",
"0:\tlearn: 0.6548541\ttotal: 6.44ms\tremaining: 1.93s\n",
"0:\tlearn: 0.6598798\ttotal: 5.01ms\tremaining: 1.5s\n",
"0:\tlearn: 0.5774307\ttotal: 4.63ms\tremaining: 1.38s\n",
"0:\tlearn: 0.5668689\ttotal: 4.13ms\tremaining: 1.23s\n",
"0:\tlearn: 0.5781998\ttotal: 6.09ms\tremaining: 1.82s\n",
"0:\tlearn: 0.5142247\ttotal: 3.16ms\tremaining: 946ms\n",
"0:\tlearn: 0.5143508\ttotal: 7.32ms\tremaining: 2.19s\n",
"0:\tlearn: 0.5403051\ttotal: 4.19ms\tremaining: 1.25s\n",
"0:\tlearn: 0.4991611\ttotal: 3.77ms\tremaining: 1.13s\n",
"0:\tlearn: 0.5026274\ttotal: 3.74ms\tremaining: 1.12s\n",
"0:\tlearn: 0.4694354\ttotal: 3.36ms\tremaining: 1s\n",
"0:\tlearn: 0.6751688\ttotal: 5.55ms\tremaining: 1.66s\n",
"0:\tlearn: 0.6768356\ttotal: 5.58ms\tremaining: 1.67s\n",
"0:\tlearn: 0.6781108\ttotal: 5.31ms\tremaining: 1.59s\n",
"0:\tlearn: 0.6472860\ttotal: 5.26ms\tremaining: 1.57s\n",
"0:\tlearn: 0.6510930\ttotal: 5.03ms\tremaining: 1.5s\n",
"0:\tlearn: 0.6373150\ttotal: 4.96ms\tremaining: 1.48s\n",
"0:\tlearn: 0.5472785\ttotal: 7.44ms\tremaining: 2.22s\n",
"0:\tlearn: 0.5470841\ttotal: 6.8ms\tremaining: 2.03s\n",
"0:\tlearn: 0.5564576\ttotal: 5.54ms\tremaining: 1.66s\n",
"0:\tlearn: 0.4574589\ttotal: 3.83ms\tremaining: 1.15s\n",
"0:\tlearn: 0.5068552\ttotal: 6.29ms\tremaining: 1.88s\n",
"0:\tlearn: 0.4698458\ttotal: 6.92ms\tremaining: 2.07s\n",
"0:\tlearn: 0.4582054\ttotal: 9.29ms\tremaining: 2.78s\n",
"0:\tlearn: 0.4170985\ttotal: 6.4ms\tremaining: 1.91s\n",
"0:\tlearn: 0.4430913\ttotal: 5.62ms\tremaining: 1.68s\n",
"0:\tlearn: 0.6801569\ttotal: 6.89ms\tremaining: 2.06s\n",
"0:\tlearn: 0.6801750\ttotal: 5.85ms\tremaining: 1.75s\n",
"0:\tlearn: 0.6793489\ttotal: 7.54ms\tremaining: 2.25s\n",
"0:\tlearn: 0.6451369\ttotal: 4.36ms\tremaining: 1.3s\n",
"0:\tlearn: 0.6530426\ttotal: 7.35ms\tremaining: 2.2s\n",
"0:\tlearn: 0.6466303\ttotal: 4.19ms\tremaining: 1.25s\n",
"0:\tlearn: 0.5653691\ttotal: 4.78ms\tremaining: 1.43s\n",
"0:\tlearn: 0.5677601\ttotal: 6.22ms\tremaining: 1.86s\n",
"0:\tlearn: 0.5618567\ttotal: 4.52ms\tremaining: 1.35s\n",
"0:\tlearn: 0.4829952\ttotal: 6.14ms\tremaining: 1.84s\n",
"0:\tlearn: 0.4819169\ttotal: 7.4ms\tremaining: 2.21s\n",
"0:\tlearn: 0.5061863\ttotal: 5.63ms\tremaining: 1.68s\n",
"0:\tlearn: 0.4419334\ttotal: 6.24ms\tremaining: 1.86s\n",
"0:\tlearn: 0.4700497\ttotal: 9.03ms\tremaining: 2.7s\n",
"0:\tlearn: 0.4660648\ttotal: 6.77ms\tremaining: 2.02s\n",
"0:\tlearn: 0.6788608\ttotal: 3.93ms\tremaining: 1.18s\n",
"0:\tlearn: 0.6795216\ttotal: 7.44ms\tremaining: 2.22s\n",
"0:\tlearn: 0.6790256\ttotal: 5.95ms\tremaining: 1.78s\n",
"0:\tlearn: 0.6536589\ttotal: 7.77ms\tremaining: 2.32s\n",
"0:\tlearn: 0.6558451\ttotal: 5.55ms\tremaining: 1.66s\n",
"0:\tlearn: 0.6492718\ttotal: 6.75ms\tremaining: 2.02s\n",
"0:\tlearn: 0.5636530\ttotal: 7.41ms\tremaining: 2.21s\n",
"0:\tlearn: 0.5627985\ttotal: 10.6ms\tremaining: 3.16s\n",
"0:\tlearn: 0.5665024\ttotal: 7.59ms\tremaining: 2.27s\n",
"0:\tlearn: 0.5126904\ttotal: 7.97ms\tremaining: 2.38s\n",
"0:\tlearn: 0.5128502\ttotal: 7.9ms\tremaining: 2.36s\n",
"0:\tlearn: 0.5070520\ttotal: 6.3ms\tremaining: 1.88s\n",
"0:\tlearn: 0.4321557\ttotal: 5.16ms\tremaining: 1.54s\n",
"0:\tlearn: 0.4899331\ttotal: 5.38ms\tremaining: 1.61s\n",
"0:\tlearn: 0.4464896\ttotal: 6.79ms\tremaining: 2.03s\n",
"0:\tlearn: 0.6791433\ttotal: 4.08ms\tremaining: 1.22s\n",
"0:\tlearn: 0.6805458\ttotal: 5.56ms\tremaining: 1.66s\n",
"0:\tlearn: 0.6786656\ttotal: 8.62ms\tremaining: 2.58s\n",
"0:\tlearn: 0.6519203\ttotal: 5.95ms\tremaining: 1.78s\n",
"0:\tlearn: 0.6537454\ttotal: 5.17ms\tremaining: 1.54s\n",
"0:\tlearn: 0.6540700\ttotal: 7.41ms\tremaining: 2.21s\n",
"0:\tlearn: 0.5581209\ttotal: 5.27ms\tremaining: 1.57s\n",
"0:\tlearn: 0.5631094\ttotal: 8.7ms\tremaining: 2.6s\n",
"0:\tlearn: 0.5831950\ttotal: 6.31ms\tremaining: 1.89s\n",
"0:\tlearn: 0.5063605\ttotal: 7.37ms\tremaining: 2.21s\n",
"0:\tlearn: 0.5228038\ttotal: 6.5ms\tremaining: 1.94s\n",
"0:\tlearn: 0.5241394\ttotal: 6.79ms\tremaining: 2.03s\n",
"0:\tlearn: 0.4518047\ttotal: 5.77ms\tremaining: 1.73s\n",
"0:\tlearn: 0.4909012\ttotal: 4.94ms\tremaining: 1.48s\n",
"0:\tlearn: 0.4568951\ttotal: 5.66ms\tremaining: 1.69s\n",
"0:\tlearn: 0.6809809\ttotal: 4.14ms\tremaining: 1.24s\n",
"0:\tlearn: 0.6804813\ttotal: 7.35ms\tremaining: 2.2s\n",
"0:\tlearn: 0.6801523\ttotal: 7.47ms\tremaining: 2.23s\n",
"0:\tlearn: 0.6559878\ttotal: 5.33ms\tremaining: 1.59s\n",
"0:\tlearn: 0.6632372\ttotal: 5.29ms\tremaining: 1.58s\n",
"0:\tlearn: 0.6543028\ttotal: 6.57ms\tremaining: 1.96s\n",
"0:\tlearn: 0.5685541\ttotal: 6.56ms\tremaining: 1.96s\n",
"0:\tlearn: 0.5777365\ttotal: 7.9ms\tremaining: 2.36s\n",
"0:\tlearn: 0.5726313\ttotal: 4.86ms\tremaining: 1.45s\n",
"0:\tlearn: 0.5218100\ttotal: 6.29ms\tremaining: 1.88s\n",
"0:\tlearn: 0.5213705\ttotal: 4.8ms\tremaining: 1.44s\n",
"0:\tlearn: 0.5433809\ttotal: 7.36ms\tremaining: 2.2s\n",
"0:\tlearn: 0.4780379\ttotal: 5.88ms\tremaining: 1.76s\n",
"0:\tlearn: 0.4772615\ttotal: 4.07ms\tremaining: 1.22s\n",
"0:\tlearn: 0.4759924\ttotal: 7.23ms\tremaining: 2.16s\n",
"0:\tlearn: 0.6753106\ttotal: 12.4ms\tremaining: 3.71s\n",
"0:\tlearn: 0.6765844\ttotal: 13.2ms\tremaining: 3.94s\n",
"0:\tlearn: 0.6773752\ttotal: 9.67ms\tremaining: 2.89s\n",
"0:\tlearn: 0.6440750\ttotal: 14.3ms\tremaining: 4.27s\n",
"0:\tlearn: 0.6436587\ttotal: 15.5ms\tremaining: 4.64s\n",
"0:\tlearn: 0.6435990\ttotal: 15.5ms\tremaining: 4.65s\n",
"0:\tlearn: 0.5368855\ttotal: 10.2ms\tremaining: 3.04s\n",
"0:\tlearn: 0.5629497\ttotal: 11.8ms\tremaining: 3.52s\n",
"0:\tlearn: 0.5383464\ttotal: 10.9ms\tremaining: 3.27s\n",
"0:\tlearn: 0.4660597\ttotal: 13.3ms\tremaining: 3.97s\n",
"0:\tlearn: 0.4785735\ttotal: 12.3ms\tremaining: 3.67s\n",
"0:\tlearn: 0.4806738\ttotal: 12ms\tremaining: 3.6s\n",
"0:\tlearn: 0.4301159\ttotal: 13.6ms\tremaining: 4.07s\n",
"0:\tlearn: 0.4409287\ttotal: 11.6ms\tremaining: 3.48s\n",
"0:\tlearn: 0.4023186\ttotal: 12ms\tremaining: 3.59s\n",
"0:\tlearn: 0.6762833\ttotal: 11ms\tremaining: 3.3s\n",
"0:\tlearn: 0.6783536\ttotal: 10.6ms\tremaining: 3.16s\n",
"0:\tlearn: 0.6802052\ttotal: 13.3ms\tremaining: 3.99s\n",
"0:\tlearn: 0.6470541\ttotal: 12.1ms\tremaining: 3.62s\n",
"0:\tlearn: 0.6511499\ttotal: 10.1ms\tremaining: 3.03s\n",
"0:\tlearn: 0.6500411\ttotal: 12.7ms\tremaining: 3.79s\n",
"0:\tlearn: 0.5623595\ttotal: 11.6ms\tremaining: 3.48s\n",
"0:\tlearn: 0.5650627\ttotal: 10.6ms\tremaining: 3.15s\n",
"0:\tlearn: 0.5600656\ttotal: 15.8ms\tremaining: 4.74s\n",
"0:\tlearn: 0.5016617\ttotal: 14.8ms\tremaining: 4.41s\n",
"0:\tlearn: 0.4934577\ttotal: 11.8ms\tremaining: 3.53s\n",
"0:\tlearn: 0.5177063\ttotal: 13.1ms\tremaining: 3.9s\n",
"0:\tlearn: 0.4478103\ttotal: 13.8ms\tremaining: 4.14s\n",
"0:\tlearn: 0.4742268\ttotal: 13.1ms\tremaining: 3.92s\n",
"0:\tlearn: 0.4608241\ttotal: 9.53ms\tremaining: 2.85s\n",
"0:\tlearn: 0.6794672\ttotal: 12.2ms\tremaining: 3.65s\n",
"0:\tlearn: 0.6806254\ttotal: 11.2ms\tremaining: 3.36s\n",
"0:\tlearn: 0.6793541\ttotal: 11.2ms\tremaining: 3.34s\n",
"0:\tlearn: 0.6537638\ttotal: 13.1ms\tremaining: 3.92s\n",
"0:\tlearn: 0.6580973\ttotal: 8.95ms\tremaining: 2.67s\n",
"0:\tlearn: 0.6549324\ttotal: 11ms\tremaining: 3.28s\n",
"0:\tlearn: 0.5594035\ttotal: 11.3ms\tremaining: 3.39s\n",
"0:\tlearn: 0.5739827\ttotal: 12.4ms\tremaining: 3.7s\n",
"0:\tlearn: 0.5689560\ttotal: 10.7ms\tremaining: 3.2s\n",
"0:\tlearn: 0.5417376\ttotal: 13.9ms\tremaining: 4.16s\n",
"0:\tlearn: 0.5166053\ttotal: 13.6ms\tremaining: 4.06s\n",
"0:\tlearn: 0.5220027\ttotal: 13.1ms\tremaining: 3.92s\n",
"0:\tlearn: 0.4706977\ttotal: 13.4ms\tremaining: 4s\n",
"0:\tlearn: 0.4903132\ttotal: 9.3ms\tremaining: 2.78s\n",
"0:\tlearn: 0.4932043\ttotal: 18ms\tremaining: 5.39s\n",
"0:\tlearn: 0.6789853\ttotal: 15.1ms\tremaining: 4.53s\n",
"0:\tlearn: 0.6820307\ttotal: 15.5ms\tremaining: 4.65s\n",
"0:\tlearn: 0.6797196\ttotal: 13.7ms\tremaining: 4.1s\n",
"0:\tlearn: 0.6527218\ttotal: 15ms\tremaining: 4.48s\n",
"0:\tlearn: 0.6564538\ttotal: 14.9ms\tremaining: 4.45s\n",
"0:\tlearn: 0.6564694\ttotal: 9.8ms\tremaining: 2.93s\n",
"0:\tlearn: 0.5678658\ttotal: 12.7ms\tremaining: 3.79s\n",
"0:\tlearn: 0.5787331\ttotal: 10.4ms\tremaining: 3.1s\n",
"0:\tlearn: 0.5843310\ttotal: 13.9ms\tremaining: 4.15s\n",
"0:\tlearn: 0.5091080\ttotal: 11ms\tremaining: 3.28s\n",
"0:\tlearn: 0.5445095\ttotal: 14.3ms\tremaining: 4.27s\n",
"0:\tlearn: 0.5083195\ttotal: 12.7ms\tremaining: 3.79s\n",
"0:\tlearn: 0.4879522\ttotal: 14ms\tremaining: 4.17s\n",
"0:\tlearn: 0.4933299\ttotal: 11.6ms\tremaining: 3.48s\n",
"0:\tlearn: 0.4994554\ttotal: 11.6ms\tremaining: 3.48s\n",
"0:\tlearn: 0.6815244\ttotal: 12.5ms\tremaining: 3.75s\n",
"0:\tlearn: 0.6791065\ttotal: 9.89ms\tremaining: 2.96s\n",
"0:\tlearn: 0.6818853\ttotal: 16.3ms\tremaining: 4.86s\n",
"0:\tlearn: 0.6587271\ttotal: 13.6ms\tremaining: 4.06s\n",
"0:\tlearn: 0.6539367\ttotal: 13.7ms\tremaining: 4.1s\n",
"0:\tlearn: 0.6542301\ttotal: 13.4ms\tremaining: 4.01s\n",
"0:\tlearn: 0.5768087\ttotal: 9.09ms\tremaining: 2.72s\n",
"0:\tlearn: 0.5904323\ttotal: 14.4ms\tremaining: 4.31s\n",
"0:\tlearn: 0.5760325\ttotal: 16.1ms\tremaining: 4.83s\n",
"0:\tlearn: 0.5269511\ttotal: 12.4ms\tremaining: 3.71s\n",
"0:\tlearn: 0.5182570\ttotal: 12.3ms\tremaining: 3.68s\n",
"0:\tlearn: 0.5323913\ttotal: 15.7ms\tremaining: 4.68s\n",
"0:\tlearn: 0.4918639\ttotal: 15.9ms\tremaining: 4.75s\n",
"0:\tlearn: 0.4952882\ttotal: 9.28ms\tremaining: 2.77s\n",
"0:\tlearn: 0.4653270\ttotal: 15.1ms\tremaining: 4.52s\n",
"0:\tlearn: 0.6741232\ttotal: 37.9ms\tremaining: 11.3s\n",
"0:\tlearn: 0.6780967\ttotal: 36.1ms\tremaining: 10.8s\n",
"0:\tlearn: 0.6754606\ttotal: 44.9ms\tremaining: 13.4s\n",
"0:\tlearn: 0.6412782\ttotal: 35.5ms\tremaining: 10.6s\n",
"0:\tlearn: 0.6479846\ttotal: 44.1ms\tremaining: 13.2s\n",
"0:\tlearn: 0.6450649\ttotal: 37.9ms\tremaining: 11.3s\n",
"0:\tlearn: 0.5307034\ttotal: 38.3ms\tremaining: 11.4s\n",
"0:\tlearn: 0.5526915\ttotal: 39.1ms\tremaining: 11.7s\n",
"0:\tlearn: 0.5282242\ttotal: 34.6ms\tremaining: 10.3s\n",
"0:\tlearn: 0.4926921\ttotal: 31.9ms\tremaining: 9.54s\n",
"0:\tlearn: 0.4828428\ttotal: 33.3ms\tremaining: 9.96s\n",
"0:\tlearn: 0.4600306\ttotal: 34.1ms\tremaining: 10.2s\n",
"0:\tlearn: 0.4357598\ttotal: 32.8ms\tremaining: 9.81s\n",
"0:\tlearn: 0.4552467\ttotal: 39.8ms\tremaining: 11.9s\n",
"0:\tlearn: 0.4285548\ttotal: 30.5ms\tremaining: 9.13s\n",
"0:\tlearn: 0.6785309\ttotal: 33.1ms\tremaining: 9.9s\n",
"0:\tlearn: 0.6788459\ttotal: 43.7ms\tremaining: 13.1s\n",
"0:\tlearn: 0.6809102\ttotal: 39.5ms\tremaining: 11.8s\n",
"0:\tlearn: 0.6489408\ttotal: 43.6ms\tremaining: 13s\n",
"0:\tlearn: 0.6559222\ttotal: 39.1ms\tremaining: 11.7s\n",
"0:\tlearn: 0.6512155\ttotal: 42.2ms\tremaining: 12.6s\n",
"0:\tlearn: 0.5559243\ttotal: 45.4ms\tremaining: 13.6s\n",
"0:\tlearn: 0.5583208\ttotal: 34ms\tremaining: 10.2s\n",
"0:\tlearn: 0.5600832\ttotal: 34.9ms\tremaining: 10.4s\n",
"0:\tlearn: 0.4840771\ttotal: 32.6ms\tremaining: 9.74s\n",
"0:\tlearn: 0.5206319\ttotal: 47.2ms\tremaining: 14.1s\n",
"0:\tlearn: 0.4982290\ttotal: 32.6ms\tremaining: 9.76s\n",
"0:\tlearn: 0.4621061\ttotal: 33.8ms\tremaining: 10.1s\n",
"0:\tlearn: 0.4815773\ttotal: 42.1ms\tremaining: 12.6s\n",
"0:\tlearn: 0.4659351\ttotal: 37ms\tremaining: 11.1s\n",
"0:\tlearn: 0.6799076\ttotal: 37.5ms\tremaining: 11.2s\n",
"0:\tlearn: 0.6807307\ttotal: 35.4ms\tremaining: 10.6s\n",
"0:\tlearn: 0.6810847\ttotal: 40.3ms\tremaining: 12s\n",
"0:\tlearn: 0.6487218\ttotal: 29.8ms\tremaining: 8.91s\n",
"0:\tlearn: 0.6536955\ttotal: 39.4ms\tremaining: 11.8s\n",
"0:\tlearn: 0.6539244\ttotal: 33.5ms\tremaining: 10s\n",
"0:\tlearn: 0.5723258\ttotal: 35.1ms\tremaining: 10.5s\n",
"0:\tlearn: 0.5868016\ttotal: 46.1ms\tremaining: 13.8s\n",
"0:\tlearn: 0.5665293\ttotal: 35ms\tremaining: 10.5s\n",
"0:\tlearn: 0.5001921\ttotal: 33.2ms\tremaining: 9.94s\n",
"0:\tlearn: 0.5323607\ttotal: 47.7ms\tremaining: 14.3s\n",
"0:\tlearn: 0.5266318\ttotal: 33.4ms\tremaining: 9.99s\n",
"0:\tlearn: 0.4607401\ttotal: 46.5ms\tremaining: 13.9s\n",
"0:\tlearn: 0.4707496\ttotal: 39.8ms\tremaining: 11.9s\n",
"0:\tlearn: 0.4584523\ttotal: 37.5ms\tremaining: 11.2s\n",
"0:\tlearn: 0.6806191\ttotal: 44.9ms\tremaining: 13.4s\n",
"0:\tlearn: 0.6825254\ttotal: 45.8ms\tremaining: 13.7s\n",
"0:\tlearn: 0.6802844\ttotal: 33.5ms\tremaining: 10s\n",
"0:\tlearn: 0.6569362\ttotal: 32ms\tremaining: 9.56s\n",
"0:\tlearn: 0.6576371\ttotal: 59.2ms\tremaining: 17.7s\n",
"0:\tlearn: 0.6574163\ttotal: 41.9ms\tremaining: 12.5s\n",
"0:\tlearn: 0.5842646\ttotal: 33ms\tremaining: 9.86s\n",
"0:\tlearn: 0.5785312\ttotal: 44.5ms\tremaining: 13.3s\n",
"0:\tlearn: 0.5887974\ttotal: 34.5ms\tremaining: 10.3s\n",
"0:\tlearn: 0.5237318\ttotal: 34.5ms\tremaining: 10.3s\n",
"0:\tlearn: 0.5449079\ttotal: 30.4ms\tremaining: 9.1s\n",
"0:\tlearn: 0.5246303\ttotal: 35ms\tremaining: 10.5s\n",
"0:\tlearn: 0.5070652\ttotal: 38.1ms\tremaining: 11.4s\n",
"0:\tlearn: 0.5062342\ttotal: 40.6ms\tremaining: 12.1s\n",
"0:\tlearn: 0.4849864\ttotal: 37.4ms\tremaining: 11.2s\n",
"0:\tlearn: 0.6813369\ttotal: 41.9ms\tremaining: 12.5s\n",
"0:\tlearn: 0.6803472\ttotal: 34.8ms\tremaining: 10.4s\n",
"0:\tlearn: 0.6811404\ttotal: 34.9ms\tremaining: 10.4s\n",
"0:\tlearn: 0.6581069\ttotal: 44.9ms\tremaining: 13.4s\n",
"0:\tlearn: 0.6627382\ttotal: 35.9ms\tremaining: 10.7s\n",
"0:\tlearn: 0.6642628\ttotal: 34.4ms\tremaining: 10.3s\n",
"0:\tlearn: 0.5886341\ttotal: 34ms\tremaining: 10.2s\n",
"0:\tlearn: 0.5833613\ttotal: 19.5ms\tremaining: 5.84s\n",
"0:\tlearn: 0.5840337\ttotal: 36.1ms\tremaining: 10.8s\n",
"0:\tlearn: 0.5281502\ttotal: 43ms\tremaining: 12.8s\n",
"0:\tlearn: 0.5400897\ttotal: 34.4ms\tremaining: 10.3s\n",
"0:\tlearn: 0.5415765\ttotal: 40.4ms\tremaining: 12.1s\n",
"0:\tlearn: 0.4894145\ttotal: 38.8ms\tremaining: 11.6s\n",
"0:\tlearn: 0.5044201\ttotal: 40.4ms\tremaining: 12.1s\n",
"0:\tlearn: 0.4858793\ttotal: 34.6ms\tremaining: 10.3s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=1)]: Done 375 out of 375 | elapsed: 23.1min finished\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"0:\tlearn: 0.6784282\ttotal: 15.9ms\tremaining: 4.76s\n",
"Computation time (parameter tunning): 1387 seconds...\n",
"Best parameters: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 5, 'learning_rate': 0.01}\n",
"0:\tlearn: 0.9584200\ttotal: 10.4ms\tremaining: 3.1s\n",
"30:\tlearn: 0.9907435\ttotal: 317ms\tremaining: 2.75s\n",
"60:\tlearn: 0.9912389\ttotal: 639ms\tremaining: 2.5s\n",
"90:\tlearn: 0.9919738\ttotal: 942ms\tremaining: 2.16s\n",
"120:\tlearn: 0.9928656\ttotal: 1.23s\tremaining: 1.82s\n",
"150:\tlearn: 0.9937409\ttotal: 1.54s\tremaining: 1.52s\n",
"180:\tlearn: 0.9949217\ttotal: 1.84s\tremaining: 1.21s\n",
"210:\tlearn: 0.9957970\ttotal: 2.14s\tremaining: 901ms\n",
"240:\tlearn: 0.9964989\ttotal: 2.44s\tremaining: 597ms\n",
"270:\tlearn: 0.9971842\ttotal: 2.76s\tremaining: 295ms\n",
"********** SUMMARY: XGBoost (Exp6-Undersampled) **************\n",
"- Parameters used: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 5, 'learning_rate': 0.01}\n",
"- AUC(Train|Test): (0.99749801823226314, 0.98262419384882727)\n",
"- Metrics: Train data (n=696) **\n",
" TN=346, FP=2\n",
" FN=20, TP=328\n",
" Precision: 0.99\n",
" Recall(TPR): 0.94\n",
" BAcc=(TPR+TNR)/2: 0.97\n",
"- Metrics: Test data (n=85443) **\n",
" TN=83452, FP=1847\n",
" FN=14, TP=130\n",
" Precision: 0.07\n",
" Recall(TPR): 0.90\n",
" BAcc=(TPR+TNR)/2: 0.94\n",
"Computation time (Total): 4 seconds\n",
"Using parameters: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 5, 'learning_rate': 0.01}\n",
"0:\tlearn: 0.9989367\ttotal: 141ms\tremaining: 42s\n",
"30:\tlearn: 0.9998128\ttotal: 4.74s\tremaining: 41.1s\n",
"60:\tlearn: 0.9998139\ttotal: 9.56s\tremaining: 37.5s\n",
"90:\tlearn: 0.9998439\ttotal: 14.4s\tremaining: 33.1s\n",
"120:\tlearn: 0.9998609\ttotal: 19.3s\tremaining: 28.6s\n",
"150:\tlearn: 0.9998736\ttotal: 24.4s\tremaining: 24s\n",
"180:\tlearn: 0.9998828\ttotal: 29.2s\tremaining: 19.2s\n",
"210:\tlearn: 0.9998955\ttotal: 34.2s\tremaining: 14.4s\n",
"240:\tlearn: 0.9999014\ttotal: 39.3s\tremaining: 9.62s\n",
"270:\tlearn: 0.9999076\ttotal: 44.4s\tremaining: 4.75s\n",
"********** SUMMARY: XGBoost (Exp6-SMOTE2 (best parameters from Undersampled)) **************\n",
"- Parameters used: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 5, 'learning_rate': 0.01}\n",
"- AUC(Train|Test): (0.99991711073861844, 0.97858574445968505)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=198864, FP=152\n",
" FN=269, TP=198747\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85224, FP=75\n",
" FN=22, TP=122\n",
" Precision: 0.62\n",
" Recall(TPR): 0.85\n",
" BAcc=(TPR+TNR)/2: 0.92\n",
"Computation time (Total): 54 seconds\n",
"\n",
"\n",
"\n",
"Selected feats: Index(['Time', 'V1', 'V2', 'V3', 'V4', 'V5', 'V7', 'V8', 'V10', 'V11', 'V12',\n",
" 'V14', 'V16', 'V17', 'Amount'],\n",
" dtype='object')\n",
"Fitting 3 folds for each of 125 candidates, totalling 375 fits\n",
"0:\tlearn: 0.6811737\ttotal: 5.03ms\tremaining: 1.5s\n",
"0:\tlearn: 0.6781727\ttotal: 3.62ms\tremaining: 1.08s\n",
"0:\tlearn: 0.6795889\ttotal: 4.28ms\tremaining: 1.28s\n",
"0:\tlearn: 0.6605948\ttotal: 4.22ms\tremaining: 1.26s\n",
"0:\tlearn: 0.6579221\ttotal: 3.47ms\tremaining: 1.04s\n",
"0:\tlearn: 0.6582529\ttotal: 3.84ms\tremaining: 1.15s\n",
"0:\tlearn: 0.5757370\ttotal: 4.91ms\tremaining: 1.47s\n",
"0:\tlearn: 0.5849017\ttotal: 2.96ms\tremaining: 887ms\n",
"0:\tlearn: 0.5651304\ttotal: 4.57ms\tremaining: 1.36s\n",
"0:\tlearn: 0.4857447\ttotal: 4.63ms\tremaining: 1.38s\n",
"0:\tlearn: 0.4887973\ttotal: 5.26ms\tremaining: 1.57s\n",
"0:\tlearn: 0.4816090\ttotal: 3.72ms\tremaining: 1.11s\n",
"0:\tlearn: 0.4847416\ttotal: 3.58ms\tremaining: 1.07s\n",
"0:\tlearn: 0.4960911\ttotal: 4.92ms\tremaining: 1.47s\n",
"0:\tlearn: 0.4907393\ttotal: 6.84ms\tremaining: 2.04s\n",
"0:\tlearn: 0.6785571\ttotal: 3.11ms\tremaining: 930ms\n",
"0:\tlearn: 0.6829771\ttotal: 2.92ms\tremaining: 872ms\n",
"0:\tlearn: 0.6765936\ttotal: 7.63ms\tremaining: 2.28s\n",
"0:\tlearn: 0.6520815\ttotal: 3.67ms\tremaining: 1.1s\n",
"0:\tlearn: 0.6532371\ttotal: 4.54ms\tremaining: 1.36s\n",
"0:\tlearn: 0.6590196\ttotal: 3.61ms\tremaining: 1.08s\n",
"0:\tlearn: 0.5864782\ttotal: 4.41ms\tremaining: 1.32s\n",
"0:\tlearn: 0.5845312\ttotal: 4.57ms\tremaining: 1.37s\n",
"0:\tlearn: 0.5455172\ttotal: 2.92ms\tremaining: 873ms\n",
"0:\tlearn: 0.5309490\ttotal: 4.01ms\tremaining: 1.2s\n",
"0:\tlearn: 0.5254868\ttotal: 4.28ms\tremaining: 1.28s\n",
"0:\tlearn: 0.5265885\ttotal: 4.34ms\tremaining: 1.3s\n",
"0:\tlearn: 0.4607577\ttotal: 4.08ms\tremaining: 1.22s\n",
"0:\tlearn: 0.4730144\ttotal: 3.6ms\tremaining: 1.07s\n",
"0:\tlearn: 0.4822948\ttotal: 3.34ms\tremaining: 998ms\n",
"0:\tlearn: 0.6819179\ttotal: 4.27ms\tremaining: 1.28s\n",
"0:\tlearn: 0.6814178\ttotal: 1.83ms\tremaining: 548ms\n",
"0:\tlearn: 0.6797171\ttotal: 4.33ms\tremaining: 1.29s\n",
"0:\tlearn: 0.6607184\ttotal: 5.33ms\tremaining: 1.59s\n",
"0:\tlearn: 0.6567587\ttotal: 2.08ms\tremaining: 621ms\n",
"0:\tlearn: 0.6537688\ttotal: 4.95ms\tremaining: 1.48s\n",
"0:\tlearn: 0.5473731\ttotal: 1.88ms\tremaining: 562ms\n",
"0:\tlearn: 0.5920758\ttotal: 3.36ms\tremaining: 1s\n",
"0:\tlearn: 0.5647569\ttotal: 4.61ms\tremaining: 1.38s\n",
"0:\tlearn: 0.5201905\ttotal: 6.24ms\tremaining: 1.87s\n",
"0:\tlearn: 0.5052682\ttotal: 3.06ms\tremaining: 914ms\n",
"0:\tlearn: 0.5273659\ttotal: 3.83ms\tremaining: 1.14s\n",
"0:\tlearn: 0.4679016\ttotal: 2.74ms\tremaining: 818ms\n",
"0:\tlearn: 0.4854989\ttotal: 2.59ms\tremaining: 775ms\n",
"0:\tlearn: 0.5041880\ttotal: 4.23ms\tremaining: 1.26s\n",
"0:\tlearn: 0.6795468\ttotal: 1.81ms\tremaining: 541ms\n",
"0:\tlearn: 0.6800887\ttotal: 4.31ms\tremaining: 1.29s\n",
"0:\tlearn: 0.6819712\ttotal: 2.91ms\tremaining: 870ms\n",
"0:\tlearn: 0.6593779\ttotal: 1.96ms\tremaining: 585ms\n",
"0:\tlearn: 0.6527492\ttotal: 4.32ms\tremaining: 1.29s\n",
"0:\tlearn: 0.6557601\ttotal: 3.71ms\tremaining: 1.11s\n",
"0:\tlearn: 0.5739257\ttotal: 2.51ms\tremaining: 751ms\n",
"0:\tlearn: 0.5995242\ttotal: 2.49ms\tremaining: 743ms\n",
"0:\tlearn: 0.5771958\ttotal: 5.91ms\tremaining: 1.77s\n",
"0:\tlearn: 0.5189495\ttotal: 3.38ms\tremaining: 1.01s\n",
"0:\tlearn: 0.5058536\ttotal: 3.01ms\tremaining: 899ms\n",
"0:\tlearn: 0.5186911\ttotal: 4.13ms\tremaining: 1.24s\n",
"0:\tlearn: 0.5118129\ttotal: 3.85ms\tremaining: 1.15s\n",
"0:\tlearn: 0.4788255\ttotal: 1.96ms\tremaining: 587ms\n",
"0:\tlearn: 0.5030879\ttotal: 2.5ms\tremaining: 748ms\n",
"0:\tlearn: 0.6811733\ttotal: 1.85ms\tremaining: 552ms\n",
"0:\tlearn: 0.6798400\ttotal: 5.34ms\tremaining: 1.59s\n",
"0:\tlearn: 0.6805461\ttotal: 3.05ms\tremaining: 911ms\n",
"0:\tlearn: 0.6564644\ttotal: 4.22ms\tremaining: 1.26s\n",
"0:\tlearn: 0.6554412\ttotal: 4.36ms\tremaining: 1.3s\n",
"0:\tlearn: 0.6553212\ttotal: 2.17ms\tremaining: 649ms\n",
"0:\tlearn: 0.5839658\ttotal: 5.21ms\tremaining: 1.56s\n",
"0:\tlearn: 0.5839877\ttotal: 2.99ms\tremaining: 893ms\n",
"0:\tlearn: 0.5842528\ttotal: 4.58ms\tremaining: 1.37s\n",
"0:\tlearn: 0.5300295\ttotal: 3.09ms\tremaining: 924ms\n",
"0:\tlearn: 0.5159890\ttotal: 3.59ms\tremaining: 1.07s\n",
"0:\tlearn: 0.5518013\ttotal: 2.61ms\tremaining: 782ms\n",
"0:\tlearn: 0.4946356\ttotal: 3.18ms\tremaining: 952ms\n",
"0:\tlearn: 0.4965874\ttotal: 4.56ms\tremaining: 1.36s\n",
"0:\tlearn: 0.4801556\ttotal: 2.94ms\tremaining: 878ms\n",
"0:\tlearn: 0.6762353\ttotal: 4.13ms\tremaining: 1.24s\n",
"0:\tlearn: 0.6769167\ttotal: 3.67ms\tremaining: 1.1s\n",
"0:\tlearn: 0.6803318\ttotal: 5.67ms\tremaining: 1.7s\n",
"0:\tlearn: 0.6368041\ttotal: 5.72ms\tremaining: 1.71s\n",
"0:\tlearn: 0.6442553\ttotal: 5.09ms\tremaining: 1.52s\n",
"0:\tlearn: 0.6399545\ttotal: 3.59ms\tremaining: 1.07s\n",
"0:\tlearn: 0.5491263\ttotal: 4.72ms\tremaining: 1.41s\n",
"0:\tlearn: 0.5608892\ttotal: 5.83ms\tremaining: 1.74s\n",
"0:\tlearn: 0.5420116\ttotal: 4.37ms\tremaining: 1.31s\n",
"0:\tlearn: 0.4799198\ttotal: 3.81ms\tremaining: 1.14s\n",
"0:\tlearn: 0.4902238\ttotal: 5.19ms\tremaining: 1.55s\n",
"0:\tlearn: 0.4874746\ttotal: 4.27ms\tremaining: 1.28s\n",
"0:\tlearn: 0.4402565\ttotal: 5.53ms\tremaining: 1.65s\n",
"0:\tlearn: 0.4185764\ttotal: 5.36ms\tremaining: 1.6s\n",
"0:\tlearn: 0.4495436\ttotal: 7.57ms\tremaining: 2.26s\n",
"0:\tlearn: 0.6792367\ttotal: 3.2ms\tremaining: 957ms\n",
"0:\tlearn: 0.6792421\ttotal: 2.6ms\tremaining: 779ms\n",
"0:\tlearn: 0.6779895\ttotal: 3.42ms\tremaining: 1.02s\n",
"0:\tlearn: 0.6494206\ttotal: 4.79ms\tremaining: 1.43s\n",
"0:\tlearn: 0.6489510\ttotal: 6.78ms\tremaining: 2.03s\n",
"0:\tlearn: 0.6539344\ttotal: 7.72ms\tremaining: 2.31s\n",
"0:\tlearn: 0.5366223\ttotal: 2.82ms\tremaining: 844ms\n",
"0:\tlearn: 0.5534076\ttotal: 3.48ms\tremaining: 1.04s\n",
"0:\tlearn: 0.5475824\ttotal: 6.32ms\tremaining: 1.89s\n",
"0:\tlearn: 0.5011835\ttotal: 2.77ms\tremaining: 828ms\n",
"0:\tlearn: 0.4980559\ttotal: 4.02ms\tremaining: 1.2s\n",
"0:\tlearn: 0.4926420\ttotal: 4.07ms\tremaining: 1.22s\n",
"0:\tlearn: 0.4575217\ttotal: 5.95ms\tremaining: 1.78s\n",
"0:\tlearn: 0.4492829\ttotal: 5.98ms\tremaining: 1.79s\n",
"0:\tlearn: 0.4964578\ttotal: 3.71ms\tremaining: 1.11s\n",
"0:\tlearn: 0.6812170\ttotal: 3.21ms\tremaining: 961ms\n",
"0:\tlearn: 0.6803928\ttotal: 3.25ms\tremaining: 973ms\n",
"0:\tlearn: 0.6785636\ttotal: 6.22ms\tremaining: 1.86s\n",
"0:\tlearn: 0.6473526\ttotal: 4.76ms\tremaining: 1.42s\n",
"0:\tlearn: 0.6516992\ttotal: 4.38ms\tremaining: 1.31s\n",
"0:\tlearn: 0.6483996\ttotal: 3.11ms\tremaining: 930ms\n",
"0:\tlearn: 0.5549852\ttotal: 5.16ms\tremaining: 1.54s\n",
"0:\tlearn: 0.5677647\ttotal: 5.49ms\tremaining: 1.64s\n",
"0:\tlearn: 0.5621337\ttotal: 8.58ms\tremaining: 2.56s\n",
"0:\tlearn: 0.4938191\ttotal: 4.56ms\tremaining: 1.36s\n",
"0:\tlearn: 0.5243246\ttotal: 4.27ms\tremaining: 1.28s\n",
"0:\tlearn: 0.5305228\ttotal: 5.65ms\tremaining: 1.69s\n",
"0:\tlearn: 0.4849977\ttotal: 4.54ms\tremaining: 1.36s\n",
"0:\tlearn: 0.4812688\ttotal: 2.45ms\tremaining: 733ms\n",
"0:\tlearn: 0.4932915\ttotal: 3.72ms\tremaining: 1.11s\n",
"0:\tlearn: 0.6770008\ttotal: 4.51ms\tremaining: 1.35s\n",
"0:\tlearn: 0.6784012\ttotal: 3.99ms\tremaining: 1.19s\n",
"0:\tlearn: 0.6798355\ttotal: 5.63ms\tremaining: 1.68s\n",
"0:\tlearn: 0.6522712\ttotal: 5.3ms\tremaining: 1.58s\n",
"0:\tlearn: 0.6573517\ttotal: 3.03ms\tremaining: 907ms\n",
"0:\tlearn: 0.6501602\ttotal: 2.92ms\tremaining: 874ms\n",
"0:\tlearn: 0.5624186\ttotal: 5.96ms\tremaining: 1.78s\n",
"0:\tlearn: 0.5656457\ttotal: 3.9ms\tremaining: 1.17s\n",
"0:\tlearn: 0.5668067\ttotal: 3.35ms\tremaining: 1s\n",
"0:\tlearn: 0.5273837\ttotal: 4.16ms\tremaining: 1.24s\n",
"0:\tlearn: 0.5314670\ttotal: 3.1ms\tremaining: 928ms\n",
"0:\tlearn: 0.5139923\ttotal: 3.46ms\tremaining: 1.03s\n",
"0:\tlearn: 0.4610674\ttotal: 3.75ms\tremaining: 1.12s\n",
"0:\tlearn: 0.4960814\ttotal: 4.04ms\tremaining: 1.21s\n",
"0:\tlearn: 0.4749245\ttotal: 5.17ms\tremaining: 1.54s\n",
"0:\tlearn: 0.6793134\ttotal: 3.66ms\tremaining: 1.09s\n",
"0:\tlearn: 0.6788214\ttotal: 5.44ms\tremaining: 1.63s\n",
"0:\tlearn: 0.6789164\ttotal: 5.53ms\tremaining: 1.65s\n",
"0:\tlearn: 0.6477455\ttotal: 3.76ms\tremaining: 1.13s\n",
"0:\tlearn: 0.6598470\ttotal: 6.35ms\tremaining: 1.9s\n",
"0:\tlearn: 0.6518626\ttotal: 4.92ms\tremaining: 1.47s\n",
"0:\tlearn: 0.5720136\ttotal: 3.64ms\tremaining: 1.09s\n",
"0:\tlearn: 0.5765766\ttotal: 3.08ms\tremaining: 921ms\n",
"0:\tlearn: 0.5914926\ttotal: 5.24ms\tremaining: 1.57s\n",
"0:\tlearn: 0.5392716\ttotal: 2.38ms\tremaining: 711ms\n",
"0:\tlearn: 0.5408431\ttotal: 4.25ms\tremaining: 1.27s\n",
"0:\tlearn: 0.5299999\ttotal: 5.95ms\tremaining: 1.78s\n",
"0:\tlearn: 0.4611052\ttotal: 4.49ms\tremaining: 1.34s\n",
"0:\tlearn: 0.4677294\ttotal: 6.63ms\tremaining: 1.98s\n",
"0:\tlearn: 0.4665631\ttotal: 5.14ms\tremaining: 1.54s\n",
"0:\tlearn: 0.6763189\ttotal: 7.51ms\tremaining: 2.25s\n",
"0:\tlearn: 0.6791868\ttotal: 5.41ms\tremaining: 1.62s\n",
"0:\tlearn: 0.6782774\ttotal: 5.77ms\tremaining: 1.73s\n",
"0:\tlearn: 0.6399973\ttotal: 8.67ms\tremaining: 2.59s\n",
"0:\tlearn: 0.6448821\ttotal: 5.32ms\tremaining: 1.59s\n",
"0:\tlearn: 0.6380162\ttotal: 5.67ms\tremaining: 1.7s\n",
"0:\tlearn: 0.5470828\ttotal: 5.68ms\tremaining: 1.7s\n",
"0:\tlearn: 0.5607834\ttotal: 6.42ms\tremaining: 1.92s\n",
"0:\tlearn: 0.5300373\ttotal: 5.87ms\tremaining: 1.75s\n",
"0:\tlearn: 0.4839300\ttotal: 4.83ms\tremaining: 1.45s\n",
"0:\tlearn: 0.5137769\ttotal: 6.01ms\tremaining: 1.8s\n",
"0:\tlearn: 0.5159837\ttotal: 5.52ms\tremaining: 1.65s\n",
"0:\tlearn: 0.4521045\ttotal: 7.5ms\tremaining: 2.24s\n",
"0:\tlearn: 0.4539012\ttotal: 6.83ms\tremaining: 2.04s\n",
"0:\tlearn: 0.4322570\ttotal: 6.11ms\tremaining: 1.83s\n",
"0:\tlearn: 0.6782346\ttotal: 6.96ms\tremaining: 2.08s\n",
"0:\tlearn: 0.6791731\ttotal: 7.54ms\tremaining: 2.25s\n",
"0:\tlearn: 0.6772909\ttotal: 8.13ms\tremaining: 2.43s\n",
"0:\tlearn: 0.6481739\ttotal: 7.7ms\tremaining: 2.3s\n",
"0:\tlearn: 0.6568169\ttotal: 7.17ms\tremaining: 2.14s\n",
"0:\tlearn: 0.6522204\ttotal: 6.08ms\tremaining: 1.82s\n",
"0:\tlearn: 0.5358036\ttotal: 9.17ms\tremaining: 2.74s\n",
"0:\tlearn: 0.5689410\ttotal: 7.52ms\tremaining: 2.25s\n",
"0:\tlearn: 0.5545618\ttotal: 8.38ms\tremaining: 2.51s\n",
"0:\tlearn: 0.4899930\ttotal: 6.61ms\tremaining: 1.98s\n",
"0:\tlearn: 0.4923427\ttotal: 6.23ms\tremaining: 1.86s\n",
"0:\tlearn: 0.4960295\ttotal: 7.08ms\tremaining: 2.12s\n",
"0:\tlearn: 0.4606173\ttotal: 6.47ms\tremaining: 1.93s\n",
"0:\tlearn: 0.4578639\ttotal: 6.37ms\tremaining: 1.9s\n",
"0:\tlearn: 0.4515551\ttotal: 5.3ms\tremaining: 1.58s\n",
"0:\tlearn: 0.6792081\ttotal: 5.94ms\tremaining: 1.78s\n",
"0:\tlearn: 0.6787121\ttotal: 4.7ms\tremaining: 1.4s\n",
"0:\tlearn: 0.6791757\ttotal: 6.21ms\tremaining: 1.86s\n",
"0:\tlearn: 0.6558036\ttotal: 5.9ms\tremaining: 1.76s\n",
"0:\tlearn: 0.6578874\ttotal: 5.31ms\tremaining: 1.59s\n",
"0:\tlearn: 0.6488785\ttotal: 5.28ms\tremaining: 1.58s\n",
"0:\tlearn: 0.5467265\ttotal: 5.66ms\tremaining: 1.69s\n",
"0:\tlearn: 0.5858871\ttotal: 7.44ms\tremaining: 2.23s\n",
"0:\tlearn: 0.5493058\ttotal: 6.32ms\tremaining: 1.89s\n",
"0:\tlearn: 0.5044646\ttotal: 5.74ms\tremaining: 1.72s\n",
"0:\tlearn: 0.5180618\ttotal: 4.94ms\tremaining: 1.48s\n",
"0:\tlearn: 0.5340242\ttotal: 4.23ms\tremaining: 1.26s\n",
"0:\tlearn: 0.4201193\ttotal: 5.37ms\tremaining: 1.6s\n",
"0:\tlearn: 0.4420500\ttotal: 7.03ms\tremaining: 2.1s\n",
"0:\tlearn: 0.4854969\ttotal: 6.94ms\tremaining: 2.07s\n",
"0:\tlearn: 0.6796970\ttotal: 4.99ms\tremaining: 1.49s\n",
"0:\tlearn: 0.6811578\ttotal: 8.38ms\tremaining: 2.5s\n",
"0:\tlearn: 0.6794285\ttotal: 5.55ms\tremaining: 1.66s\n",
"0:\tlearn: 0.6548784\ttotal: 3.98ms\tremaining: 1.19s\n",
"0:\tlearn: 0.6552105\ttotal: 6.55ms\tremaining: 1.96s\n",
"0:\tlearn: 0.6572882\ttotal: 5.95ms\tremaining: 1.78s\n",
"0:\tlearn: 0.5726606\ttotal: 6.26ms\tremaining: 1.87s\n",
"0:\tlearn: 0.5829516\ttotal: 4.16ms\tremaining: 1.24s\n",
"0:\tlearn: 0.5790845\ttotal: 5.38ms\tremaining: 1.61s\n",
"0:\tlearn: 0.5246937\ttotal: 6.66ms\tremaining: 1.99s\n",
"0:\tlearn: 0.5205658\ttotal: 7.51ms\tremaining: 2.24s\n",
"0:\tlearn: 0.5094707\ttotal: 7.37ms\tremaining: 2.2s\n",
"0:\tlearn: 0.4672145\ttotal: 7.42ms\tremaining: 2.22s\n",
"0:\tlearn: 0.4820505\ttotal: 5ms\tremaining: 1.5s\n",
"0:\tlearn: 0.4780065\ttotal: 5.08ms\tremaining: 1.52s\n",
"0:\tlearn: 0.6812331\ttotal: 4.98ms\tremaining: 1.49s\n",
"0:\tlearn: 0.6796369\ttotal: 6.58ms\tremaining: 1.97s\n",
"0:\tlearn: 0.6798470\ttotal: 7.09ms\tremaining: 2.12s\n",
"0:\tlearn: 0.6514228\ttotal: 7.81ms\tremaining: 2.33s\n",
"0:\tlearn: 0.6586789\ttotal: 9.39ms\tremaining: 2.81s\n",
"0:\tlearn: 0.6587177\ttotal: 8.18ms\tremaining: 2.44s\n",
"0:\tlearn: 0.5758170\ttotal: 5.6ms\tremaining: 1.68s\n",
"0:\tlearn: 0.5695221\ttotal: 6.23ms\tremaining: 1.86s\n",
"0:\tlearn: 0.5689711\ttotal: 5.57ms\tremaining: 1.67s\n",
"0:\tlearn: 0.5019668\ttotal: 4.79ms\tremaining: 1.43s\n",
"0:\tlearn: 0.5136667\ttotal: 5.85ms\tremaining: 1.75s\n",
"0:\tlearn: 0.5307984\ttotal: 6.21ms\tremaining: 1.86s\n",
"0:\tlearn: 0.4808830\ttotal: 7.34ms\tremaining: 2.19s\n",
"0:\tlearn: 0.4876688\ttotal: 7.84ms\tremaining: 2.34s\n",
"0:\tlearn: 0.4881987\ttotal: 8.86ms\tremaining: 2.65s\n",
"0:\tlearn: 0.6765672\ttotal: 10.3ms\tremaining: 3.09s\n",
"0:\tlearn: 0.6761378\ttotal: 12.8ms\tremaining: 3.82s\n",
"0:\tlearn: 0.6741468\ttotal: 12.4ms\tremaining: 3.71s\n",
"0:\tlearn: 0.6406999\ttotal: 10.7ms\tremaining: 3.2s\n",
"0:\tlearn: 0.6454692\ttotal: 9.8ms\tremaining: 2.93s\n",
"0:\tlearn: 0.6463584\ttotal: 12.6ms\tremaining: 3.76s\n",
"0:\tlearn: 0.5216249\ttotal: 14.2ms\tremaining: 4.25s\n",
"0:\tlearn: 0.5423190\ttotal: 10.8ms\tremaining: 3.23s\n",
"0:\tlearn: 0.5297589\ttotal: 10.9ms\tremaining: 3.27s\n",
"0:\tlearn: 0.4740024\ttotal: 9.81ms\tremaining: 2.93s\n",
"0:\tlearn: 0.4863410\ttotal: 10.4ms\tremaining: 3.11s\n",
"0:\tlearn: 0.4911348\ttotal: 13.4ms\tremaining: 4.01s\n",
"0:\tlearn: 0.4087145\ttotal: 10.9ms\tremaining: 3.25s\n",
"0:\tlearn: 0.4529791\ttotal: 13.7ms\tremaining: 4.1s\n",
"0:\tlearn: 0.4286149\ttotal: 10.6ms\tremaining: 3.17s\n",
"0:\tlearn: 0.6779843\ttotal: 10.3ms\tremaining: 3.08s\n",
"0:\tlearn: 0.6793634\ttotal: 10.1ms\tremaining: 3.01s\n",
"0:\tlearn: 0.6785046\ttotal: 13.7ms\tremaining: 4.09s\n",
"0:\tlearn: 0.6493639\ttotal: 11ms\tremaining: 3.3s\n",
"0:\tlearn: 0.6496168\ttotal: 13.6ms\tremaining: 4.05s\n",
"0:\tlearn: 0.6496367\ttotal: 11.5ms\tremaining: 3.45s\n",
"0:\tlearn: 0.5485536\ttotal: 12.9ms\tremaining: 3.84s\n",
"0:\tlearn: 0.5664884\ttotal: 15.8ms\tremaining: 4.72s\n",
"0:\tlearn: 0.5572717\ttotal: 8.48ms\tremaining: 2.54s\n",
"0:\tlearn: 0.5232135\ttotal: 13.7ms\tremaining: 4.1s\n",
"0:\tlearn: 0.5203572\ttotal: 13.6ms\tremaining: 4.05s\n",
"0:\tlearn: 0.5093712\ttotal: 14.1ms\tremaining: 4.21s\n",
"0:\tlearn: 0.4318639\ttotal: 12.1ms\tremaining: 3.63s\n",
"0:\tlearn: 0.4849647\ttotal: 9.13ms\tremaining: 2.73s\n",
"0:\tlearn: 0.4481933\ttotal: 14ms\tremaining: 4.18s\n",
"0:\tlearn: 0.6807299\ttotal: 12.4ms\tremaining: 3.71s\n",
"0:\tlearn: 0.6795601\ttotal: 14.6ms\tremaining: 4.38s\n",
"0:\tlearn: 0.6789115\ttotal: 11.8ms\tremaining: 3.54s\n",
"0:\tlearn: 0.6495328\ttotal: 11.3ms\tremaining: 3.37s\n",
"0:\tlearn: 0.6505629\ttotal: 15.3ms\tremaining: 4.57s\n",
"0:\tlearn: 0.6560628\ttotal: 16.4ms\tremaining: 4.9s\n",
"0:\tlearn: 0.5702048\ttotal: 12.1ms\tremaining: 3.6s\n",
"0:\tlearn: 0.5721039\ttotal: 12.4ms\tremaining: 3.72s\n",
"0:\tlearn: 0.5778547\ttotal: 12.6ms\tremaining: 3.77s\n",
"0:\tlearn: 0.5162702\ttotal: 13ms\tremaining: 3.88s\n",
"0:\tlearn: 0.5281649\ttotal: 17.2ms\tremaining: 5.13s\n",
"0:\tlearn: 0.5007893\ttotal: 17ms\tremaining: 5.08s\n",
"0:\tlearn: 0.4726059\ttotal: 13.4ms\tremaining: 4s\n",
"0:\tlearn: 0.4819736\ttotal: 11.3ms\tremaining: 3.39s\n",
"0:\tlearn: 0.4395302\ttotal: 11.8ms\tremaining: 3.54s\n",
"0:\tlearn: 0.6810760\ttotal: 10.1ms\tremaining: 3.03s\n",
"0:\tlearn: 0.6815671\ttotal: 13.7ms\tremaining: 4.09s\n",
"0:\tlearn: 0.6795634\ttotal: 12ms\tremaining: 3.58s\n",
"0:\tlearn: 0.6516506\ttotal: 14.3ms\tremaining: 4.27s\n",
"0:\tlearn: 0.6515003\ttotal: 12ms\tremaining: 3.58s\n",
"0:\tlearn: 0.6520882\ttotal: 14ms\tremaining: 4.2s\n",
"0:\tlearn: 0.5778300\ttotal: 14.3ms\tremaining: 4.26s\n",
"0:\tlearn: 0.5803605\ttotal: 16.6ms\tremaining: 4.96s\n",
"0:\tlearn: 0.5887525\ttotal: 10.2ms\tremaining: 3.05s\n",
"0:\tlearn: 0.5189867\ttotal: 14.3ms\tremaining: 4.29s\n",
"0:\tlearn: 0.5378705\ttotal: 14.8ms\tremaining: 4.41s\n",
"0:\tlearn: 0.5121929\ttotal: 9.66ms\tremaining: 2.89s\n",
"0:\tlearn: 0.4652151\ttotal: 17.6ms\tremaining: 5.25s\n",
"0:\tlearn: 0.4969731\ttotal: 13ms\tremaining: 3.88s\n",
"0:\tlearn: 0.4970850\ttotal: 13.2ms\tremaining: 3.96s\n",
"0:\tlearn: 0.6809442\ttotal: 11.9ms\tremaining: 3.57s\n",
"0:\tlearn: 0.6801504\ttotal: 12.3ms\tremaining: 3.67s\n",
"0:\tlearn: 0.6806916\ttotal: 12.6ms\tremaining: 3.76s\n",
"0:\tlearn: 0.6550788\ttotal: 13.7ms\tremaining: 4.09s\n",
"0:\tlearn: 0.6557985\ttotal: 15.7ms\tremaining: 4.7s\n",
"0:\tlearn: 0.6572001\ttotal: 12.5ms\tremaining: 3.74s\n",
"0:\tlearn: 0.5699050\ttotal: 11ms\tremaining: 3.29s\n",
"0:\tlearn: 0.5921490\ttotal: 11.5ms\tremaining: 3.44s\n",
"0:\tlearn: 0.5833878\ttotal: 12.4ms\tremaining: 3.71s\n",
"0:\tlearn: 0.5358695\ttotal: 9.83ms\tremaining: 2.94s\n",
"0:\tlearn: 0.5430965\ttotal: 12.3ms\tremaining: 3.68s\n",
"0:\tlearn: 0.5475019\ttotal: 13.4ms\tremaining: 4.02s\n",
"0:\tlearn: 0.4647570\ttotal: 12ms\tremaining: 3.58s\n",
"0:\tlearn: 0.4896463\ttotal: 10.8ms\tremaining: 3.22s\n",
"0:\tlearn: 0.4966204\ttotal: 9.53ms\tremaining: 2.85s\n",
"0:\tlearn: 0.6767626\ttotal: 30.9ms\tremaining: 9.25s\n",
"0:\tlearn: 0.6761846\ttotal: 43ms\tremaining: 12.9s\n",
"0:\tlearn: 0.6751544\ttotal: 16ms\tremaining: 4.77s\n",
"0:\tlearn: 0.6464528\ttotal: 36.5ms\tremaining: 10.9s\n",
"0:\tlearn: 0.6480471\ttotal: 40.1ms\tremaining: 12s\n",
"0:\tlearn: 0.6405739\ttotal: 41.4ms\tremaining: 12.4s\n",
"0:\tlearn: 0.5384913\ttotal: 42.1ms\tremaining: 12.6s\n",
"0:\tlearn: 0.5417444\ttotal: 37ms\tremaining: 11.1s\n",
"0:\tlearn: 0.5486953\ttotal: 38.9ms\tremaining: 11.6s\n",
"0:\tlearn: 0.4733382\ttotal: 36.4ms\tremaining: 10.9s\n",
"0:\tlearn: 0.4777080\ttotal: 39.2ms\tremaining: 11.7s\n",
"0:\tlearn: 0.4656260\ttotal: 37.2ms\tremaining: 11.1s\n",
"0:\tlearn: 0.4098114\ttotal: 33ms\tremaining: 9.86s\n",
"0:\tlearn: 0.4202076\ttotal: 42.4ms\tremaining: 12.7s\n",
"0:\tlearn: 0.4278623\ttotal: 48.1ms\tremaining: 14.4s\n",
"0:\tlearn: 0.6783987\ttotal: 38ms\tremaining: 11.4s\n",
"0:\tlearn: 0.6791731\ttotal: 41.6ms\tremaining: 12.4s\n",
"0:\tlearn: 0.6799685\ttotal: 44.9ms\tremaining: 13.4s\n",
"0:\tlearn: 0.6497340\ttotal: 35.1ms\tremaining: 10.5s\n",
"0:\tlearn: 0.6485758\ttotal: 30ms\tremaining: 8.96s\n",
"0:\tlearn: 0.6534440\ttotal: 48.3ms\tremaining: 14.4s\n",
"0:\tlearn: 0.5608971\ttotal: 34.1ms\tremaining: 10.2s\n",
"0:\tlearn: 0.5710493\ttotal: 43ms\tremaining: 12.9s\n",
"0:\tlearn: 0.5648622\ttotal: 33ms\tremaining: 9.87s\n",
"0:\tlearn: 0.5184665\ttotal: 37.3ms\tremaining: 11.1s\n",
"0:\tlearn: 0.5079963\ttotal: 36.1ms\tremaining: 10.8s\n",
"0:\tlearn: 0.4992106\ttotal: 39.3ms\tremaining: 11.8s\n",
"0:\tlearn: 0.4587841\ttotal: 35.1ms\tremaining: 10.5s\n",
"0:\tlearn: 0.4588962\ttotal: 32.1ms\tremaining: 9.6s\n",
"0:\tlearn: 0.4563440\ttotal: 39.5ms\tremaining: 11.8s\n",
"0:\tlearn: 0.6799946\ttotal: 39.1ms\tremaining: 11.7s\n",
"0:\tlearn: 0.6802986\ttotal: 44.7ms\tremaining: 13.4s\n",
"0:\tlearn: 0.6780665\ttotal: 14.9ms\tremaining: 4.46s\n",
"0:\tlearn: 0.6499439\ttotal: 42.7ms\tremaining: 12.8s\n",
"0:\tlearn: 0.6615347\ttotal: 37.7ms\tremaining: 11.3s\n",
"0:\tlearn: 0.6591326\ttotal: 54.3ms\tremaining: 16.2s\n",
"0:\tlearn: 0.5696869\ttotal: 36.3ms\tremaining: 10.9s\n",
"0:\tlearn: 0.5755912\ttotal: 38.7ms\tremaining: 11.6s\n",
"0:\tlearn: 0.5756533\ttotal: 34.1ms\tremaining: 10.2s\n",
"0:\tlearn: 0.5195007\ttotal: 42.4ms\tremaining: 12.7s\n",
"0:\tlearn: 0.5239030\ttotal: 36.6ms\tremaining: 10.9s\n",
"0:\tlearn: 0.5180728\ttotal: 44.5ms\tremaining: 13.3s\n",
"0:\tlearn: 0.4715064\ttotal: 32.5ms\tremaining: 9.71s\n",
"0:\tlearn: 0.4831016\ttotal: 45.7ms\tremaining: 13.7s\n",
"0:\tlearn: 0.4752611\ttotal: 34.4ms\tremaining: 10.3s\n",
"0:\tlearn: 0.6807020\ttotal: 15.7ms\tremaining: 4.69s\n",
"0:\tlearn: 0.6807825\ttotal: 41.8ms\tremaining: 12.5s\n",
"0:\tlearn: 0.6805748\ttotal: 45.5ms\tremaining: 13.6s\n",
"0:\tlearn: 0.6555959\ttotal: 50.1ms\tremaining: 15s\n",
"0:\tlearn: 0.6574481\ttotal: 45.9ms\tremaining: 13.7s\n",
"0:\tlearn: 0.6583286\ttotal: 25.6ms\tremaining: 7.66s\n",
"0:\tlearn: 0.5669769\ttotal: 40.1ms\tremaining: 12s\n",
"0:\tlearn: 0.5900235\ttotal: 42.5ms\tremaining: 12.7s\n",
"0:\tlearn: 0.5817783\ttotal: 46.4ms\tremaining: 13.9s\n",
"0:\tlearn: 0.5357351\ttotal: 39.3ms\tremaining: 11.8s\n",
"0:\tlearn: 0.5280574\ttotal: 40.7ms\tremaining: 12.2s\n",
"0:\tlearn: 0.5317676\ttotal: 39.7ms\tremaining: 11.9s\n",
"0:\tlearn: 0.4684350\ttotal: 32.4ms\tremaining: 9.7s\n",
"0:\tlearn: 0.4836682\ttotal: 32.6ms\tremaining: 9.73s\n",
"0:\tlearn: 0.4748951\ttotal: 35.7ms\tremaining: 10.7s\n",
"0:\tlearn: 0.6814425\ttotal: 33ms\tremaining: 9.88s\n",
"0:\tlearn: 0.6822991\ttotal: 37.8ms\tremaining: 11.3s\n",
"0:\tlearn: 0.6816844\ttotal: 45.7ms\tremaining: 13.7s\n",
"0:\tlearn: 0.6520566\ttotal: 6.4ms\tremaining: 1.91s\n",
"0:\tlearn: 0.6596794\ttotal: 37.9ms\tremaining: 11.3s\n",
"0:\tlearn: 0.6594869\ttotal: 40.6ms\tremaining: 12.1s\n",
"0:\tlearn: 0.5906718\ttotal: 37.8ms\tremaining: 11.3s\n",
"0:\tlearn: 0.5914675\ttotal: 30.9ms\tremaining: 9.24s\n",
"0:\tlearn: 0.5770167\ttotal: 41.8ms\tremaining: 12.5s\n",
"0:\tlearn: 0.5324715\ttotal: 32.8ms\tremaining: 9.81s\n",
"0:\tlearn: 0.5435696\ttotal: 35.3ms\tremaining: 10.6s\n",
"0:\tlearn: 0.5438259\ttotal: 47.5ms\tremaining: 14.2s\n",
"0:\tlearn: 0.4979287\ttotal: 42.8ms\tremaining: 12.8s\n",
"0:\tlearn: 0.5053222\ttotal: 31.8ms\tremaining: 9.52s\n",
"0:\tlearn: 0.5070885\ttotal: 37.5ms\tremaining: 11.2s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=1)]: Done 375 out of 375 | elapsed: 23.0min finished\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"0:\tlearn: 0.5041958\ttotal: 7.36ms\tremaining: 2.2s\n",
"Computation time (parameter tunning): 1382 seconds...\n",
"Best parameters: {'depth': 6, 'iterations': 300, 'l2_leaf_reg': 3, 'learning_rate': 0.15}\n",
"0:\tlearn: 0.9646172\ttotal: 4.72ms\tremaining: 1.41s\n",
"30:\tlearn: 0.9992981\ttotal: 204ms\tremaining: 1.77s\n",
"60:\tlearn: 1.0000000\ttotal: 407ms\tremaining: 1.59s\n",
"90:\tlearn: 1.0000000\ttotal: 592ms\tremaining: 1.36s\n",
"120:\tlearn: 1.0000000\ttotal: 789ms\tremaining: 1.17s\n",
"150:\tlearn: 1.0000000\ttotal: 975ms\tremaining: 962ms\n",
"180:\tlearn: 1.0000000\ttotal: 1.16s\tremaining: 763ms\n",
"210:\tlearn: 1.0000000\ttotal: 1.36s\tremaining: 573ms\n",
"240:\tlearn: 1.0000000\ttotal: 1.55s\tremaining: 380ms\n",
"270:\tlearn: 1.0000000\ttotal: 1.74s\tremaining: 186ms\n",
"********** SUMMARY: XGBoost (Exp7-Undersampled) **************\n",
"- Parameters used: {'depth': 6, 'iterations': 300, 'l2_leaf_reg': 3, 'learning_rate': 0.15}\n",
"- AUC(Train|Test): (1.0, 0.97934650790487321)\n",
"- Metrics: Train data (n=696) **\n",
" TN=348, FP=0\n",
" FN=0, TP=348\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=82290, FP=3009\n",
" FN=16, TP=128\n",
" Precision: 0.04\n",
" Recall(TPR): 0.89\n",
" BAcc=(TPR+TNR)/2: 0.93\n",
"Computation time (Total): 2 seconds\n",
"Using parameters: {'depth': 6, 'iterations': 300, 'l2_leaf_reg': 3, 'learning_rate': 0.15}\n",
"0:\tlearn: 0.9990926\ttotal: 122ms\tremaining: 36.4s\n",
"30:\tlearn: 0.9999242\ttotal: 4.53s\tremaining: 39.3s\n",
"60:\tlearn: 0.9999650\ttotal: 9.14s\tremaining: 35.8s\n",
"90:\tlearn: 0.9999757\ttotal: 13.8s\tremaining: 31.6s\n",
"120:\tlearn: 0.9999824\ttotal: 18.4s\tremaining: 27.3s\n",
"150:\tlearn: 0.9999861\ttotal: 23s\tremaining: 22.7s\n",
"180:\tlearn: 0.9999899\ttotal: 27.6s\tremaining: 18.2s\n",
"210:\tlearn: 0.9999918\ttotal: 32s\tremaining: 13.5s\n",
"240:\tlearn: 0.9999940\ttotal: 36.5s\tremaining: 8.95s\n",
"270:\tlearn: 0.9999958\ttotal: 41s\tremaining: 4.39s\n",
"********** SUMMARY: XGBoost (Exp7-SMOTE2 (best parameters from Undersampled)) **************\n",
"- Parameters used: {'depth': 6, 'iterations': 300, 'l2_leaf_reg': 3, 'learning_rate': 0.15}\n",
"- AUC(Train|Test): (0.99999699591249702, 0.98101832312740411)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=199002, FP=14\n",
" FN=23, TP=198993\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85278, FP=21\n",
" FN=23, TP=121\n",
" Precision: 0.85\n",
" Recall(TPR): 0.84\n",
" BAcc=(TPR+TNR)/2: 0.92\n",
"Computation time (Total): 51 seconds\n"
]
}
],
"source": [
"## EXPERIMENT 5 - Train data: SMOTE + Selected Feats, Test data: all (best parameters from Undersampled data)\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp5-SMOTE (best parameters from Undersampled)'\n",
"iselected = X_train.columns[select_feature1.get_support()] # Univariate ANOVA\n",
"XX_train = X_train_SMOTE[iselected].copy()\n",
"yy_train = y_train_SMOTE.copy()\n",
"XX_test = X_test[iselected].copy()\n",
"yy_test = y_test.copy()\n",
"# 2) Get best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 1 # smaller set\n",
"flag.pars2use = {} # will not run GridSearch if selected\n",
"tmp = run_CatBoost(X_undersampled[iselected],y_undersampled,XX_test,yy_test,'Exp5-Undersampled',flag)\n",
"# 3) Run on whole data with the best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = tmp # will not run GridSearch if selected\n",
"best5 = run_CatBoost(XX_train,yy_train,XX_test,yy_test,mylabel,flag)\n",
"print('\\n\\n')\n",
"\n",
"## EXPERIMENT 6 - Train data: SMOTE2 + Selected Feats, Test data: all (best parameters from Undersampled data)\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp6-SMOTE2 (best parameters from Undersampled)'\n",
"iselected = X_train.columns[select_feature1.get_support()] # Univariate ANOVA\n",
"XX_train = X_train_SMOTE2[iselected].copy()\n",
"yy_train = y_train_SMOTE2.copy()\n",
"XX_test = X_test[iselected].copy()\n",
"yy_test = y_test.copy()\n",
"# 2) Get best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 1 # smaller set\n",
"flag.pars2use = {} # will not run GridSearch if selected\n",
"tmp = run_CatBoost(X_undersampled[iselected],y_undersampled,XX_test,yy_test,'Exp6-Undersampled',flag)\n",
"# 3) Run on whole data with the best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = tmp # will not run GridSearch if selected\n",
"best6 = run_CatBoost(XX_train,yy_train,XX_test,yy_test,mylabel,flag)\n",
"print('\\n\\n')\n",
"\n",
"## EXPERIMENT 7 - Train data: SMOTE2 + Selected Feats (Chi2), Test data: all (best parameters from Undersampled data)\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp7-SMOTE2 (best parameters from Undersampled)'\n",
"iselected = X_train.columns[select_feature2.get_support()] # Chi2\n",
"print('Selected feats: ',iselected)\n",
"XX_train = X_train_SMOTE2[iselected].copy()\n",
"yy_train = y_train_SMOTE2.copy()\n",
"XX_test = X_test[iselected].copy() \n",
"yy_test = y_test.copy()\n",
"# 2) Get best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 1 # smaller set\n",
"flag.pars2use = {} # will not run GridSearch if selected\n",
"tmp = run_CatBoost(X_undersampled[iselected],y_undersampled,XX_test,yy_test,'Exp7-Undersampled',flag)\n",
"# 3) Run on whole data with the best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = tmp # will not run GridSearch if selected\n",
"best7 = run_CatBoost(XX_train,yy_train,XX_test,yy_test,mylabel,flag)"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Using parameters: {'depth': 10, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"0:\tlearn: 0.9967790\ttotal: 178ms\tremaining: 53.2s\n",
"30:\tlearn: 0.9989170\ttotal: 5.56s\tremaining: 48.2s\n",
"60:\tlearn: 0.9991546\ttotal: 11.3s\tremaining: 44.4s\n",
"90:\tlearn: 0.9994009\ttotal: 17.4s\tremaining: 40s\n",
"120:\tlearn: 0.9995627\ttotal: 23.2s\tremaining: 34.4s\n",
"150:\tlearn: 0.9996614\ttotal: 28.8s\tremaining: 28.5s\n",
"180:\tlearn: 0.9997330\ttotal: 34.4s\tremaining: 22.6s\n",
"210:\tlearn: 0.9997918\ttotal: 40.4s\tremaining: 17s\n",
"240:\tlearn: 0.9998277\ttotal: 46.7s\tremaining: 11.4s\n",
"270:\tlearn: 0.9998588\ttotal: 52.9s\tremaining: 5.66s\n",
"********** SUMMARY: CatBoost (Exp5-SMOTE (best parameters from Undersampled)) **************\n",
"- Parameters used: {'depth': 10, 'iterations': 300, 'l2_leaf_reg': 9, 'learning_rate': 0.01}\n",
"- AUC(Train|Test): (0.99987329757767385, 0.98412341358697708)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=198630, FP=386\n",
" FN=616, TP=198400\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85118, FP=181\n",
" FN=21, TP=123\n",
" Precision: 0.40\n",
" Recall(TPR): 0.85\n",
" BAcc=(TPR+TNR)/2: 0.93\n",
"Computation time (Total): 64 seconds\n",
"\n",
"\n",
"\n",
"Using parameters: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 5, 'learning_rate': 0.01}\n",
"0:\tlearn: 0.9993274\ttotal: 139ms\tremaining: 41.6s\n",
"30:\tlearn: 0.9998200\ttotal: 4.63s\tremaining: 40.2s\n",
"60:\tlearn: 0.9998294\ttotal: 9.33s\tremaining: 36.6s\n",
"90:\tlearn: 0.9998489\ttotal: 14s\tremaining: 32.2s\n",
"120:\tlearn: 0.9998633\ttotal: 18.8s\tremaining: 27.9s\n",
"150:\tlearn: 0.9998750\ttotal: 23.8s\tremaining: 23.5s\n",
"180:\tlearn: 0.9998830\ttotal: 28.6s\tremaining: 18.8s\n",
"210:\tlearn: 0.9998968\ttotal: 33.6s\tremaining: 14.2s\n",
"240:\tlearn: 0.9999066\ttotal: 38.6s\tremaining: 9.45s\n",
"270:\tlearn: 0.9999123\ttotal: 43.6s\tremaining: 4.66s\n",
"********** SUMMARY: CatBoost (Exp6-SMOTE2 (best parameters from Undersampled)) **************\n",
"- Parameters used: {'depth': 8, 'iterations': 300, 'l2_leaf_reg': 5, 'learning_rate': 0.01}\n",
"- AUC(Train|Test): (0.99992036154789155, 0.97838457302482373)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=198873, FP=143\n",
" FN=275, TP=198741\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85228, FP=71\n",
" FN=22, TP=122\n",
" Precision: 0.63\n",
" Recall(TPR): 0.85\n",
" BAcc=(TPR+TNR)/2: 0.92\n",
"Computation time (Total): 54 seconds\n",
"\n",
"\n",
"\n",
"Selected feats: Index(['Time', 'V1', 'V2', 'V3', 'V4', 'V5', 'V7', 'V8', 'V10', 'V11', 'V12',\n",
" 'V14', 'V16', 'V17', 'Amount'],\n",
" dtype='object')\n",
"Using parameters: {'depth': 6, 'iterations': 300, 'l2_leaf_reg': 3, 'learning_rate': 0.15}\n",
"0:\tlearn: 0.9986019\ttotal: 152ms\tremaining: 45.6s\n",
"30:\tlearn: 0.9999167\ttotal: 4.78s\tremaining: 41.4s\n",
"60:\tlearn: 0.9999711\ttotal: 9.53s\tremaining: 37.3s\n",
"90:\tlearn: 0.9999814\ttotal: 14.2s\tremaining: 32.5s\n",
"120:\tlearn: 0.9999855\ttotal: 18.8s\tremaining: 27.8s\n",
"150:\tlearn: 0.9999890\ttotal: 23.4s\tremaining: 23.1s\n",
"180:\tlearn: 0.9999914\ttotal: 27.8s\tremaining: 18.3s\n",
"210:\tlearn: 0.9999931\ttotal: 32.3s\tremaining: 13.6s\n",
"240:\tlearn: 0.9999947\ttotal: 36.8s\tremaining: 9.01s\n",
"270:\tlearn: 0.9999963\ttotal: 41.3s\tremaining: 4.42s\n",
"********** SUMMARY: CatBoost (Exp7-SMOTE2 (best parameters from Undersampled)) **************\n",
"- Parameters used: {'depth': 6, 'iterations': 300, 'l2_leaf_reg': 3, 'learning_rate': 0.15}\n",
"- AUC(Train|Test): (0.99999696061603427, 0.97547125080273178)\n",
"- Metrics: Train data (n=398032) **\n",
" TN=199000, FP=16\n",
" FN=22, TP=198994\n",
" Precision: 1.00\n",
" Recall(TPR): 1.00\n",
" BAcc=(TPR+TNR)/2: 1.00\n",
"- Metrics: Test data (n=85443) **\n",
" TN=85277, FP=22\n",
" FN=22, TP=122\n",
" Precision: 0.85\n",
" Recall(TPR): 0.85\n",
" BAcc=(TPR+TNR)/2: 0.92\n",
"Computation time (Total): 51 seconds\n"
]
}
],
"source": [
"# Rerun wuth the best parameters\n",
"\n",
"## EXPERIMENT 5 - Train data: SMOTE + Selected Feats, Test data: all (best parameters from Undersampled data)\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp5-SMOTE (best parameters from Undersampled)'\n",
"iselected = X_train.columns[select_feature1.get_support()] # Univariate ANOVA\n",
"XX_train = X_train_SMOTE[iselected].copy()\n",
"yy_train = y_train_SMOTE.copy()\n",
"XX_test = X_test[iselected].copy()\n",
"yy_test = y_test.copy()\n",
"# # 2) Get best parameters from Undersampled data\n",
"# flag = matlab_like()\n",
"# flag.set2use = 1 # smaller set\n",
"# flag.pars2use = {} # will not run GridSearch if selected\n",
"# tmp = run_CatBoost(X_undersampled[iselected],y_undersampled,XX_test,yy_test,'Exp5-Undersampled',flag)\n",
"# 3) Run on whole data with the best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = best5 # will not run GridSearch if selected\n",
"tmp = run_CatBoost(XX_train,yy_train,XX_test,yy_test,mylabel,flag)\n",
"print('\\n\\n')\n",
"\n",
"## EXPERIMENT 6 - Train data: SMOTE2 + Selected Feats, Test data: all (best parameters from Undersampled data)\n",
"\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp6-SMOTE2 (best parameters from Undersampled)'\n",
"iselected = X_train.columns[select_feature1.get_support()] # Univariate ANOVA\n",
"XX_train = X_train_SMOTE2[iselected].copy()\n",
"yy_train = y_train_SMOTE2.copy()\n",
"XX_test = X_test[iselected].copy()\n",
"yy_test = y_test.copy()\n",
"# # 2) Get best parameters from Undersampled data\n",
"# flag = matlab_like()\n",
"# flag.set2use = 1 # smaller set\n",
"# flag.pars2use = {} # will not run GridSearch if selected\n",
"# tmp = run_CatBoost(X_undersampled[iselected],y_undersampled,XX_test,yy_test,'Exp6-Undersampled',flag)\n",
"# 3) Run on whole data with the best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = best6 # will not run GridSearch if selected\n",
"tmp = run_CatBoost(XX_train,yy_train,XX_test,yy_test,mylabel,flag)\n",
"print('\\n\\n')\n",
"\n",
"## EXPERIMENT 7 - Train data: SMOTE2 + Selected Feats (Chi2), Test data: all (best parameters from Undersampled data)\n",
"# 1) Choose dataset\n",
"mylabel = 'Exp7-SMOTE2 (best parameters from Undersampled)'\n",
"iselected = X_train.columns[select_feature2.get_support()] # Chi2\n",
"print('Selected feats: ',iselected)\n",
"XX_train = X_train_SMOTE2[iselected].copy()\n",
"yy_train = y_train_SMOTE2.copy()\n",
"XX_test = X_test[iselected].copy() \n",
"yy_test = y_test.copy()\n",
"# # 2) Get best parameters from Undersampled data\n",
"# flag = matlab_like()\n",
"# flag.set2use = 1 # smaller set\n",
"# flag.pars2use = {} # will not run GridSearch if selected\n",
"# tmp = run_CatBoost(X_undersampled[iselected],y_undersampled,XX_test,yy_test,'Exp7-Undersampled',flag)\n",
"# 3) Run on whole data with the best parameters from Undersampled data\n",
"flag = matlab_like()\n",
"flag.set2use = 0 # smaller set\n",
"flag.pars2use = best7 # will not run GridSearch if selected\n",
"tmp = run_CatBoost(XX_train,yy_train,XX_test,yy_test,mylabel,flag)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment