Precision and recall are evaluation metrics that were designed for binary classification models, but precision and recall can be adapted for multi-class classification problems. Let me preface this blog post by saying that I don’t use precision and recall for multi-class classification problems because I don’t think they give much useful information. But some of my colleagues do compute precision and recall for multi-class problems so I don’t ignore these metrics.
Imagine you want to predict the political leaning of a person (conservative, moderate, liberal) based on sex, age, State, and income. This is a multi-class classification because there are more than two classes to predict.
If you want to predict sex (male, female) from age, State, income, and political leaning, then you have a binary classification problem because there are just two classes. For binary classification problems, it’s common to compute overall accuracy and accuracy by class, but also metrics called precision, recall, and F1 score. If you set one of the two possible classes as “positive” and the other as “negative” (the choice is arbitrary) then precision = TP / (TP + FP) and recall = TP / (TP + FN) where TP, TN, FP, FN are counts of:
TP = true positive = correctly predicted for positive class
TN = true negative = correctly predicted for negative class
FP = false positive = wrong prediction for positive class
FN = false negative = wrong prediction for negative class
The F1 score is the harmonic mean of precision and recall. For a particular problem, the choice of which class is “positive” and which is “negative” is arbitrary. If you swap meanings, the values of precision and recall swap. This can cause significant confusion.
So, computing precision, recall, and F1 score, for a binary classification model, makes sense and is common.
It is possible to compute a form of precision and recall for multi-class problems too. In fact, there are a few different techniques I’ve seen. The most logical approach is to compute precision and recall for each of the classes. For the political leaning example, for the conservative class people/items, you set “positive” as conservative and “negative” as moderate or liberal (i.e., not conservative). Then for moderate class people/items, you set “positive” as moderate and “negative” as conservative or liberal (not moderate). And for liberal class people/items, you set “positive” as liberal and “negative” as conservative or moderate (not liberal). This is a form of “one-versus-rest”.
I implemented a function to compute precision and recall (and F1 score) for a PyTorch multi-class classifier. My synthetic data looks like:
1, 0.24, 1, 0, 0, 0.2950, 2 -1, 0.39, 0, 0, 1, 0.5120, 1 1, 0.63, 0, 1, 0, 0.7580, 0 -1, 0.36, 1, 0, 0, 0.4450, 1 . . .
The fields are sex (-1 = male, 1 = female), age (divided by 100), State (Michigan = 100, Nebraska = 010, Oklahoma = 001), income (divided by $100,000), and political leaning (0 = conservative, 1 = moderate, 2 = liberal). The goal is to predict political leaning. There are 200 training items and 40 test items.
After training the model, the output of the evaluation functions is:
Computing model accuracy Accuracy on training data = 0.8150 Accuracy on test data = 0.7500 Accuracy on test by class: class: 0 | count = 11 accuracy = 0.5455 class: 1 | count = 14 accuracy = 0.9286 class: 2 | count = 15 accuracy = 0.7333 Confusion matrix for test data: actual 0: 6 4 1 actual 1: 1 13 0 actual 2: 2 2 11 ------------ predicted 0 1 2 Precision, recall, F1 on test: Class: 0 precision = 0.5455 recall = 0.5455 F1 = 0.5455 Class: 1 precision = 0.9286 recall = 0.5909 F1 = 0.7222 Class: 2 precision = 0.7333 recall = 0.6471 F1 = 0.6875
The overall accuracies of 0.8150 and 0.7500 for the training and test data give a good coarse overview. The accuracies and counts by class (on the test data) give a more detailed view of the model, and for me, the best view of the model. The confusion matrix gives even more detailed information, but in some sense a confusion matrix is too detailed for me.
The recall, precision, and F1 don’t really add much in my opinion. Notice that by-class precision is the same as ordinary accuracy for the class, so you don’t gain any information. The by-class recall values don’t have any obvious interpretation other than “larger values are better”. And because the F1 score is just a kind of average of precision and recall, again, you only get “larger values are better”. (Not quite true but I don’t want to get side-tracked here).
Anyway, an interesting topic. Maybe the moral of the story is, “more data is usually better, but not always”.
When I hear the word “precision” in a normal (not machine learning) context, I often think of “precision engineering”. I grew up in the 1950s and like most guys of my generation, I had (and still have) a love of automobiles. My first car was a 1957 Ford Custom 300. Cars of the 1950s were often beautiful, but believe me, they were not examples of precision engineering.
Top: The 1952 Chrysler D’Elegance. Beautiful styling. Only one example was built as a design exercise.
Bottom: The 1954 Pontiac Bonneville Special featured innovative styling like a Plexiglas canopy with gull-wing windows on a fiberglass body. Only two were built.
Demo program.
# people_politics_precision_recall.py
# PyTorch 2.2.1-CPU Anaconda3-2023.09 Python 3.11.5
# Windows 10/11
# compute accuracy, precision, recall by class
import numpy as np
import torch as T
device = T.device('cpu') # apply to Tensor or Module
# -----------------------------------------------------------
class PeopleDataset(T.utils.data.Dataset):
# sex age state income politics
# -1 0.27 0 1 0 0.7610 2
# +1 0.19 0 0 1 0.6550 0
# sex: -1 = male, +1 = female
# state: michigan, nebraska, oklahoma
# politics: conservative, moderate, liberal
def __init__(self, src_file):
all_xy = np.loadtxt(src_file, usecols=range(0,7),
delimiter=",", comments="#", dtype=np.float32)
tmp_x = all_xy[:,0:6] # cols [0,6) = [0,5]
tmp_y = all_xy[:,6] # 1-D
self.x_data = T.tensor(tmp_x,
dtype=T.float32).to(device)
self.y_data = T.tensor(tmp_y,
dtype=T.int64).to(device) # 1-D
def __len__(self):
return len(self.x_data)
def __getitem__(self, idx):
preds = self.x_data[idx]
trgts = self.y_data[idx]
return preds, trgts # as a Tuple
# -----------------------------------------------------------
class Net(T.nn.Module):
def __init__(self):
super(Net, self).__init__()
self.hid1 = T.nn.Linear(6, 10) # 6-(10-10)-3
self.hid2 = T.nn.Linear(10, 10)
self.oupt = T.nn.Linear(10, 3)
T.nn.init.xavier_uniform_(self.hid1.weight)
T.nn.init.zeros_(self.hid1.bias)
T.nn.init.xavier_uniform_(self.hid2.weight)
T.nn.init.zeros_(self.hid2.bias)
T.nn.init.xavier_uniform_(self.oupt.weight)
T.nn.init.zeros_(self.oupt.bias)
def forward(self, x):
z = T.tanh(self.hid1(x))
z = T.tanh(self.hid2(z))
z = T.log_softmax(self.oupt(z), dim=1) # NLLLoss()
return z
# -----------------------------------------------------------
def accuracy(model, ds):
# assumes model.eval()
# item-by-item version
n_correct = 0; n_wrong = 0
for i in range(len(ds)):
X = ds[i][0].reshape(1,-1) # make it a batch
Y = ds[i][1].reshape(1) # 0 1 or 2, 1D
with T.no_grad():
oupt = model(X) # logits form
big_idx = T.argmax(oupt) # 0 or 1 or 2
if big_idx == Y:
n_correct += 1
else:
n_wrong += 1
acc = (n_correct * 1.0) / (n_correct + n_wrong)
return acc
# -----------------------------------------------------------
def accuracy_quick(model, dataset):
# assumes model.eval()
X = dataset[0:len(dataset)][0]
# Y = T.flatten(dataset[0:len(dataset)][1])
Y = dataset[0:len(dataset)][1]
with T.no_grad():
oupt = model(X) # [40,3] logits
# (_, arg_maxs) = T.max(oupt, dim=1)
arg_maxs = T.argmax(oupt, dim=1) # argmax() is new
num_correct = T.sum(Y==arg_maxs)
acc = (num_correct * 1.0 / len(dataset))
return acc.item()
# -----------------------------------------------------------
def show_acc(model, dataset, n_classes):
counts = np.zeros(n_classes, dtype=np.int64)
for i in range(len(dataset)):
trgt = dataset[i][1].item()
counts[trgt] += 1
X = dataset[0:len(dataset)][0] # all X values
Y = dataset[0:len(dataset)][1] # all Y values
with T.no_grad():
oupt = model(X) # [40,3] all logits
for c in range(n_classes):
idxs = np.where(Y==c) # indices where Y is c
logits_c = oupt[idxs] # logits corresponding to Y == c
arg_maxs_c = T.argmax(logits_c, dim=1) # predicted class
num_correct = T.sum(arg_maxs_c == c)
acc_c = num_correct.item() / len(arg_maxs_c)
print("class: %3d | count = %3d accuracy = %0.4f " % \
(c, counts[c], acc_c))
# -----------------------------------------------------------
def confusion_matrix_multi(model, ds, n_classes):
# n_classes must be 3 or greater
cm = np.zeros((n_classes,n_classes), dtype=np.int64)
for i in range(len(ds)):
X = ds[i][0].reshape(1,-1) # make it a batch
Y = ds[i][1].reshape(1) # actual class 0 1 or 2, 1D
with T.no_grad():
oupt = model(X) # logits form
pred_class = T.argmax(oupt) # 0,1,2
cm[Y][pred_class] += 1
return cm
# -----------------------------------------------------------
def show_confusion(cm):
dim = len(cm)
mx = np.max(cm) # largest count in cm
wid = len(str(mx)) + 1 # width to print
fmt = "%" + str(wid) + "d" # like "%3d"
for i in range(dim):
print("actual ", end="")
print("%3d:" % i, end="")
for j in range(dim):
print(fmt % cm[i][j], end="")
print("")
print("------------")
print("predicted ", end="")
for j in range(dim):
print(fmt % j, end="")
print("")
# -----------------------------------------------------------
def show_precision_recall(model, dataset, n_classes):
# accuracy = (TP + TN) / N
# precision = TP / (TP + FP)
# recall = TP / (TP + FN)
# F1 = 2 / [(1 / precision) + (1 / recall)]
for c in range(n_classes):
# assume class c is positive, other classes are negative
tp = 0; tn = 0; fp = 0; fn = 0
for i in range(len(dataset)):
X = dataset[i][0].reshape(1,-1) # make it a batch
trgt = dataset[i][1].reshape(1) # 0 1 or 2, 1D
with T.no_grad():
oupt = model(X) # logits form
pred = T.argmax(oupt)
if trgt == c and pred == trgt:
tp += 1 # correct pred class c
elif trgt != c and pred == trgt:
tn += 1 # correct pred for not class c
elif trgt != c and pred != trgt:
fn += 1 # FN is incorrect pred for not class c
elif trgt == c and pred != trgt:
fp += 1 # FP is incorrect pred for class c
if tp + fp == 0:
precision_c = 0.0
else:
precision_c = (1.0 * tp) / (tp + fp) # tp + fp != 0
if tp + fn == 0:
recall_c = 0.0
else:
recall_c = (1.0 * tp) / (tp + fn) # tp + fn != 0
if precision_c == 0.0 or recall_c == 0.0:
f1_c = 0.0
else:
f1_c = 2.0 / ((1.0 / precision_c) + (1.0 / recall_c))
# print(tp, tn, fp, fn)
print("\nClass: " + str(c))
print("precision = %0.4f " % precision_c)
print("recall = %0.4f " % recall_c)
print("F1 = %0.4f " % f1_c)
# -----------------------------------------------------------
def show_precision_recall_x(model, dataset, n_classes):
# slightly more efficient but less clear version
# accuracy = (TP + TN) / N
# precision = TP / (TP + FP)
# recall = TP / (TP + FN)
# F1 = 2 / [(1 / precision) + (1 / recall)]
counts = np.zeros((n_classes, 4), dtype=np.int64)
# cols = [TP, TN, FP, FN]
for i in range(len(dataset)):
X = dataset[i][0].reshape(1,-1) # make it a batch
trgt = dataset[i][1].reshape(1) # 0 1 or 2, 1D
with T.no_grad():
oupt = model(X) # logits form
pred = T.argmax(oupt)
for c in range(n_classes):
if trgt == c and pred == trgt:
counts[c][0] += 1 # TP correct pred class c
elif trgt != c and pred == trgt:
counts[c][1] += 1 # # TN correct pred not class c
elif trgt == c and pred != trgt:
counts[c][2] += 1 # # FP incorrect pred class c
elif trgt != c and pred != trgt:
counts[c][3] += 1 # # FN incorrect pred not class c
for c in range(n_classes):
tp = counts[c][0]
tn = counts[c][1]
fp = counts[c][2]
fn = counts[c][3]
if tp + fp == 0:
precision_c = 0.0
else:
precision_c = tp / (tp + fp) # tp + fp != 0
if tp + fn == 0:
recall_c = 0.0
else:
recall_c = tp / (tp + fn) # tp + fn != 0
if precision_c == 0.0 or recall_c == 0.0:
f1_c = 0.0
else:
f1_c = 2.0 / ((1.0 / precision_c) + (1.0 / recall_c))
# print(tp, tn, fp, fn)
print("\nClass: " + str(c))
print("precision = %0.4f " % precision_c)
print("recall = %0.4f " % recall_c)
print("F1 = %0.4f " % f1_c)
# -----------------------------------------------------------
def main():
# 0. get started
print("\nBegin People predict politics type ")
T.manual_seed(1)
np.random.seed(1)
# 1. create DataLoader objects
print("\nCreating People Datasets ")
train_file = ".\\Data\\people_train.txt"
train_ds = PeopleDataset(train_file) # 200 rows
test_file = ".\\Data\\people_test.txt"
test_ds = PeopleDataset(test_file) # 40 rows
bat_size = 10
train_ldr = T.utils.data.DataLoader(train_ds,
batch_size=bat_size, shuffle=True)
# -----------------------------------------------------------
# 2. create network
print("\nCreating 6-(10-10)-3 neural network ")
net = Net().to(device)
net.train() # set mode
# -----------------------------------------------------------
# 3. train model
max_epochs = 1000
ep_log_interval = 200
lrn_rate = 0.01
loss_func = T.nn.NLLLoss() # assumes log_softmax()
optimizer = T.optim.SGD(net.parameters(), lr=lrn_rate)
print("\nbat_size = %3d " % bat_size)
print("loss = " + str(loss_func))
print("optimizer = SGD")
print("max_epochs = %3d " % max_epochs)
print("lrn_rate = %0.3f " % lrn_rate)
print("\nStarting training")
for epoch in range(0, max_epochs):
# T.manual_seed(epoch+1) # checkpoint reproducibility
epoch_loss = 0 # for one full epoch
for (batch_idx, batch) in enumerate(train_ldr):
X = batch[0] # inputs
Y = batch[1] # correct class/label/politics
optimizer.zero_grad()
oupt = net(X)
loss_val = loss_func(oupt, Y) # a tensor
epoch_loss += loss_val.item() # accumulate
loss_val.backward()
optimizer.step()
if epoch % ep_log_interval == 0:
print("epoch = %5d | loss = %10.4f" % \
(epoch, epoch_loss))
print("Done ")
# -----------------------------------------------------------
# 4. evaluate model accuracy
print("\nComputing model accuracy")
net.eval()
acc_train = accuracy(net, train_ds) # item-by-item
print("Accuracy on training data = %0.4f" % acc_train)
acc_test = accuracy_quick(net, test_ds) # all at once
print("Accuracy on test data = %0.4f" % acc_test)
# 4b. accuracy by class
print("\nAccuracy on test by class: ")
show_acc(net, test_ds, 3)
# 4c. confusion matrix
print("\nConfusion matrix for test data: ")
cm = confusion_matrix_multi(net, test_ds, n_classes=3)
show_confusion(cm)
# 4d. precision and recall
print("\nPrecision, recall, F1 on test ")
show_precision_recall(net, test_ds, 3)
# 5. make a prediction
print("\nPredict politics for M 30 oklahoma $50,000: ")
X = np.array([[-1, 0.30, 0,0,1, 0.5000]],
dtype=np.float32)
X = T.tensor(X, dtype=T.float32).to(device)
with T.no_grad():
logits = net(X) # do not sum to 1.0
probs = T.exp(logits) # sum to 1.0
probs = probs.numpy() # numpy vector prints better
np.set_printoptions(precision=4, suppress=True)
print(probs)
# 6. save model (state_dict approach)
print("\nSaving trained model state")
# fn = ".\\Models\\people_model.pt"
# T.save(net.state_dict(), fn)
# saved_model = Net() # requires class definintion
# saved_model.load_state_dict(T.load(fn))
# use saved_model to make prediction(s)
print("\nEnd People predict politics demo")
if __name__ == "__main__":
main()
Training data.
# people_train.txt # sex (M=-1, F=1) age state (michigan, # nebraska, oklahoma) income # politics (consrvative, moderate, liberal) # 1, 0.24, 1, 0, 0, 0.2950, 2 -1, 0.39, 0, 0, 1, 0.5120, 1 1, 0.63, 0, 1, 0, 0.7580, 0 -1, 0.36, 1, 0, 0, 0.4450, 1 1, 0.27, 0, 1, 0, 0.2860, 2 1, 0.50, 0, 1, 0, 0.5650, 1 1, 0.50, 0, 0, 1, 0.5500, 1 -1, 0.19, 0, 0, 1, 0.3270, 0 1, 0.22, 0, 1, 0, 0.2770, 1 -1, 0.39, 0, 0, 1, 0.4710, 2 1, 0.34, 1, 0, 0, 0.3940, 1 -1, 0.22, 1, 0, 0, 0.3350, 0 1, 0.35, 0, 0, 1, 0.3520, 2 -1, 0.33, 0, 1, 0, 0.4640, 1 1, 0.45, 0, 1, 0, 0.5410, 1 1, 0.42, 0, 1, 0, 0.5070, 1 -1, 0.33, 0, 1, 0, 0.4680, 1 1, 0.25, 0, 0, 1, 0.3000, 1 -1, 0.31, 0, 1, 0, 0.4640, 0 1, 0.27, 1, 0, 0, 0.3250, 2 1, 0.48, 1, 0, 0, 0.5400, 1 -1, 0.64, 0, 1, 0, 0.7130, 2 1, 0.61, 0, 1, 0, 0.7240, 0 1, 0.54, 0, 0, 1, 0.6100, 0 1, 0.29, 1, 0, 0, 0.3630, 0 1, 0.50, 0, 0, 1, 0.5500, 1 1, 0.55, 0, 0, 1, 0.6250, 0 1, 0.40, 1, 0, 0, 0.5240, 0 1, 0.22, 1, 0, 0, 0.2360, 2 1, 0.68, 0, 1, 0, 0.7840, 0 -1, 0.60, 1, 0, 0, 0.7170, 2 -1, 0.34, 0, 0, 1, 0.4650, 1 -1, 0.25, 0, 0, 1, 0.3710, 0 -1, 0.31, 0, 1, 0, 0.4890, 1 1, 0.43, 0, 0, 1, 0.4800, 1 1, 0.58, 0, 1, 0, 0.6540, 2 -1, 0.55, 0, 1, 0, 0.6070, 2 -1, 0.43, 0, 1, 0, 0.5110, 1 -1, 0.43, 0, 0, 1, 0.5320, 1 -1, 0.21, 1, 0, 0, 0.3720, 0 1, 0.55, 0, 0, 1, 0.6460, 0 1, 0.64, 0, 1, 0, 0.7480, 0 -1, 0.41, 1, 0, 0, 0.5880, 1 1, 0.64, 0, 0, 1, 0.7270, 0 -1, 0.56, 0, 0, 1, 0.6660, 2 1, 0.31, 0, 0, 1, 0.3600, 1 -1, 0.65, 0, 0, 1, 0.7010, 2 1, 0.55, 0, 0, 1, 0.6430, 0 -1, 0.25, 1, 0, 0, 0.4030, 0 1, 0.46, 0, 0, 1, 0.5100, 1 -1, 0.36, 1, 0, 0, 0.5350, 0 1, 0.52, 0, 1, 0, 0.5810, 1 1, 0.61, 0, 0, 1, 0.6790, 0 1, 0.57, 0, 0, 1, 0.6570, 0 -1, 0.46, 0, 1, 0, 0.5260, 1 -1, 0.62, 1, 0, 0, 0.6680, 2 1, 0.55, 0, 0, 1, 0.6270, 0 -1, 0.22, 0, 0, 1, 0.2770, 1 -1, 0.50, 1, 0, 0, 0.6290, 0 -1, 0.32, 0, 1, 0, 0.4180, 1 -1, 0.21, 0, 0, 1, 0.3560, 0 1, 0.44, 0, 1, 0, 0.5200, 1 1, 0.46, 0, 1, 0, 0.5170, 1 1, 0.62, 0, 1, 0, 0.6970, 0 1, 0.57, 0, 1, 0, 0.6640, 0 -1, 0.67, 0, 0, 1, 0.7580, 2 1, 0.29, 1, 0, 0, 0.3430, 2 1, 0.53, 1, 0, 0, 0.6010, 0 -1, 0.44, 1, 0, 0, 0.5480, 1 1, 0.46, 0, 1, 0, 0.5230, 1 -1, 0.20, 0, 1, 0, 0.3010, 1 -1, 0.38, 1, 0, 0, 0.5350, 1 1, 0.50, 0, 1, 0, 0.5860, 1 1, 0.33, 0, 1, 0, 0.4250, 1 -1, 0.33, 0, 1, 0, 0.3930, 1 1, 0.26, 0, 1, 0, 0.4040, 0 1, 0.58, 1, 0, 0, 0.7070, 0 1, 0.43, 0, 0, 1, 0.4800, 1 -1, 0.46, 1, 0, 0, 0.6440, 0 1, 0.60, 1, 0, 0, 0.7170, 0 -1, 0.42, 1, 0, 0, 0.4890, 1 -1, 0.56, 0, 0, 1, 0.5640, 2 -1, 0.62, 0, 1, 0, 0.6630, 2 -1, 0.50, 1, 0, 0, 0.6480, 1 1, 0.47, 0, 0, 1, 0.5200, 1 -1, 0.67, 0, 1, 0, 0.8040, 2 -1, 0.40, 0, 0, 1, 0.5040, 1 1, 0.42, 0, 1, 0, 0.4840, 1 1, 0.64, 1, 0, 0, 0.7200, 0 -1, 0.47, 1, 0, 0, 0.5870, 2 1, 0.45, 0, 1, 0, 0.5280, 1 -1, 0.25, 0, 0, 1, 0.4090, 0 1, 0.38, 1, 0, 0, 0.4840, 0 1, 0.55, 0, 0, 1, 0.6000, 1 -1, 0.44, 1, 0, 0, 0.6060, 1 1, 0.33, 1, 0, 0, 0.4100, 1 1, 0.34, 0, 0, 1, 0.3900, 1 1, 0.27, 0, 1, 0, 0.3370, 2 1, 0.32, 0, 1, 0, 0.4070, 1 1, 0.42, 0, 0, 1, 0.4700, 1 -1, 0.24, 0, 0, 1, 0.4030, 0 1, 0.42, 0, 1, 0, 0.5030, 1 1, 0.25, 0, 0, 1, 0.2800, 2 1, 0.51, 0, 1, 0, 0.5800, 1 -1, 0.55, 0, 1, 0, 0.6350, 2 1, 0.44, 1, 0, 0, 0.4780, 2 -1, 0.18, 1, 0, 0, 0.3980, 0 -1, 0.67, 0, 1, 0, 0.7160, 2 1, 0.45, 0, 0, 1, 0.5000, 1 1, 0.48, 1, 0, 0, 0.5580, 1 -1, 0.25, 0, 1, 0, 0.3900, 1 -1, 0.67, 1, 0, 0, 0.7830, 1 1, 0.37, 0, 0, 1, 0.4200, 1 -1, 0.32, 1, 0, 0, 0.4270, 1 1, 0.48, 1, 0, 0, 0.5700, 1 -1, 0.66, 0, 0, 1, 0.7500, 2 1, 0.61, 1, 0, 0, 0.7000, 0 -1, 0.58, 0, 0, 1, 0.6890, 1 1, 0.19, 1, 0, 0, 0.2400, 2 1, 0.38, 0, 0, 1, 0.4300, 1 -1, 0.27, 1, 0, 0, 0.3640, 1 1, 0.42, 1, 0, 0, 0.4800, 1 1, 0.60, 1, 0, 0, 0.7130, 0 -1, 0.27, 0, 0, 1, 0.3480, 0 1, 0.29, 0, 1, 0, 0.3710, 0 -1, 0.43, 1, 0, 0, 0.5670, 1 1, 0.48, 1, 0, 0, 0.5670, 1 1, 0.27, 0, 0, 1, 0.2940, 2 -1, 0.44, 1, 0, 0, 0.5520, 0 1, 0.23, 0, 1, 0, 0.2630, 2 -1, 0.36, 0, 1, 0, 0.5300, 2 1, 0.64, 0, 0, 1, 0.7250, 0 1, 0.29, 0, 0, 1, 0.3000, 2 -1, 0.33, 1, 0, 0, 0.4930, 1 -1, 0.66, 0, 1, 0, 0.7500, 2 -1, 0.21, 0, 0, 1, 0.3430, 0 1, 0.27, 1, 0, 0, 0.3270, 2 1, 0.29, 1, 0, 0, 0.3180, 2 -1, 0.31, 1, 0, 0, 0.4860, 1 1, 0.36, 0, 0, 1, 0.4100, 1 1, 0.49, 0, 1, 0, 0.5570, 1 -1, 0.28, 1, 0, 0, 0.3840, 0 -1, 0.43, 0, 0, 1, 0.5660, 1 -1, 0.46, 0, 1, 0, 0.5880, 1 1, 0.57, 1, 0, 0, 0.6980, 0 -1, 0.52, 0, 0, 1, 0.5940, 1 -1, 0.31, 0, 0, 1, 0.4350, 1 -1, 0.55, 1, 0, 0, 0.6200, 2 1, 0.50, 1, 0, 0, 0.5640, 1 1, 0.48, 0, 1, 0, 0.5590, 1 -1, 0.22, 0, 0, 1, 0.3450, 0 1, 0.59, 0, 0, 1, 0.6670, 0 1, 0.34, 1, 0, 0, 0.4280, 2 -1, 0.64, 1, 0, 0, 0.7720, 2 1, 0.29, 0, 0, 1, 0.3350, 2 -1, 0.34, 0, 1, 0, 0.4320, 1 -1, 0.61, 1, 0, 0, 0.7500, 2 1, 0.64, 0, 0, 1, 0.7110, 0 -1, 0.29, 1, 0, 0, 0.4130, 0 1, 0.63, 0, 1, 0, 0.7060, 0 -1, 0.29, 0, 1, 0, 0.4000, 0 -1, 0.51, 1, 0, 0, 0.6270, 1 -1, 0.24, 0, 0, 1, 0.3770, 0 1, 0.48, 0, 1, 0, 0.5750, 1 1, 0.18, 1, 0, 0, 0.2740, 0 1, 0.18, 1, 0, 0, 0.2030, 2 1, 0.33, 0, 1, 0, 0.3820, 2 -1, 0.20, 0, 0, 1, 0.3480, 0 1, 0.29, 0, 0, 1, 0.3300, 2 -1, 0.44, 0, 0, 1, 0.6300, 0 -1, 0.65, 0, 0, 1, 0.8180, 0 -1, 0.56, 1, 0, 0, 0.6370, 2 -1, 0.52, 0, 0, 1, 0.5840, 1 -1, 0.29, 0, 1, 0, 0.4860, 0 -1, 0.47, 0, 1, 0, 0.5890, 1 1, 0.68, 1, 0, 0, 0.7260, 2 1, 0.31, 0, 0, 1, 0.3600, 1 1, 0.61, 0, 1, 0, 0.6250, 2 1, 0.19, 0, 1, 0, 0.2150, 2 1, 0.38, 0, 0, 1, 0.4300, 1 -1, 0.26, 1, 0, 0, 0.4230, 0 1, 0.61, 0, 1, 0, 0.6740, 0 1, 0.40, 1, 0, 0, 0.4650, 1 -1, 0.49, 1, 0, 0, 0.6520, 1 1, 0.56, 1, 0, 0, 0.6750, 0 -1, 0.48, 0, 1, 0, 0.6600, 1 1, 0.52, 1, 0, 0, 0.5630, 2 -1, 0.18, 1, 0, 0, 0.2980, 0 -1, 0.56, 0, 0, 1, 0.5930, 2 -1, 0.52, 0, 1, 0, 0.6440, 1 -1, 0.18, 0, 1, 0, 0.2860, 1 -1, 0.58, 1, 0, 0, 0.6620, 2 -1, 0.39, 0, 1, 0, 0.5510, 1 -1, 0.46, 1, 0, 0, 0.6290, 1 -1, 0.40, 0, 1, 0, 0.4620, 1 -1, 0.60, 1, 0, 0, 0.7270, 2 1, 0.36, 0, 1, 0, 0.4070, 2 1, 0.44, 1, 0, 0, 0.5230, 1 1, 0.28, 1, 0, 0, 0.3130, 2 1, 0.54, 0, 0, 1, 0.6260, 0
Test data.
# people_test.txt # -1, 0.51, 1, 0, 0, 0.6120, 1 -1, 0.32, 0, 1, 0, 0.4610, 1 1, 0.55, 1, 0, 0, 0.6270, 0 1, 0.25, 0, 0, 1, 0.2620, 2 1, 0.33, 0, 0, 1, 0.3730, 2 -1, 0.29, 0, 1, 0, 0.4620, 0 1, 0.65, 1, 0, 0, 0.7270, 0 -1, 0.43, 0, 1, 0, 0.5140, 1 -1, 0.54, 0, 1, 0, 0.6480, 2 1, 0.61, 0, 1, 0, 0.7270, 0 1, 0.52, 0, 1, 0, 0.6360, 0 1, 0.30, 0, 1, 0, 0.3350, 2 1, 0.29, 1, 0, 0, 0.3140, 2 -1, 0.47, 0, 0, 1, 0.5940, 1 1, 0.39, 0, 1, 0, 0.4780, 1 1, 0.47, 0, 0, 1, 0.5200, 1 -1, 0.49, 1, 0, 0, 0.5860, 1 -1, 0.63, 0, 0, 1, 0.6740, 2 -1, 0.30, 1, 0, 0, 0.3920, 0 -1, 0.61, 0, 0, 1, 0.6960, 2 -1, 0.47, 0, 0, 1, 0.5870, 1 1, 0.30, 0, 0, 1, 0.3450, 2 -1, 0.51, 0, 0, 1, 0.5800, 1 -1, 0.24, 1, 0, 0, 0.3880, 1 -1, 0.49, 1, 0, 0, 0.6450, 1 1, 0.66, 0, 0, 1, 0.7450, 0 -1, 0.65, 1, 0, 0, 0.7690, 0 -1, 0.46, 0, 1, 0, 0.5800, 0 -1, 0.45, 0, 0, 1, 0.5180, 1 -1, 0.47, 1, 0, 0, 0.6360, 0 -1, 0.29, 1, 0, 0, 0.4480, 0 -1, 0.57, 0, 0, 1, 0.6930, 2 -1, 0.20, 1, 0, 0, 0.2870, 2 -1, 0.35, 1, 0, 0, 0.4340, 1 -1, 0.61, 0, 0, 1, 0.6700, 2 -1, 0.31, 0, 0, 1, 0.3730, 1 1, 0.18, 1, 0, 0, 0.2080, 2 1, 0.26, 0, 0, 1, 0.2920, 2 -1, 0.28, 1, 0, 0, 0.3640, 2 -1, 0.59, 0, 0, 1, 0.6940, 2
.NET Test Automation Recipes
Software Testing
SciPy Programming Succinctly
Keras Succinctly
R Programming
Visual Studio Live
Microsoft MLADS Conference
DevIntersection Conference
Machine Learning Week
Ai4 Conference
G2E Conference
iSC West Conference
You must be logged in to post a comment.