EX_09_Logistic_Regression_NumPy

Logistic Regression¶

[예제 9] Logistic Regression(NumPy)¶

Load modules¶

In [ ]:

import numpy as np
import matplotlib.pyplot as plt

print("NumPy Version :{}".format(np.__version__))
print("Matplotlib Version :{}".format(plt.matplotlib.__version__))

NumPy Version :1.24.3
Matplotlib Version :3.7.1

Input and Label¶

In [ ]:

# Logistic regression : Binary Classification data
x_input = np.array([[1, 1], [2, 1], [1, 2], [0.5, 4], [4, 1], [2.5, 2.3]], dtype= np.float32)
labels = np.array([[0], [0], [0], [1], [1], [1]], dtype= np.float32)

# Weight, Bias
W = np.random.normal(size=(2,1))
B = np.random.normal(size=(1,))

Activation Function : Sigmoid Function¶
$\sigma(x) = \frac{1}{1+e^{-x}}$

Hypothesis : Logistic Equation¶
$H(x) = \sigma(XW + b$)¶

In [ ]:

# Activate Function: Sigmoid Function
def Sigmoid(x):
    return 1 / (1+np.exp(-x))

# Hypothesis
def Hypothesis(x):
    return Sigmoid(np.matmul(x, W) + B)

Cost Function : Cross Entropy Error¶
$cost(W,b) = \frac{1}{m}\sum_{i=1}^{m}t\log{H(x_{i}})+(1-t)\log{(1-H(x_{i}}))$¶

In [ ]:

# Cost Function: Cross Entropy Error
def Cost():
    return -np.mean(labels*np.log(Hypothesis(x_input)) + (1-labels)*np.log(1-Hypothesis(x_input)))

Gradient¶
$\frac{d}{dx}f(x) = \lim_{\delta = 0} \frac{f(x+\delta) - f(x-\delta)}{2\delta}$¶

In [ ]:

# Gradient
def Gradient():
    global W, B
    pres_W = W.copy()
    grad_W = np.zeros_like(W)
    pres_B = B.copy()
    grad_B = np.zeros_like(B)    
    delta = 5e-7

    for idx in range(W.size):
        W[idx,0] = pres_W[idx,0] + delta
        cost_p = Cost()
        W[idx,0] = pres_W[idx,0] - delta
        cost_m = Cost()
        grad_W[idx,0] = (cost_p-cost_m)/(2*delta)
        W[idx,0] = pres_W[idx,0]

    for idx in range(B.size):
        B[idx] = pres_B[idx] + delta
        cost_p = Cost()
        B[idx] = pres_B[idx] - delta
        cost_m = Cost()
        grad_B[idx] = (cost_p-cost_m)/(2*delta)
        B[idx] = pres_B[idx]

    return grad_W, grad_B

학습 준비 과정¶

Training¶
$\mu$ : Learning rate¶
$W = W - \mu\frac{\partial}{\partial W}cost(W, b)$¶
$b = b - \mu\frac{\partial}{\partial b}cost(W, b)$¶

In [ ]:

%%time

# Parameter Set
epochs = 20000
learning_rate = 0.05

training_idx = np.arange(0, epochs+1, 1)
cost_graph = np.zeros(epochs+1)
check = np.array([0, epochs*0.01, epochs*0.08, epochs*0.2, epochs*0.4, epochs])

w_trained = []
b_trained = []
w_trained.append(W.copy())
b_trained.append(B.copy())
check_idx = 1

# 학습 (Training)
for cnt in range(0, epochs+1):
    cost_graph[cnt] = Cost()
    if cnt % (epochs//20) == 0:
        print("[{:>6}] cost = {:>10.4}, w = [{:>7.4} {:>7.4}], b = {:>7.4}".format(cnt, cost_graph[cnt], W[0,0], W[1,0], B[0]))
    if check[check_idx] == cnt:
        w_trained.append(W.copy())
        b_trained.append(B.copy())
        check_idx += 1

    grad_W, grad_B = Gradient()
    W -= learning_rate * grad_W
    B -= learning_rate* grad_B

[     0] cost =      2.297, w = [-0.6338  -1.365], b =  0.3261
[  1000] cost =     0.3082, w = [  0.958   1.124], b =  -3.586
[  2000] cost =     0.1874, w = [  1.507   1.746], b =  -5.874
[  3000] cost =     0.1343, w = [  1.873    2.16], b =  -7.398
[  4000] cost =     0.1045, w = [  2.148   2.471], b =  -8.543
[  5000] cost =    0.08549, w = [  2.367    2.72], b =  -9.459
[  6000] cost =    0.07226, w = [  2.551   2.927], b =  -10.22
[  7000] cost =    0.06254, w = [  2.708   3.105], b =  -10.88
[  8000] cost =    0.05511, w = [  2.845   3.261], b =  -11.45
[  9000] cost =    0.04924, w = [  2.967   3.399], b =  -11.96
[ 10000] cost =    0.04449, w = [  3.077   3.524], b =  -12.42
[ 11000] cost =    0.04057, w = [  3.177   3.637], b =  -12.83
[ 12000] cost =    0.03728, w = [  3.268   3.741], b =  -13.21
[ 13000] cost =    0.03448, w = [  3.353   3.837], b =  -13.57
[ 14000] cost =    0.03207, w = [  3.431   3.926], b =  -13.89
[ 15000] cost =    0.02998, w = [  3.504   4.009], b =   -14.2
[ 16000] cost =    0.02813, w = [  3.573   4.086], b =  -14.48
[ 17000] cost =     0.0265, w = [  3.637    4.16], b =  -14.75
[ 18000] cost =    0.02505, w = [  3.698   4.229], b =   -15.0
[ 19000] cost =    0.02375, w = [  3.755   4.294], b =  -15.24
[ 20000] cost =    0.02258, w = [   3.81   4.356], b =  -15.47
CPU times: total: 3.59 s
Wall time: 3.52 s

In [ ]:

print("[Training Test]")

H_x = Hypothesis(x_input)
H_x = H_x.reshape((-1,))
H = [int(h>0.5) for h in H_x]
for idx in range(x_input.shape[0]):
    print("Input {} , Label : {} => H :{:>2}(H_x:{:>5.2})".format(x_input[idx], labels[idx], H[idx], H_x[idx]))

[Training Test]
Input [1. 1.] , Label : [0.] => H : 0(H_x:0.00067)
Input [2. 1.] , Label : [0.] => H : 0(H_x:0.029)
Input [1. 2.] , Label : [0.] => H : 0(H_x: 0.05)
Input [0.5 4. ] , Label : [1.] => H : 1(H_x: 0.98)
Input [4. 1.] , Label : [1.] => H : 1(H_x: 0.98)
Input [2.5 2.3] , Label : [1.] => H : 1(H_x: 0.98)

In [ ]:

print("\n[ Prediction by specific data ]")    
test_in = np.array([[1.5,1.5],[3.0,3.0],[4.0,2.0],[1.9,1.9]])
for i in test_in:
    print("input [{},{}] => Group {:.2f}".format(i[0],i[1],Hypothesis(i)[0]))

[ Prediction by specific data ]
input [1.5,1.5] => Group 0.04
input [3.0,3.0] => Group 1.00
input [4.0,2.0] => Group 1.00
input [1.9,1.9] => Group 0.51

Plotting¶

In [ ]:

# Training 상황에 대한 그래프 출력
# Training 회수 별 Cost 값
plt.title("'Cost / Epochs' Graph")
plt.xlabel("Epochs")
plt.ylabel("Cost")
plt.plot(training_idx, cost_graph)
plt.xlim(0, epochs)
plt.grid(True)
plt.semilogy()
plt.show()

No description has been provided for this image

In [ ]:

# 구분선 그리기
x_decision = np.linspace(0, 5, 1000)
fig, ax = plt.subplots(2, 3, figsize=(15, 11))
fig.suptitle("'Hypothesis / Training Count' Graph")

for ax_idx in range(check.size):
    W = w_trained[ax_idx]
    B = b_trained[ax_idx]
    y_decision = -(W[0] * x_decision + B[0])/W[1] 

    #   label의 값에 따라서 blue 또는 red 점 찍기
    for i in range(labels.shape[0]):
        if(labels[i][0] == 0):
            ax[ax_idx // 3][ax_idx % 3].scatter(x_input[i][0], x_input[i][1], color='blue')
        else:
            ax[ax_idx // 3][ax_idx % 3].scatter(x_input[i][0], x_input[i][1], color='red')
   
    ax[ax_idx // 3][ax_idx % 3].plot(x_decision, y_decision, label=' Decision Boundary', color='green')

    ax[ax_idx // 3][ax_idx % 3].set_title("Epochs : {}".format(check[ax_idx]))
    ax[ax_idx // 3][ax_idx % 3].set_xlim((0, 5))
    ax[ax_idx // 3][ax_idx % 3].set_ylim((0, 5))
    ax[ax_idx // 3][ax_idx % 3].set_xlabel("x0")
    ax[ax_idx // 3][ax_idx % 3].set_ylabel("x1")
    ax[ax_idx // 3][ax_idx % 3].grid(True)
    ax[ax_idx // 3][ax_idx % 3].legend()
    
plt.show()

In [ ]:

import numpy as np

import matplotlib.pyplot as plt

print("NumPy Version :{}".format(np.__version__))

print("Matplotlib Version :{}".format(plt.matplotlib.__version__))

# Logistic regression : Binary Classification data

x_input = np.array([[1, 1], [2, 1], [1, 2], [0.5, 4], [4, 1], [2.5, 2.3]], dtype= np.float32)

labels = np.array([[0], [0], [0], [1], [1], [1]], dtype= np.float32)

# Weight, Bias

W = np.random.normal(size=(2,1))

B = np.random.normal(size=(1,))

# Activate Function: Sigmoid Function

def Sigmoid(x):

return 1 / (1+np.exp(-x))

# Hypothesis

def Hypothesis(x):

return Sigmoid(np.matmul(x, W) + B)

# Cost Function: Cross Entropy Error

def Cost():

return -np.mean(labels*np.log(Hypothesis(x_input)) + (1-labels)*np.log(1-Hypothesis(x_input)))

# Gradient

def Gradient():

global W, B

pres_W = W.copy()

grad_W = np.zeros_like(W)

pres_B = B.copy()

grad_B = np.zeros_like(B)

delta = 5e-7

for idx in range(W.size):

W[idx,0] = pres_W[idx,0] + delta

cost_p = Cost()

W[idx,0] = pres_W[idx,0] - delta

cost_m = Cost()

grad_W[idx,0] = (cost_p-cost_m)/(2*delta)

W[idx,0] = pres_W[idx,0]

for idx in range(B.size):

B[idx] = pres_B[idx] + delta

cost_p = Cost()

B[idx] = pres_B[idx] - delta

cost_m = Cost()

grad_B[idx] = (cost_p-cost_m)/(2*delta)

B[idx] = pres_B[idx]

return grad_W, grad_B

# Parameter Set

epochs = 20000

learning_rate = 0.05

training_idx = np.arange(0, epochs+1, 1)

cost_graph = np.zeros(epochs+1)

check = np.array([0, epochs*0.01, epochs*0.08, epochs*0.2, epochs*0.4, epochs])

w_trained = []

b_trained = []

w_trained.append(W.copy())

b_trained.append(B.copy())

check_idx = 1

# 학습 (Training)

for cnt in range(0, epochs+1):

cost_graph[cnt] = Cost()

if cnt % (epochs//20) == 0:

print("[{:>6}] cost = {:>10.4}, w = [{:>7.4} {:>7.4}], b = {:>7.4}".format(cnt, cost_graph[cnt], W[0,0], W[1,0], B[0]))

if check[check_idx] == cnt:

w_trained.append(W.copy())

b_trained.append(B.copy())

check_idx += 1

grad_W, grad_B = Gradient()

W -= learning_rate * grad_W

B -= learning_rate* grad_B

print("[Training Test]")

H_x = Hypothesis(x_input)

H_x = H_x.reshape((-1,))

H = [int(h>0.5) for h in H_x]

for idx in range(x_input.shape[0]):

print("Input {} , Label : {} => H :{:>2}(H_x:{:>5.2})".format(x_input[idx], labels[idx], H[idx], H_x[idx]))

print("\n[ Prediction by specific data ]")

test_in = np.array([[1.5,1.5],[3.0,3.0],[4.0,2.0],[1.9,1.9]])

for i in test_in:

print("input [{},{}] => Group {:.2f}".format(i[0],i[1],Hypothesis(i)[0]))

# Training 상황에 대한 그래프 출력

# Training 회수 별 Cost 값

plt.title("'Cost / Epochs' Graph")

plt.xlabel("Epochs")

plt.ylabel("Cost")

plt.plot(training_idx, cost_graph)

plt.xlim(0, epochs)

plt.grid(True)

plt.semilogy()

plt.show()

# 구분선 그리기

x_decision = np.linspace(0, 5, 1000)

fig, ax = plt.subplots(2, 3, figsize=(15, 11))

fig.suptitle("'Hypothesis / Training Count' Graph")

for ax_idx in range(check.size):

W = w_trained[ax_idx]

B = b_trained[ax_idx]

y_decision = -(W[0] * x_decision + B[0])/W[1]

# label의 값에 따라서 blue 또는 red 점 찍기

for i in range(labels.shape[0]):

if(labels[i][0] == 0):

ax[ax_idx // 3][ax_idx % 3].scatter(x_input[i][0], x_input[i][1], color='blue')

else:

ax[ax_idx // 3][ax_idx % 3].scatter(x_input[i][0], x_input[i][1], color='red')

ax[ax_idx // 3][ax_idx % 3].plot(x_decision, y_decision, label=' Decision Boundary', color='green')

ax[ax_idx // 3][ax_idx % 3].set_title("Epochs : {}".format(check[ax_idx]))

ax[ax_idx // 3][ax_idx % 3].set_xlim((0, 5))

ax[ax_idx // 3][ax_idx % 3].set_ylim((0, 5))

ax[ax_idx // 3][ax_idx % 3].set_xlabel("x0")

ax[ax_idx // 3][ax_idx % 3].set_ylabel("x1")

ax[ax_idx // 3][ax_idx % 3].grid(True)

ax[ax_idx // 3][ax_idx % 3].legend()

plt.show()

'Tensorflow' 카테고리의 다른 글

Softmax classification code example numpy (1)	2023.12.31
Logistic Regression code numpy (1)	2023.12.31
Min_max_Scaling 이유 (0)	2023.12.28
Multi in Multi out blood pressure _Tensorflow (0)	2023.12.28
Multi variable Linear Regression (Numpy) (1)	2023.12.27

설화스터디 소프트웨어학과

Logistic Regression code

Logistic Regression¶

[예제 9] Logistic Regression(NumPy)¶

Load modules¶

Input and Label¶

Activation Function : Sigmoid Function¶

Hypothesis : Logistic Equation¶

$H(x) = \sigma(XW + b$)¶

Cost Function : Cross Entropy Error¶

$cost(W,b) = \frac{1}{m}\sum_{i=1}^{m}t\log{H(x_{i}})+(1-t)\log{(1-H(x_{i}}))$¶

Gradient¶

$\frac{d}{dx}f(x) = \lim_{\delta = 0} \frac{f(x+\delta) - f(x-\delta)}{2\delta}$¶

학습 준비 과정¶

Training¶

$\mu$ : Learning rate¶

$W = W - \mu\frac{\partial}{\partial W}cost(W, b)$¶

$b = b - \mu\frac{\partial}{\partial b}cost(W, b)$¶

Plotting¶

'Tensorflow' 카테고리의 다른 글

티스토리툴바

Logistic Regression code

Logistic Regression¶

[예제 9] Logistic Regression(NumPy)¶

Load modules¶

Input and Label¶

Activation Function : Sigmoid Function¶

Hypothesis : Logistic Equation¶

$H(x) = \sigma(XW + b$)¶

Cost Function : Cross Entropy Error¶

$cost(W,b) = \frac{1}{m}\sum_{i=1}^{m}t\log{H(x_{i}})+(1-t)\log{(1-H(x_{i}}))$¶

Gradient¶

$\frac{d}{dx}f(x) = \lim_{\delta = 0} \frac{f(x+\delta) - f(x-\delta)}{2\delta}$¶

학습 준비 과정¶

Training¶

$\mu$ : Learning rate¶

$W = W - \mu\frac{\partial}{\partial W}cost(W, b)$¶

$b = b - \mu\frac{\partial}{\partial b}cost(W, b)$¶

Plotting¶

'Tensorflow' 카테고리의 다른 글

관련글

티스토리툴바