Dense Layer란?

• Fully Connected Layer, multi layer perceptron이라고도 불리며 뉴런끼리 모두 연결되어 있는 것을 말한다.
• 뉴런이 한 개만 증가해도 연쇄적으로 parameter의 개수가 늘어난다.

Pytorch로 구현하기

import torch
import torch.nn as nn

N, n_feature = 8, 10
X = torch.rand(size = (N, n_feature))

n_neuron = 3
dense = nn.Linear(in_features = n_feature, out_features = n_neuron)
Y = dense(X)
relu = nn.ReLU() # pytorch에서는 한번에 할 수 없고 따로 activateion을 해야한다.
Y = relu(Y)

W = dense.weight
B = dense.bias

print('=============Input/Weight/Bias============')
print("X : ",X.shape)
print("W : ",W.shape) # out_features x in_features
print("B : ",B.shape) # 1 x out_features
print("Y : ",y.shape)

print(B)
print(W.T) # transpose 해야 tf의 dense의 weight와 같은의미이다.

=============Input/Weight/Bias============
X : torch.Size([8, 10])
W : torch.Size([3, 10])
B : torch.Size([3])
Y : (8, 3)
Parameter containing:
tensor([0.1751, 0.2799, 0.2905], requires_grad=True)
tensor([[-0.1762, -0.1470, 0.0741],
[-0.1600, -0.1453, -0.2921],
[ 0.0579, -0.1740, -0.2142],
[ 0.2837, 0.3077, 0.1427],
[ 0.2793, -0.1423, -0.2961],
[ 0.0388, 0.0185, 0.1269],
[ 0.1719, 0.2887, -0.1673],
[-0.1791, 0.0566, -0.3136],
[-0.0520, -0.2398, -0.0354],
[-0.2688, -0.1188, -0.1965]], grad_fn=<PermuteBackward0>)

• W.T를 해야 기존 tf의 코드와 같아진다는 것에 유의해야한다.

Tensorflow로 구현하기

import tensorflow as tf

from tensorflow.keras.layers import Dense


N, n_feature = 8,10
X = tf.random.normal(shape = (N, n_feature))

n_neuron =3
dense = Dense(units = n_neuron, activation = 'sigmoid')
y = dense(X)

W, B = dense.get_weights()

print('=============Input/Weight/Bias============')
print("X : ",X.shape)
print("W : ",W.shape)
print("B : ",B.shape)
print("Y : ",y.shape)

print(B)
print(W)

Pytorch, matmul, dot product의 결과가 같은지 확인 구현

import numpy as np
import torch

from torch.linalg import matmul
import torch.nn as nn

N, n_neuron = 4,10
X = torch.rand(size = (N, n_feature))

n_neuron = 3
dense = nn.Linear(in_features = n_feature, out_features = n_neuron)
Y_pt = dense(X)
sigmoid = nn.Sigmoid() # pytorch에서는 한번에 할 수 없고 따로 activateion을 해야한다.
Y_pt = sigmoid(Y_pt)

W = dense.weight.T
B = dense.bias

print("Y(Pytorch): \n", Y_pt.detach().numpy())

# calculate with matrix multiplication
z = matmul(X, W) + B
Y_man_matmul = 1/(1+torch.exp(-z))
print("Y(with matrix multiplication: \n", Y_man_matmul.detach().numpy())

# calculate with dot products
Y_man_vec = np.zeros(shape=(N, n_neuron))
for x_idx in range(N):
  x = X[x_idx]
  for nu_idx in range(n_neuron):
    w, b = W[:, nu_idx], B[nu_idx]
    z = torch.dot(x, w) + b
    a = 1 / (1 + np.exp(-z.detach().numpy()))
    Y_man_vec[x_idx, nu_idx] = a

print("Y(with dot products): \n", Y_man_vec)

Y(Pytorch):
[[0.64615285 0.533632 0.62989026][0.58182204 0.5037469 0.6478663 ]
[0.5675521 0.51351106 0.69402564][0.5874338 0.5004672 0.625019 ]]
Y(with matrix multiplication:
[[0.64615285 0.533632 0.62989026][0.58182204 0.5037469 0.6478663 ]
[0.5675521 0.51351106 0.69402564][0.5874338 0.5004672 0.625019 ]]
Y(with dot products):
[[0.64615283 0.53363195 0.62989031][0.58182206 0.50374695 0.64786634]
[0.56755209 0.51351109 0.6940256 ][0.58743379 0.50046723 0.62501899]]

• 모두 같은 결과가 나온다는 것으로 보아 연산이 잘 수행되고 있다는 것을 확인할 수 있다.