MNIST CNN optimizer comparison with tensorflow.keras

List of optimizers

• Adadelta
• Adagrad
• Adam
• Adamax
• Nadam
• RMSprop
• SGD

Libraries

import os, time
import numpy as np
import tensorflow as tf # version 1.14
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense
from tensorflow.keras.datasets import mnist
from tensorflow.keras.callbacks import TensorBoard

Prepare Dataset

(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = np.expand_dims(X_train, 3)
X_test = np.expand_dims(X_test, 3)

np.expand_dims : (60000, 28, 28) → (60000, 28, 28, 1) for input_shape in first layer of model

Model shape

Optimizer Comparison with CNN Model

optimizers = [
    'Adadelta',
    'Adagrad',
    'Adam',
    'Adamax',
    'Nadam',
    'RMSprop',
    'SGD'
]
for optimizer in optimizers:
    model = Sequential()
    model.add(Conv2D(16, (3,3), activation='relu', input_shape=X_train[0].shape))
    model.add(MaxPooling2D())
    model.add(Conv2D(16, (3,3), activation='relu'))
    model.add(MaxPooling2D())
    model.add(Flatten())
    model.add(Dense(10, activation='softmax'))
    model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    tensorboard = TensorBoard(os.path.join('log_test4',f'{optimizer}_{time.time()}'))
    model.fit(X_train, y_train, batch_size=32, epochs=5, callbacks=[tensorboard])

In terminal, activate tensorflow environment and go to root dir.

tensorboard --logdir log_test4

In the browser,

http://localhost:6006

Result

Adam = Nadam = RMSProp > Adamax > SGD > Adagrad >> Adadelta

Conclusion

Among various types of optimizer, I’ve confirmed RMSprop and Adam is really effective in terms of CNN sequentail model.