This week I built two small generative models, each a network that learns to produce something new: a sentence about a photo, and a picture of clothing that does not exist. Both run as demos on Hugging Face, and their code is on GitHub.

A sentence for a photo

The caption generator is an encoder and a decoder. The encoder is InceptionV3, a network already trained to recognise the objects in ImageNet’s photos. With its last layer removed, it turns any image into a list of 2,048 numbers that describe what is in it. The decoder is an LSTM, a network that writes one word at a time: given those numbers and the words so far, it picks the next word, until the sentence ends.

It was trained on Flickr8k: 8,000 photos, 6,000 to learn from, 1,000 to check against during training and 1,000 for the final test, each with five captions written by people, 40,000 in all. Five per photo matters, because there is more than one right way to describe a picture, and the model gets to see that.

Clothes from noise

The fashion generator is a DCGAN, a generative adversarial network built from convolutions. Two networks train against each other. The generator takes 100 random numbers and grows them, through transposed convolutions, into a 28×28 picture. The discriminator looks at an image, real or generated, and says which it thinks it is. The generator gets better by fooling it.

It learned from Fashion-MNIST, small greyscale photos of T-shirts, trousers, trainers, bags and ankle boots, and after 50 epochs it draws new items of each kind. The idea behind it is data augmentation: where real examples are scarce or expensive to collect, a generator can add more.

Knowing it learned

A GAN’s losses do not fall towards zero the way most models’ do, so on their own they cannot tell you that training worked. I watched three things instead:

  • The discriminator’s accuracy settled at about 50 to 55 per cent. It was guessing, which means the generated pictures had become hard to tell from real ones.
  • Neither loss collapsed to zero. That would have meant one network had won and the other had stopped learning.
  • The spread of pixel brightness in the generated images closely matched the real ones, compared as density curves: a rough check that the model draws the whole range of the data, not just a few of its shapes.

The code for both is on GitHub: the caption generator and the fashion generator.