Making AI’s out of Steins;Gate characters

In the past couple of weeks, I’ve been working on a project to make characters of the visual novel Steins;Gate come to life through AI – the usage of certain machine learning algorithms. So far, I’ve looked into GPT-2 (AI/machine learning model that generates somewhat coherent text) and finetuned the 117M model from OpenAI for GPT-2 to the ingame lines of Faris, Okabe and Kurisu, creating a new model for each character that enables GPT-2 to behave like that character. GPT-2 and the programs to retrain it can be found on GitHub. The models can be tested through a Discord bot, whose source code is available on GitHub. It is currently running on my Discord server. If you’d like to test it on my Discord server, please send me an email to bot[at]futuregadgetlab.de

There’s more that can be done. Using Tacotron2, it seems possible that we can create a Text-To-Speech AI that uses the voice we choose and create! It has to be trained on many pairs of text lines & voice recordings of the desired voice. The Steins;Gate visual novel provides these pairs for their characters, making Text to Speech AI, that uses the voice of the ingame character Kurisu possible. This GitHub project of mine roughly outlines how to make the game content readable for GPT-2 and Tacotron2. The text-voice pairs generated for Tacotron2 have to be proofread, proofheard and corrected before training a Tacotron2 model on that dataset. To help do so, I have created TacoTranscribe. I still have to write a README for it to explain how to use it correctly.

However, since Kurisu has 2,8k text-voice pairs, I ask for your help in proofreading, proofhearing and correcting the pairs. If you’d like to help, please send me an email to the address above.

If we manage to couple the output of a Speech to Text system with GPT-2, and then couple GPT-2’s output with Tacotron2, we can make a bot that autocompletes your sentences and reads it in a way Kurisu would do.