I play video games but not for entertainment

I always used to think that video games are just for fun and never thought about them as anything more than just another form of entertainment.

Before getting into the process that eventually changed my mind, I want to give you a little introduction about myself and what I do. I am an AI researcher. I really love to work on research projects that have a direct impact on people’s lives. Having said that, I am currently working on an AI-based computer vision solution that aims to help blind people in one of their most crucial daily needs, the need of navigating through the environment, without getting hurt or run into obstacles.

I am not going to fully explain my project here but I am trying to point out the biggest problem that I had along the way and how using video games helped me.

Data is everything

A couple of years ago when I just started to inform myself about this topic, I didn’t know how much of a big hassle it was to develop a safe solution for this problem. But among all of the difficulties, the most unfortunate one was the absence of a high-quality dataset to work on.

The mentioned navigation system is supposed to work in outdoor environments (mostly in sidewalks) and there is no dataset of scenes of sidewalks out there. Especially when we are talking about AI and machine learning applications no matter how great your algorithm is or how smart your approach is, when there is not enough high-quality data, everything fails.

Thus, I needed some sort of dataset to work on and after a while, I started to wondering about collecting all the data that I need on my own. Then I confronted with two big problems:

1. the amount of time I needed to collect this dataset alone

2. the cost of annotation and labeling of that huge amount of data

These problems were big enough for me to try to think of an alternative. After a couple of days, I had already read a lot of papers that were mostly about self-driving cars or navigation systems built for drones or robots and I kept noticing a common pattern among those papers. Most of them had experienced the same problem that I had and their solution was to use simulation instead of working on real-world data. Using simulators, it is possible to basically generate a ton of data with zero cost and in no time.

There are a lot of simulators for developing autonomous vehicles such as:

• CARLA Simulator

• AirSim (by Microsoft)

• Udacity Car Simulator

• Deepdrive

• Unity SimViz

• TORCS (The Open Racing Car Simulator)

and a lot of indoor environment simulators and datasets for developing navigation systems for robots (aka embodied artificial intelligence):

• Habitat AI (by Facebook)

• MINOS

• Gibson

• Replica

• MatterPort3D

Video games are way better than open source simulators

Despite the existence of all of these simulators, my problem persisted. None of the above open source simulators were suitable for my use case. I needed a super-realistic simulation of the urban environment, something that exactly looks like reality and I couldn’t find it anywhere but in high budget commercial games. I found everything that I needed in GTA V, one of the best open-world action-adventure games released in the last decade.

These three papers have worked on GTA V and similar games to extract valuable datasets. They provide valuable insight into the process of extraction of data from commercial games.

• Playing for Data

• Playing for Benchmarks

• Free supervision from video games

By using solutions provided by these papers, we can extract not only RGB frames from the games but also a lot of useful ground truths such as instance segmentation, semantic segmentation, bounding boxes, albedo, depth map, and optical flow.

Tools you need for collecting data from games

There are a couple of open source libraries on GitHub that you can use to easily extract information from the GTA V and other video games. These are some of the most important ones:

• Gamehook

• DeepGTAV

• DeepGTAV PreSIL [Lidar]

• GTAVisionExport

Alongside these tools, you can also use Renderdoc and Nvidia Nsight Graphics to manually intercept GPU rendering pipeline and extract a lot of useful data from video games.

So far, we discovered how video games are used to synthesize data that are visually close to the real world. In the rest of this article, we discuss another important use case of video games in the world of machine learning.

Using video games for developing better machine learning algorithms

Video games are not only used for extracting realistic-looking datasets, but also for empowering state of the art machine learning algorithms and optimization techniques.

One of the fields of machine learning called Reinforcement Learning which was around for many years gained a lot of popularity when famous AlphaGo, an AI developed by a British company called DeepMind, played against the greatest Go player of the decade Mr. Lee Sedol, the winner of 18 world titles, and beat him 4 to 1 in the world championship back in 2016.

Reinforcement Learning is a type of machine learning technique that enables an agent to learn in an interactive environment (like a game environment) by trial and error using feedback from its own actions and experiences. Here is an explanation by Shweta Bhatt in a blog post published on kdnuggets.com which describes the relation of games and reinforcement learning in a nice clear way:

The association of reinforcement learning with games is so inevitable that a lot of big names of the industry such as DeepMind, Microsoft, OpenAI, and Unity have already developed their own open source gaming environments in order to contribute to the field. These are 4 of the popular environments developed by these companies:

• DeepMind Lab: https://deepmind.com/blog/article/open-sourcing-deepmind-lab

• Project Malmo: https://www.microsoft.com/en-us/research/project/project-malmo/

• OpenAI gym: https://gym.openai.com/

• Unity ML-Agents: https://unity3d.com/machine-learning

Gaming Industry has so many potentials

Artificial intelligence is evolving and progressing rapidly. In today's world, despite the availability of an unbelievably huge amount of data across dozens of industries, the need for high-quality data is being felt more than ever before. In the field of computer vision, there is still a huge demand for data that could be partially provided by video games and accurate simulation environments. The gaming industry is getting more and more advanced every single day and it is very interesting to think about what kind of advancements it could provide for the AI industry.