Author: Eylam Tagor

This directory contains data for the Type 3 game I imported to retro, Sonic Advance for the GameBoy Advance. Although retro uses game files solely from its own package directory (.../site-packages/retro/), I included a copy of this directory here so data such as done condition and reward function are easily visible.

This Python module contains various files I used in my exploration of Gym Retro:

• brute_modified.py: a modified version of retro.examples.brute, an implementation of "the Brute" algorithm that has been fitted to work with custom integrated games.

• interactive_modified.py: a modified version of retro.examples.interactive, which runs a ROM file with keyboard-bound controls. This script is used to verify that ROMs were imported successfully in Retro and can run as desired.

• platform_dists: this file contains a single function, gen_platform_dists, that is used in the first part of this study to calculate the distribution of the ~1000 Type 1 and 2 games' consoles.

This directory contains the necessary files for running the Integration UI tool that is used to integrate new games, play movies, define variables, etc. Similarly to custom_integrations, this folder was copied into the Retro package directory in order to run it correctly, but this copy was kept to highlight the progress of the study.

This directory contains some movies, or replays, of the Brute algorithm playing some games using Retro.

This directory contains the ROM files for Super Mario Bros (NES) and Sonic Advance (GBA), which were then imported into their respective game directories. You can see this for Sonic by finding custom_integrations/SonicAdvance-GbAdvance/rom.gba, which is the same file as roms/Sonic Advance (USA) (En,Ja).gba.

If someone wants to use any of the code in this project, this file makes sure they meet the language and dependency requirements.

This Jupyter Notebook contains the organized set of responses to the questions in this exercise.

Installing stable-retro:

• Problem: "failed building wheel"; installing wheel and a bunch of other packages using pip didn't work either.

• Solution: migrating entire project from Windows to WSL.

• Result: Successfully installed stable-retro.

Importing a Type 2 ROM for Super Mario Bros (NES):

• Problem: importing seemed to work once I put it in the right directory, but the game wasn't running with the interactive script.

• Solution: I realized X11 forwarding wasn't set up properly, so I updated it and tested it with WSL outside this project before coming back to this issue.

• Result: Successfully ran interactive script with Super Mario Bros.

Integrating a Type 3 game:

• Problem 1: stable-retro is missing an integration UI tool! Download links lead to NoSuchBucket dead ends.

• Solution 1: I found OpenAI's Gym Retro repository, extracted the executable and core files, and installed them over to this project.

• Problem 2a: I chose to integrate Spider-Man (Atari 2600), but ran into importing problems and concluded that it was a bad ROM source. I then found Sonic Advance (GBA) and encountered the same issue trying to import the ROM:

• Problem 2b: the integration tool isn't doing anything for any game, just staying a grey screen.

• Solution 2: googling around, I found some old documentation for OpenAI's Gym Retro that suggested that the executable needs to be located inside the Python package directory (in site-packages/retro). Copying it over there and running the new copy worked. Importing the ROM worked as well now, and I chose to proceed with Sonic over Spider-Man as it was more of a platformer game.

• Result: Successfully integrated a Type 3 game.

Resolving unexpected bug with Gym:

• Problem: at this point in the study, for some reason running any script on any game now failed with the following error:

  AttributeError: module 'gym.utils.seeding' has no attribute 'hash_seed'

• Solution: a Stack Overflow forum pointed out that hash seeding is deprecated in my Gym version (0.26.2), so downgrading to 0.25.2 got things up and running again (albeit with warnings).

• Result: Successfully ran the interactive script with Mario again.

Preparing Sonic Advanced for RL:

• While I did not run into a specific problem, preparing the game for defining its done condition and reward function was tedious.

• Example 1: I was only able to narrow down the time variable to around 180 different addresses whose values changed in unison, although some were in different scales than others. I ended up choosing the address I did because of its relatively small scale which translates better into reward functions (or in this case penalty).

• Example 2: In the game, the player starts with 2 lives and the Game Over screen triggers when the player dies with 0 lives already left. Because of this mechanic, the variable's actual value is 1 higher than what is displayed due to its unsigned nature (i.e. when 2 lives are displayed the value is 3, and the game is over when the value reaches 0). It took me a long time to figure this out, and I had the wrong address mapped to the lives variable until I did.

• Result: Successfully integrated Sonic Advance to the same level as a Type 1 game.

Verifying Type 3 Integration:

• Problem 1: running the interactive script on Sonic Advance doesn't work.

• Solution 2: turns out the interactive script is meant only for Type 1 and 2 games. In Gym Retro, integrations are classified into a bunch of types, such as STABLE (Type 1 and 2) and CUSTOM (Type 3). I made a modified version of the script (see funcs_and_utils/interactive_modified.py) and made it possible to accept custom integrated games by modifying the script's Retro environment calls to treat games as either STABLE or CUSTOM based on argparse options.

• Problem 2: running the script still doesn't work. More specifically, it is not recognizing the ROM file even though it was successfully imported and the script's setup functions are looking in the right place.

• Solution 2: after examining the entire Retro setup script suite, I found a function that for some reason was not being called: init_core_info(). This function made sure that the right cores folder was being scanned for which platforms' ROMS to look for. One suspicision I have is that since the stable-retro fork was missing the integration UI tool, which came with its own cores directory, a call to this function was omitted in this fork or was looking at the wrong path by default. Adding it manually with the right path (see study_notebook.ipynb) caused the interactive script to work.

• Result: Successfully ran the interactive script with Sonic Advance.

Testing Done Condition & Reward Function:

• Note: Altough not required, I was curious and so I decided to test the Brute algorithm on Mario and Sonic to see how well it performs given the done conditions and reward functions that were either provided (for Mario) or I defined (for Sonic).

• Problem 1: Like the interactive script, the Brute script was not made for non-stable games.

• Solution 1: This was trivial; I modified the script in a similar way to how I modified the interactive script.

• Problem 2: I found yet another bug with Retro, this time an obsolescence with time_limit.py, a script that the Brute policy called which asked for a deprecated variable inside a tuple which led to the other variables being shifted when the rest of Retro function calls stopped using that variable.

• Solution 2: Editing the time_limit.py script directly to remove uses of the deprecated variable worked perfectly. It is not the cleanest solution, but the only other way was to edit step() of the Retro environment which is used in virtually everything else, so editing that would have been a lot uglier.

• Result: Successfully tested how a simple RL agent performs on the two games, and generated movies of their gameplay in the process.

Importing a Type 4 game:

• Problem 1: from the consoles I researched, the only feasible one to emulate was the Nintendo DS. I chose New Super Mario Bros (DS) to import, but it wasn't even close to working.

• Solution 1: maybe I set the bar too high by researching more recent and advanced consoles, so I decided to try a game from Atari Lynx called Switchblade II.

• Problem 2: unfortunately, even an older console that is pretty similar to the supported Atari consoles did not seem to get anywhere. The lack of core files, a reliable emulator, and overall compatibility proved too much even though I was able to derive a pretty good estimate of what the action space would look like for the Lynx.

Overall, the significant amount of time I spent on this project meant I discovered a lot of things about Gym Retro. Here are a few:

1. I learned more about emulators and ROMs, although not enough: given more time, I would like to explore how Retro emulates the supported consoles so I can see if there is a way after all to import games from consoles such as Lynx or DS by manually adding complete support.

2. I learned about the Brute algorithm (see the paper cited below) and how it differs from Q-learning, which was the only RL algorithm I was really familiar with at the time.

3. I learned about the procedure behind setting up a pre-existing game for machine learning. In my past projects, I either used "Pythonized" games or was just handed the variables and told to make the agent, reward function, etc. This time, it was refreshing to learn how the variables are detected from a game that is already an executable.

4. Regarding Retro itself, some interesting findings were the few bugs that I ran into that are likely a result of several factors, including:

   • Stable-retro being incomplete, requiring modifications on my end such as adding the integration UI tool manually.

   • OpenAI's Gym Retro also being incomplete, which can be seen through the brute script's bug with time_limit.py.

   • Importing a new game to a system such as Retro not being an exact science by nature, and therefore prone to unforeseen problems.

   However, part of the fun is encountering these imperfections and contributing my efforts to solve them. Hopefully anyone who attempts a similar study in the future can reference this project, learn from my work, and surpass it.

Online Material

• Stack Overflow

• Reddit

• Stable-Retro Documentation

• Gym Retro Documentation

• Setup YouTube Video

• Brute algorithm paper

Open-Source Software

• OpenAI Gym Retro

• Stable-Retro

• No-Intro Collection