{"id":49685,"date":"2021-08-17T01:23:26","date_gmt":"2021-08-17T01:23:26","guid":{"rendered":"https:\/\/www.libretro.com\/?p=49685"},"modified":"2022-04-03T00:51:47","modified_gmt":"2022-04-03T00:51:47","slug":"miniretro-testing-emulators-at-scale","status":"publish","type":"post","link":"https:\/\/www.libretro.com\/index.php\/miniretro-testing-emulators-at-scale\/","title":{"rendered":"Miniretro: testing emulators at scale"},"content":{"rendered":"

This is a guest article written by David GF Net republished with his authorization. Original blog article can be viewed here<\/a>.<\/p>\n

Last year I got involved in Libretro\/Retroarch development after buying an Odroid Go Advance. During this time I\u2019ve been working mainly on gpsp<\/a> and porting it to new devices and such.<\/p>\n

One of the main issues about working in software is testing, and as you can probably imagine there\u2019s no tests in most emulators. Partly due to bad practices and because it\u2019s hard to write tests for them. That\u2019s why I came up with miniretro. It is a libretro frontend designed for headless operation, so that it can be used for end-to-end\/integration testing. The frontend runs a core and a rom with fake inputs and grabs the output.<\/p>\n

Cross platform testing<\/h2>\n

Some of the emulators (like gpsp) feature a dynamic recompiler (aka dynarec), which have platform specific (CPU, OS and\/or device specific) backends targeted at them. These can translate original console instructions into your device\u2019s instruction set (for speed). Since this is device specific, it\u2019s hard to write, debug and test. One needs a toolchain for the platform and a physical device to test it. Or at least that\u2019s the theory!<\/p>\n

With miniretro it gets easier to test other platforms like for instance ARM and MIPS devices. Since it is simple (almost no dependencies) and just a Linux binary, we can use Qemu userspace emulation to run our tests! This way there\u2019s not need for a physical device, nor manual testing.<\/p>\n

The following diagram shows how it works: miniretro and the libretro core are built for the specific device\/architecture and they run under qemu. Qemu takes care of translating syscalls into host syscalls, so there\u2019s no need for a whole OS to run the program. Miniretro can open pipes and other IPC communication channels to input\/output any data.<\/p>\n

Diagram of Miniretro running under Qemu<\/strong><\/p>\n

\"\"<\/p>\n

Just as an example, let\u2019s build picodrive, gpsp and pcsx for armv6 and mips32. To do this you will need a toolchain. In my case I\u2019ve been using some Linux-generic toolchains for a variety of platforms, such as ARM, MIPS, x86 and PowerPC, freshly built using Buildroot (you can find them at my Copr if you use Fedora, see this repo). This, coupled with the regular Qemu userspace emulators (available in most Linux distros), enables us to test our emulators on a bunch of platforms effortlessly.<\/p>\n

# Build the three emus for arm and mips
\n git clone –recurse-submodules https:\/\/github.com\/irixxxx\/picodrive.git && cd picodrive
\n make platform=armv CC=\/opt\/buildroot-armv6el-eabi-uclibc\/bin\/arm-linux-gcc -j10 -f Makefile.libretro all && \\
\n mv picodrive_libretro.so picodrive_libretro_arm.so && make -f Makefile.libretro clean
\n make platform=unix CC=\/opt\/buildroot-mipsel32-o32-uclibc\/bin\/mipsel-linux-gcc -j10 -f Makefile.libretro all && \\
\n mv picodrive_libretro.so picodrive_libretro_mipsel.so && make -f Makefile.libretro clean<\/p>\n

git clone https:\/\/github.com\/libretro\/gpsp.git && cd gpsp
\n make platform=armv CC=\/opt\/buildroot-armv6el-eabi-uclibc\/bin\/arm-linux-gcc -j10 all && \\
\n mv gpsp_libretro.so gpsp_libretro_arm.so && make platform=armv clean
\n make platform=mips32 CC=\/opt\/buildroot-mipsel32-o32-uclibc\/bin\/mipsel-linux-gcc -j10 all && \\
\n mv gpsp_libretro.so gpsp_libretro_mipsel.so && make platform=mips32 clean<\/p>\n

git clone https:\/\/github.com\/libretro\/pcsx_rearmed.git && cd pcsx_rearmed
\n make platform=armv CC=\/opt\/buildroot-armv6el-eabi-uclibc\/bin\/arm-linux-gcc -j10 -f Makefile.libretro all && \\
\n mv pcsx_rearmed_libretro.so pcsx_rearmed_libretro_arm.so && make -f Makefile.libretro clean
\n make platform=unix CC=\/opt\/buildroot-mipsel32-o32-uclibc\/bin\/mipsel-linux-gcc -j10 -f Makefile.libretro all && \\
\n mv pcsx_rearmed_libretro.so pcsx_rearmed_libretro_mipsel.so && make -f Makefile.libretro clean<\/p>\n

# Build miniretro fro arm and mips too
\n git clone https:\/\/github.com\/davidgfnet\/miniretro && cd miniretro
\n PREFIX=\/opt\/buildroot-mipsel32-o32-uclibc\/bin\/mipsel-linux- make && mv miniretro miniretro.mipsel && make clean
\n PREFIX=\/opt\/buildroot-armv6el-eabi-uclibc\/bin\/arm-linux- make && mv miniretro miniretro.arm && make clean<\/p><\/blockquote>\n

And finally, let\u2019s run a full matrix of tests, choose your favourite ROMs of course!<\/p>\n

export ARCHS=”arm mipsel” EMUS=”pcsx_rearmed picodrive gpsp”
\n export FRAMES=”10000″ OUTPUT=”output\/” SYSTEM=”$HOME\/.config\/retroarch\/system\/”
\n declare -A SYSROOTS=([“arm”]=”\/opt\/buildroot-armv6el-eabi-uclibc\/arm-buildroot-linux-uclibcgnueabi\/sysroot\/” \\
\n [“mipsel”]=”\/opt\/buildroot-mipsel32-o32-uclibc\/mipsel-buildroot-linux-uclibc\/sysroot\/”)
\n declare -A ROMDIR=([“gpsp”]=”gbaroms\/path\/” [“picodrive”]=”mdroms\/path”)<\/p>\n

mkdir -p ${OUTPUT}<\/p>\n

for emu in $EMUS; do
\n for arch in $ARCHS; do
\n echo “Running $emu for $arch”
\n .\/regression.py –core ..\/${emu}\/${emu}_libretro_${arch}.so –system ~\/.config\/retroarch\/system\/ \\
\n –input ${ROMDIR[${emu}]} –output ${OUTPUT}\/${emu}-${arch} –threads=`nproc` –frames=${FRAMES} \\
\n –driver “qemu-${arch} -L ${SYSROOTS[${arch}]} .\/miniretro.${arch}”
\n done<\/p>\n

.\/report.py compare –results ${OUTPUT}\/${emu}-* –output ${emu}-report.html
\n done<\/p><\/blockquote>\n

This will produce a comparison report for every emulator, where you can see a screenshot (plus some info) for each rom and each platform. It will compare any different screenshot and report it in a red background. This is very useful to compare devices but can be also be used to compare across versions of the emulator (say, on a new commit or PR).<\/p>\n

Reports: HTML and video!<\/h2>\n

It is also possible to produce video output directly using miniretro (if ffmpeg is installed on the system). It will output BMP and RAW PCM frames to ffmpeg which will encode them as video and audio tracks. Unfortunately there\u2019s a bug in ffmpeg which prevents using two pipes to feed data and we are forced to produce separate streams (which can then later be muxed easily without reencoding).<\/p>\n

I ran the above example with gpsp and arm, mips and x86 (+interpreter on x64) and got the following video (after a bit of editing with ffmpeg :P)<\/p>\n