PlayStation 2 Recompilation Project Is Absolutely Incredible
(redgamingtech.com)553 points by croes 3 days ago
553 points by croes 3 days ago
Yes, but in practice that isn't a problem. People do write self modifying code, and jump to random places today. However it is much less common today than in the past. IT is safe to say that most games are developed and run on the developers PC and then ported to the target system. If they know the target system they will make sure it works on the system from day one, but most developers are going to prefer to run their latest changes on their current system over sending it to the target system. If you really need to take advantage of the hardware you can't do this, but most games don't.
Many games are written in a high level language (like C...) which doesn't give you easy access to self modifying code. (even higher level languages like python do, but they are not compiled and so not part of this discussion). Likewise, jumping to arbitrary code is limited to function calls for most programmers.
Many games just run on a game engine, and the game engine is something we can port or rewrite to other systems and then enable running the game.
Be careful of the above: most games don't become popular. It is likely the "big ticket games" people are most interested in emulating had the development budget and need to take advantage of the hardware in the hard ways. That is the small minority of exceptions are the ones we care about the most.
JIT isn't _that_ common in games (although it is certainly present in some, even from the PS2 era), but self-modifying or even self-referencing executables were a quite common memory saving trick that lingered into the PS2 era - binaries that would swap different parts in and out of disk were quite common, and some developers kept using really old school space-saving tricks like reusing partial functions as code gadgets, although this was dying out by the PS2 era.
Emulation actually got easier after around the PS2 era because hardware got a little closer to commodity and console makers realized they would need to emulate their own consoles in the future and banned things like self-modifying code as policy (AFAIK, the PowerPC code segment on both PS3 and Xbox 360 is mapped read only; although I think SPE code could technically self-modify I'm not sure this was widespread)
The fundamental challenges in this style of recompilation are mostly offset jump tables and virtual dispatch / function pointer passing; this is usually handled with some kind of static analysis fixup pass to deal with jump tables and some kind of function boundary detection + symbol table to deal with virtual dispatch.
I believe the main interest in recompilation is in using the recompiled source code as a base for modifications.
Otherwise, yeah, a normal emulator JIT basically points a recompiler at each jump target encountered at runtime, which avoids the static analysis problem. AFAIK translating small basic blocks and not the largest reachable set is actually desirable since you want frequent "stopping points" to support pausing, evaluating interrupts, save states, that kind of stuff, which you'd normally lose with a static recompiler.
A lot of PS2-era games unfortunately used various self-modifying executable tricks to swap code in and out of memory; Naughty Dog games are notorious for this. This got easier in the Xbox 360 and PS3 era where the vendors started banning self-modifying code as a matter of policy, probably because they recognized that they would need to emulate their own consoles in the future.
The PS2 is one of the most deeply cursed game console architectures (VU1 -> GS pipeline, VU1 microcode, use of the PS1 processor as IOP, etc) so it will be interesting to see how far this gets.
Yeah, at least in the case of most Naughty Dog games the main ELF binary is in itself a little binary format loader that fixes up and relocates proprietary binaries (compiled GOAL LISP) as they are streamed in by the IOP. It would probably be a bit pointless to recompile Naughty Dog games this way anyway though; since the GOAL compiler didn’t do a lot of optimization, the original code can be recovered fairly effectively (OpenGOAL) and recompiled from that source.
I'd say practically none, we were quite memory starved most of the time and even regular scripting engines were a hard sell at times (perhaps more so due to GC rather than interpretation performance).
Games on PS2 were C or C++ with some VU code (asm or some specialized hll) for most parts, often Lua(due to low memory usage) or similar scripting added for minor parts with bindings to native C/C++ functions.
"Normal" self-modifying code went out of favour a few years earlier in the early-mid 90s, and was perhaps more useful on CPU's like the 6502s or X86's that had few registers so adjusting constants directly into inner-loops was useful (The PS2 MIPS cpu has plenty of registers, so no need for that).
However by the mid/late 90s CPU's like the PPro already added penalties for self-modifying code so it was already frowned on, also PS2 era games already often ran with PC-versions side-by-side so you didn't want more than needed platform dependencies.
Most PS2 performance tuning we did was around resources/memory, VU and helped by DMA-chains.
Self modifying code might've been used for copy-protection but that's another issue.
See also: XenonRecomp, which does the same thing for Xbox 360, and N64:Recompiled which does the same thing for N64.
Note that this "recompilation" and the "decompilation" projects like the famous Super Mario 64 one are almost orthogonal approaches in a way that the article failed to understand; this approach turns the assembly into C++ macros and then compiles the C++ (so basically using the C++ compiler as a macro re-assembler / emulation recompiler in a very weird way). The famous Super Mario 64 decompilation (and openrct and so on) use the output from an actual decompiler which attempts to reconstruct C from assembly, and then modify that code accordingly (basically, converting the game's object code back into some semblance of its source code, which this approach does NOT do).
Even with tooling isn't this kind of effort going to be enormously complicated, because, as I understand it, so many aspects of the behaviour of 8-bit and console games relied on all sorts of clever techniques, timing quirks and other side effects.
Won't it be very difficult for the recompilation process or the dev to recognise when these are being relied on and to match the key behaviour?
Or is the idea to pull out the basics of the game structure in a form that runs on modern hardware, then the dev fleshes out the missing parts.
From the last time I looked at these recompilation projects, they take the assembly and basically convert each opcode back into an llvm instruction and then recompiled from there. This comes with a lot of caveats though, last time I looked at it we still needed a function map of some kind to tell us where all the functions are and you still needed to replace chunks of the generated code afterwards for things like rendering, sound, controller input, and just about anything that would interact with the world outside of the cpu.
Edit: After some reading on the github page, it seems they are translating it to c++ instead of using llvm directly, but the original idea still holds. They aren't decompiling it to c++ that looks like original source code, it more like they're converting it to c++ that emulates the processor and gets statically compiled.
So it's not really just a drop and go replacement like it sounds like it'd be, but it has so far enabled the recompilation of multiple n64 games. This seems like an extension into the ps2 space.
Side note: The ps2 is a 32bit console with a 64bit alu (and some 128bit simd)[1]. So a lot of the weird tricks from the 8bit days weren't really used here. Not that there aren't weird tricks, just things like using undocumented instructions and racing the beam are less prevalent. I could be wrong here, I was growing up at this time not making games. All of this is just from research I've done in the past. Someone feel free to correct me if I'm wrong.
[1] https://en.wikipedia.org/wiki/PlayStation_2_technical_specif...
I suspect many PS2 games were mostly written in C or C++, which greatly simplifies things. Games that used the interface libraries from Sony for e.g. managing input will greatly simplify things as well.
One notable exception is the Jak and Daxter games, which were written in GOAL[1], but have their own recompilation project.
1: https://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp
I think these kind of things will bring problems.
Because Nint€ndo or $ony (and others game companies) have a big problem, their old games are awesome and if the people can play these games, then the people will be happy and will not need new games or new sagas.
Because the problem is not the people playing old games, the real problem is the people will not pay for new games
And we know that these companies have army of lawyers (and "envelopes" to distribute among politicians) to change the laws and make illegal something that is not illegal.
I have very little sympathy for these companies. They make access to these titles difficult if not impossible. Preservation matters and they have no interest in anything but the bottom line today.
If you have a Mac aethersx2 it works great on Apple silicon
This sounds very cool, but I can practically hear the IP lawyers sharpening their buzz-axes...
They haven't been all that aggressive against the decompile/recompile projects, interestingly. They're sometimes/often set up so you need the original to grab assets etc., but that code is copyrighted too and I'd have to imagine a decompile that purposely compiles to an identical binary would be a derivative work.
My best guess is that for them it's not worth the hassle or any possibility of a negative result in court as long as people have to jump through some hoops by providing an original, and for the projects that don't do that, you have very straightforward easy infringement cases without even getting into the decomp stuff. Though really even ROMs seem to be tacitly tolerated to some extent lately. Maybe there's an attitude that keeping people involved with the franchise is worth it, again so long as it doesn't become too easy.
Sony have actually been fairly chill about emulators etc. so I'd be surprised if lawyers got involved here.
They actually used an open source Playstation emulator when they released the "Playstation Classic" in 2018.
Or as in cartoons, IP lawyers with dollar symbols in their eyes.
I know Nintendo is historically very litigious, but the various decomp and recomp projects seem to be doing fine…
I’ve been meaning to start decompiling one of my favorite games of the era (Hulk Ultimate Destruction) after watching the decomp of other games. Perhaps this is a sign to start?
This is amazing for preservation. Being able to run these classics on modern hardware with native recompilation is a huge step forward.
N64 as I understand it has some self rewriting code that makes this hard
N64 is the one leading the way on recompilation Mario 64, Perfect Dark, Zelda, Mario Kart etc etc have all been done
I've been working on decompiling Dance Central 3 with AI and it's been insane. It's an Xbox 360 game that leverages the Kinect to track your body as your dance. It's a great game, but even with an emulator, it's still dependent on the Kinect hardware which is proprietary and has limited supply.
Fortunately, a Debug build of this game was found on a dev unit (somehow), and that build does _not_ have crazy optimizations in place (Link-time Optimization) that make this feat impossible.
I am not somebody that is deep on low level assembly, but I love this game (and Rock Band 3 which uses the same engine), and I was curious to see how far I could get by building AI tools to help with this. A project of this magnitude is ... a gargantuan task. Maybe 50k hours of human effort? Could be 100k? Hard to say.
Anyway, I've been able to make significant progress by building tools for Claude Code to use and just letting Haiku rip. Honestly, it blows me away. Here is an example that is 100% decompiled now (they compile to the exact same code as in the binary the devs shipped).
https://github.com/freeqaz/dc3-decomp/blob/test-objdiff-work...
My branch has added over 1k functions now and worked on them[0]. Some is slop, but I wrote a skill that's been able to get the code quite decent with another pass. I even implemented vmx128 (custom 360-specific CPU instructions) into Ghidra and m2c to allow it to decompile more code. Blows my mind that this is possible with just hours of effort now!
Anybody else played with this?
0: https://github.com/freeqaz/dc3-decomp/tree/test-objdiff-work...
According to https://www.metacritic.com/browse/game/ps2/, it's a tie between Tony Hawk's Pro Skater 3 and GTA 3. But it's a subjective question with a subjective answer.
Metal Gear Solid 2: Sons of Liberty was my favorite, still is. Good story, and we are not to far off of having mechs controlling data flow with how Boston Dynamics is doing.
A more accurate version of the famous idiom:
Those who can, do (and sometimes become teachers when they get older). Those who can’t become journalists.
As far as I know, static recompilation is thwarted by self modifying code (primarily JITs) and the ability to jump to arbitrary code locations at runtime.
The latter means that even in the absence of a JIT, you would need to achieve 100% code coverage (akin to unit testing or fuzzing) to perform static recompilation, otherwise you need to compile code at runtime at which point you're back to state of the art emulation with a JIT. The only real downside of JITs is the added latency similar to the lag induced by shader compilation, but this could be addressed by having a smart code cache instead. That code cache realistically only needs to store a trace of potential starting locations, then the JIT can compile the code before starting the game.