Hi folks! My team (the Haroohie Translation Club) has recently started a blog, and in our first ROM hacking post, I go into depth on how I reverse-engineered the compression algorithm for Suzumiya Haruhi no Chokuretsu for the Nintendo DS. This is aimed at a general audience and should serve as a general tutorial for how to think about approaching problems like compressed files. Compression is a common roadblock for people early on in ROM hacking so I wanted to write something to help folks encountering that for the first time. Please take a look at it and let me know if you have any thoughts or questions!

Link

 

[FAST6191]

Interesting. Rarely see custom compression on the DS.

Don't know that I would necessarily follow that workflow; if it is going to be basic RLE or perhaps custom flag/split LZ and you have the plaintext (presumably snatched from a RAM dump) and compressed/encrypted then going manual to decode is more the approach I would take. Though recreating decompression as a black box in some scripting effort is not a bad shout if you are going to in turn recompress* things later. Though if this is a greater primer on pulling apart files (compression format today, custom 3d format, font format or the like tomorrow, or indeed custom archive as your post states) then very nice applied example.


*if it is just ROM space you are worried about (problem if you are a dev wanting to get to the next size down as you have 50000 to order with purchasing breathing down your neck/your bonus riding on things, less of a concern in current space year with emulator/flash carts where gigabytes is a rounding error) then I would often just get people to insert the this section is not compressed flag however many bytes as necessary rather than building a true compressor.

Equally if people want a primer on the compression types and how they play out at binary level then despite the name of the document/presentation then https://ece.uwaterloo.ca/~ece611/LempelZiv.pdf covers most of the approaches you will encounter in games.
Headspin's guide to GBA stuff which also largely follows for the DS and http://problemkaputt.de/gbatek.htm#biosdecompressionfunctions might also be enlightening on the more specific approaches which then get customised. https://www.romhacking.net/utilities/826/ are also open source implementations of said same BIOS methods you can twist easily enough if it is a custom flag split or something minor that will tank existing tools but ultimately be similar enough.

 

[jonko_]

Thanks so much, Fast! Always admire your work on here; really appreciate you taking the time to read.

Going manual to decode isn't a terrible idea for sure, especially if you're familiar with how compression works. However, when I was first starting out (this was my fourth project I took on I think) I really didn't know much about compression so that wasn't something I could do as easily. But checking out what the game does (especially since we did need to recompress since the there didn't seem to be an "uncompressed section" flag) is always a good starting point I think -- if you don't know how to do something, let the game teach you.

Anyway, thanks for the extra resources and for your feedback. I really appreciate it!

 

[Lestly]

Nice! This a must read for me.
The Tennis no Oujisama - Doki Doki survival has this and remains one of the untranslated DS games in the franchise