Hacking How do I merge my DS ROMs ?

[klokwerk]

Hi there!

I'm just after some quick help from you gurus.

Here's a hypothetical I need help with. Do not take the following to mean I have illegal DS ROMs...

...I currently have pretty much every NDS ROM to date, each ROM in a separate 7z file. I use OfflineList with the Advanscene DAT to keep them organised. However, I've come to the point where I want to merge the roms to save space. I don't want to use the unstable version of OfflineList that *supposedly* lets you merge ROMs (never worked for me - just crashes).

I found an XMDB file for GoodMerge to let that program do it for me (merging the ROMs), however, the ROMs have to be named a certain way and I need to have one of those "Have List" files in advance. I've tried RomCenter, ClrMamePro and other programs with a combination of different dat files, xml files and xmdb files... no such luck.

There don't seem to be any straight-forward instructions or clear and concise tutorials on how to handle this. I'm fed up with ClrMamePro and RomCenter, so can somebody please tell me the best way (what programs, files and method) to go from 3000 different 7z single-rom archives to a much smaller number of merged 7z rom archives? Why is it so hard to find this simple information?

Thanks in advance.

 

[Cheater388]

I'm sorry to seem like an ass but you're using the latest 7-zip, right? Try selecting all the archives, right click, extract, and then select everything and compress again. 7-zip has a lovely batch system now!

 

[klokwerk]

You don't seem like an ass. I'm appreciative for the response.

However, your method will just put ALL my ROMs in the one archive, won't it? I'm after a simple way to get a merged archive in the sense that the various region dupes of a single game will be put into a single archive. So, it'll be like this example:

Example:

WarioWare, Inc. - Minigame Mania (Europe) (En,Fr,De,Es,It)
--- Made in Wario (Japan)
--- Waliou Zhizao (China)
--- WarioWare, Inc. - Mega Microgames! (USA)
--- WarioWare, Inc. - Mega Microgames! (USA) (Beta)
Watashi no Makesalon (Japan)
Whac-A-Mole (USA)
Whistle! - Dai-37-kai Tokyo-to Chuugakkou Sougou Taiiku Soccer Taikai (Japan)
Who Wants to Be a Millionaire (Europe)
--- Chi Vuol Essere Milionario (Italy)
--- Qui Veut Gagner des Millions (France)
--- Quiere Ser Millonario (Spain)
--- Weekend Miljonairs (Netherlands)
--- Wer wird Millionaer (Germany)
--- Who Wants to Be a Millionaire (Australia)

The indented titles being part of a single archive. So, I'll save space by having archives for each unique ROM and merging the variants of a single game into their own archive.

I hope I'm making sense. What's worse is I've found numerous ways online. However, I can't seem to find a good set of instructions/renaming rules to get from my current setup to the desired final product. I'm sure somebody on this board must be able to help?

Thanks again for your reply.

 

[Another World]

last i checked goodmerge does not support nintendo ds roms. (http://goodmerge.sourceforge.net/About.php). i think what you want to do can't be done unless you want to do it by hand. there might be an alternative application but i am not aware of it. have you tried posting on the goodmerge forum and asking there?

-another world

 

[enarky]

Last time I tried GoodMerge (been a while, I have to admit) it relied on handmade lists of ROMs that could be merged. I'm not sure, but since there are constantly new DS ROMs being released I assume it would be too much work to keep such a list up-to-date. Not to mention the time it would take to "merge" updates into a merged collection every couple of weeks.

*speculation, speculation...*

 

[paul3100]

I currently have pretty much every NDS ROM to date, each ROM in a separate 7z file

Lol , i would keep that to yourself cause you never know who is watching!

Sorry to seem a dumb ass my self but what do you mean by merging all your roms?

paul

 

[Hehe Moo]

How about just trimming the games?

 

[enarky]

paul, read what klokwerk wrote in post #3, he explained it pretty well IMO.

Hehe Moo, trimming achieves nothing in 7-Zip packed ROMs. Trimming just cuts away on unused areas, since those unused areas are usually filled with 0s they pack to ~100% with 7-Zip.

 

[klokwerk]

The method I'm trying uses an XMDB for GoodMerge that tells it how to merge the DS ROMs. However, I can't run it as my ROMs aren't named in the proper filename format. So, my next step is to do a batch rename (automatic), which they've recommended using RomCenter or ClrMamePro to do (with the right DA/XML file). My dilemma then is finding the right file so that once I've renamed the ROMs will be in the proper format and then I can use GoodMerge to merge everything for me.

So, here's the website I'm trying to understand: DAT-o-matic. Anyone else who needs to merge DS ROMs willing to try this method out and give me a simplified rundown?

Sorry if I sound rude, it's late here and I'm going to bed now.

 

[kazumi213]

Hi klokwerk.

The XMDB file from No-Intro's DAT-o-MATIC will allow you to do what you want. Your collection, when properly 7z-merged, will go from 3000+ files to 1700+ and from 70+ GB to just around 35 GB in size.

However before using the XMDB file for NDS you need first to rename your ROMs using the No-Intro DAT for NDS. On the page you linked just choose the numbered or unnumbered DAT (first option), depending on your preferences, then click download. That's it. You will get a numbered or unnumbered full NDS DAT.

Now you need to download and install the clrmamepro ROM manager. Once installed, copy the previously downloaded DAT to the "datfiles" subfolder. Check the clrmamepro documentation or google some tutorial on how to use it. You will be able to mass rename all your ROMs in minutes and check for missing/bad ROMs in your collection. Use clrmamepro to generate a "have list" text file (see documentation).

After renaming your ROMs you are ready to use the XMDB file. Download and install GoodMerge, copy the XMDB and "have list" files to the installation folder and finally run GoodMerge. It is very straightforward to use. Important note: merging the whole NDS colection can take 10+ hours.

The whole process is not newbie-friendly and requires you to read stuff. There's no shortcut procedure without renaming step.

And enarky is right. The XMDB file is updated each week, and some 7z archives will require updating. The procedure to maintain the merged collection is to scan it using clrmamepro and No-Intro Parent-Clone DAT (XML file) in merged mode to identify the archives that have changed, rebuild those few archives to unmerged files and finally goodmerging them again. Easy but not quick, as I said.

 

[klokwerk]

Kazumi, thank you sooo much!

That's exactly what I was after, you managed to succintly explain how it all works and which files I needed to use with what.

I'm currently using ClrMamePro to rename and then I'll be able to use GoodMerge to shrink my collection down a bit.

Thank you very much!

 

[mysticwaterfall]

This really has nothing to with the question, so I waited until after it had been answered as not to hijack... but is there really a point to having the same game in 7 different versions? Even if you do speak 7 different languages, I wouldn't think you would ever have the need for this... I really fail to see the point.

 

[klokwerk]

@kazumi213:
Ah, bugger! Everything's been renamed, but now when I try to merge with GoodMerge it comes up with that Windows error box. You know the one? "Goodmerge has encountered a problem and needs to close. We are sorry for the inconvenience." Kazumi have you (or has anyone else for that matter) ever experienced this sort of error with GoodMerge and know what's causing it?

@mysticwaterfall:
I actually studied Japanese for a number of years and so I can usually make my way through the Japanese games (those exclusive to Japan) and sometimes changes are made in the transition to US/EU format. For example a lot of people like to "undub" Final Fantasy IV etc. However it mainly has to do with how I ... acquired the games. Rather than sift through and delete, for the moment I'll merge.

 

[kazumi213]

@kazumi213:
Ah, bugger! Everything's been renamed, but now when I try to merge with GoodMerge it comes up with that Windows error box. You know the one? "Goodmerge has encountered a problem and needs to close. We are sorry for the inconvenience." Kazumi have you (or has anyone else for that matter) ever experienced this sort of error with GoodMerge and know what's causing it?

No reports so far about any error at all.

When are you getting the crash? On starting GoodMerge, while configuring it, when starting to merge, while merging? Note GoodMerge requires MS .NET Framework 2.0 and cannot be run from a network drive. Also, I'm not sure whether it works on Vista.

An alternative to GoodMerge is using the "Batch Job", under Downloads->P/Clone tab for NDS system. See details txt file there for more info. It will yield same results except that you can't customize merged archive names according to your region preferences.

 

[klokwerk]

Thanks for that alternate suggestion, kazumi213.

The error in GoodMerge occurred after I'd clicked "Merge Now" and the log window had come up, so I guess when it tried to start merging.

However, I used the "batch job" file, modified to the correct output directory and also to take advantage of my quad-core CPU. My dilemma is now that I hadn't realised in the renaming debacle that some of my ROMs ended up in a different folder entirely. If I want to have these added to the merged archives can I just re-run the batch job? Say for example I have the USA version of a game in this folder that wasn't merged, but the European and Spanish versions were already merged together.

Am I right in assuming that in this case I'd have to go through the merged archives individually and extract the ROMs that have already been merged and then re-run the batch job so they all get merged together? Or is the process able to 'see' that while 2 ROMs have been merged, a third needs to be added and thus process them accordingly.

Hope I'm making sense.

Furthermore, is this how I'd have to make regular updates? Manually checking after downloading a new batch file and then extract all other variants of the new ROMs and then go through the process again? Also, what program+file combination do I need to run in order to scan these merged ".7z" archives to output a pair of "Have" and "Missing" ROM lists? I imagine this would help me isolate the few ROMs I'm missing and help to automate merging newer ROMs in with the (now) current merged archives.

Sorry if I'm not explaining myself well. :/

 

[kazumi213]

Before starting to merge, GoodMerge parses the XMDB file. If errors are found, it stops the whole process and logs an error, but doesn't crashes. I've performed a limited test (this is done by specifying just a fraction of titles in the "have list", GoodMerge will only work on the specified ROMs from the whole source folder) and the XMDB file is parsed without errors and merges the ROMs.

Check that you aren't using too overkill "7-Zip Options" (under Compression tab) for your system. There are 3 values:

MB 'Max' RAM
MB 'Max' Dict
MB 'Ultra' Dict

For a standard 2 GB RAM system, reduce the first value so the second reads "128". This is safe and gives quite good results. With more system RAM available you can try higher values, but I suggest performing limited tests so you can find your optimal value quicker. Your system being unable to allocate the specified amount of memory can be causing the crash (though GoodMerge also reports this problem and stops, but should not crash) You could also try to merge from uncompressed ROMs in your source folder like you do with the "Batch Job", just in case.

Now regarding your "update question": 7-Zip format doesn't allow to add files to solid archives (you create solid archives when merging, hence the big size drop). So you need to uncompress the merged archive and do it again from all old and new ROMs. Just edit the batch file so it only contains the lines related to these archives that require updating.

And yes, that's the problem of merging a "live" system like NDS, it requires constant updates. The XMDB/batch files are there because some people just doesn't care the extra work.

CMPro allows you to automatically 7z-merge and update your merged collection by using the XML file (see details). The problem is that CMPro doesn't currently merge in solid mode. You will get the 3000 -> 1700 file count reduction but overall size will be the same as having your files compressed independently. However by properly setting up CMPro with the XML file (again see XML details page, merging section) you can scan (but don't fix, unless you don't care about size) your previously merged collection. All reported errors will be archives you need to manually merge, but at least CMPro will automatically spot them for you.

 

[klokwerk]

I'm guessing it'd be a memory issue then as I've just checked and my brother has borrowed a stick of RAM to run his machine (his RAM went kaput). So I only have 1GB in there... bugger. The merging process would've saved even more space had I had 2GB of RAM in there, no?

However, I'm thankful for the further assistance. I'll get CMP to tell me which ones I need to remerge and do that manual bit of editing to get the batch file to merge em for me.

Is it worth putting more RAM back in (I'll just buy a 2GB stick) and then remerging with a higher dictionary size? The default for the batch file was 64MB I believe, and you're saying I could use 128MB with 2GB of RAM, if I had 3 or 4GB of RAM could I use the higher (maximum) dictionary size of 256MB - and more importantly how better would the compression really be?

Finally, I've just tried scanning the collection with CMP and the parent-clone 'xml'. It's a dat that is generated when I try to download the XML and also: the "see details" on the XML page gives me a 404 error. I must be missing how to configure the program to handle the file as it records a log with errors on every single file. I'm sure that missing details text file could explain it for me. Any chance you could fix the link?

 

[kazumi213]

First of all, by merging to solid archives you are getting a huge save in overall size. This size will approach to the one you'd get if you only had 1 member of each family. Using dictionary sizes > 64 MB for NDS will yield a better approach indeed.

However using dictionary sizes > 128 MB you start getting diminishing returns and only a *few* families on NDS will really benefit from using 256 MB dictionary. You will need 4 GB RAM to use 256 MB dictionary and this means a 64-bit Windows for that RAM to be available (3 GB is max to 32-bit Windows).

I will report the dead link ASAP. In the meantime you can check the following:

- Go to "Settings" panel. In addition to configuring your "ROMs" path, select "Regions" from the combo-box.
1. Click on "Select All"
2. Arrange the list so it reads from top to bottom "EUR > USA > JPN > GER > FRA > rest as you like". The problem here is that you are suppossed to arrange the list exactly as you did with the GoodMerge "Bias Zone Priority" for perfect naming match. However you've used the batch, which uses a fixed (and arbitrary) region preference. The order I give you is the one used in the batch, except some cases where FRA > GER.

- Go to "Scanner" panel.
1. Be sure to select "Merged ROMs" radio button. This is required for CMPro to recognize the merged archives.
2. Scan. Most of the errors should be gone. There should be some naming errors. If they are related to the region discrepancy described above, just ignore them. These will also appear if ROM naming is revised in future updates. Then there should be some "missing/incomplete errors". You are interested on the these.

 

[klokwerk]

kazumi213:

Okay, I'm busy getting the missing files and renaming and all that. Thanks for the run down on how to do that.

Back to the merging: I've ordered 2GB of RAM and I'll shortly re-merge my collection with the 128MB setting (could I go any higher with 3GB of RAM on a 32-bit CPU?). However, I wanted to ask if there was a way to modify the batch file to do this as I don't want to have to stuff around trying to get GoodMerge to work. I'm sure it's just a simple modification to make "-mx9" something different, but I've tried searching the web and can't work out what parameter I'd need to change it to, I'm afraid. Any help?

Thanks once again for all this help, that goes for me and on behalf of all the lurkers (over 700! views) who are benefiting from our discussion too.

 

[kazumi213]

With 3 GB RAM you should be able to use 192 MB dictionary. Before going for the full set, just perform a few tests by merging 5-6 families and comparing results, just to be sure.

For all your parameter and switches description needs just check the CHM help file in the 7za distribution package. Here is an example:

7za a -ms -mmt -mx=9 -m0d=192m

Change "192m" to the desired dictionary size (MBytes).