Hacking Wii.dat project: expand titles.txt, expand your GUI

[Vidboy10]

Awesome idea. but seems really hard.

 

[AlexDP]

It's both sad and hilarious that the Coby SD is the second most popular choice.

This joke is one of the best from gbatemp...
So I voted Coby SD too and sorry if the poll goes wrong because of a joke

ROFL.... i just read that thread (dont know how I even missed it... lol), hilarious

 

[Don Killah]

yep i feel like AlexDP, how come i've missed such a masterpiece?
simply awesome!

 

[zorglub07]

Thanks for those who offered to help, we could try to race to complete the file once it's online <img src="style_emoticons/<#EMO_DIR#>/smile.gif" style="vertical-align:middle" emoid="" border="0" alt="smile.gif" />

Thanks for your feedback McHaggis, this is exactly the kind of thorough input I need!

<!--quoteo(post=1928217:date=Apr 22 2009, 02:45 AM:name=McHaggis)--><div class='quotetop'>QUOTE(McHaggis @ Apr 22 2009, 02:45 AM) <a href="index.php?act=findpost&pid=1928217"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->isn't it a bit redundant downloading a database of 500+ games when you only have 20 on your hard drive? We're not all pirates <img src="style_emoticons/<#EMO_DIR#>/wink.gif" style="vertical-align:middle" emoid="" border="0" alt="wink.gif" />
...
One of the main advantages of XML is that it's human readible, so tidyness is a major issue when creating XML documents. Using a different tag name for each country code isn't a great idea, that's what attributes are for.
...
Like I said, not meaning to sandbag you or anything, just speaking as someone who deals with XML every day.<!--QuoteEnd--></div><!--QuoteEEnd-->Nobody really needs a list of 500+ games, but anyone can have a need for 10 different games from this list, that's the purpose of a general list, to suit anyone,
pirates included, they have a hard life at sea as it is to deprived them of such a feature.
Readable and tidiness, I'm all for it <img src="style_emoticons/<#EMO_DIR#>/smile.gif" style="vertical-align:middle" emoid="" border="0" alt="smile.gif" />
I did not add identation in the example but the file will have it.
No offense taken at all, your input is appreciated and the final draft will most likely use more child elements,
I will try to explain why I proposed a simplistic schema so that people can view both sides of the coin.

I understand adding attributes and child nodes is generally a good habit if not a necessity, especially for a web developer, and how such a minimal structure can be seen as flawed, but for simplicity's sake I thought this particular case may warrant an exception.
In this particular case, it's more important that anyone can edit the structure without messing it up than it is to have a flexible/elegant design.
The needs are so simple that I fear anything more complex than the minimum is not worth the added complexity.
This database purpose is to load information about a game with a known set of categories, not really process unknown data or categories.

- i placed accessories as element names because "nunchuck" is not the data that will be directly used, "true" is, and in hope to avoid people adding wrong elements by thinking they can add anything. when a new accessory is released it can be added.
- true/false or 0/1 is faster to type than "balanceboard" and less prone to error if edited by hand.
- minimal structure means less possibility of someone forgetting a quote somewhere
- minimal structure is more humanly readable in my opinion (but maybe I'm just thinking like a frakkin' machine?)

That is why I was leaning towards the "minimal XML" approach as described here <a href="http://xml.coverpages.org/minxmlspec20000413.html" target="_blank">http://xml.coverpages.org/minxmlspec20000413.html</a>

from <a href="http://www.w3schools.com/DTD/dtd_el_vs_attr.asp" target="_blank">http://www.w3schools.com/DTD/dtd_el_vs_attr.asp</a>
"
There are no rules about when to use attributes, and when to use child elements. My experience is that attributes are handy in HTML, but in XML you should try to avoid them. Use child elements if the information feels like data.
"
from <a href="http://www.ibm.com/developerworks/xml/library/x-eleatt.html" target="_blank">http://www.ibm.com/developerworks/xml/library/x-eleatt.html</a>
"
- If you consider the information in question to be part of the essential material that is being expressed or communicated in the XML, put it in an element. For human-readable documents this generally means the core content that is being communicated to the reader.
- data goes in elements, metadata in attributes
- If the information is intended to be read and understood by a person, use elements. In general this guideline places prose in element content.
"

The format used by Offlinelist also seems to favor simplicity, and was chosen for the same kind of purpose which is to be a simple game list.
The goal here is slightly different because it is not about cataloging every scene release, but about game information (name, checksum) and ultimately preservation, like Goodtools/No-Intro.

offlinelist:
<!--c1--><div class='codetop'>CODE</div><div class='codemain'><!--ec1--><game>
ÂÂÂÂ<imageNumber>1332</imageNumber>
ÂÂÂÂ<releaseNumber>1337</releaseNumber>
ÂÂÂÂ<title>Sonic & The Black Knight</title>
ÂÂÂÂ<publisher>Sega</publisher>
ÂÂÂÂ<location>0</location>
ÂÂÂÂ<sourceRom>WiiERD</sourceRom>
ÂÂÂÂ<language>4291</language>
ÂÂÂÂ<files>
ÂÂÂÂÂÂÂÂ<romCRC extension=".iso">0328DB2E</romCRC>
ÂÂÂÂ</files>
ÂÂÂÂ<im1CRC>3DFA8720</im1CRC>
ÂÂÂÂ<im2CRC>838E00D2</im2CRC>
ÂÂÂÂ<comment/>
ÂÂÂÂ<duplicateID>1332</duplicateID>
</game><!--c2--></div><!--ec2-->
for Goodtools and No-Intro images are managed in separate dats, because this is not official data for the game, the same goes for release group and release number.

<!--quoteo(post=1928217:date=Apr 22 2009, 02:45 AM:name=McHaggis)--><div class='quotetop'>QUOTE(McHaggis @ Apr 22 2009, 02:45 AM) <a href="index.php?act=findpost&pid=1928217"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->Another thing - it's much easier to code a loop through a specific set of nodes (for example <accessory>) than manually checking the existence of many nodes (<balanceboard>, <motionplus>) not to mention the redundancy of specifying the text of those nodes as false.<!--QuoteEnd--></div><!--QuoteEEnd-->There is only a single type of main element, which is <game>, and there is no need to load accessories only so it is really one pass to load all data about a game.
The need is to load true/false status for known accessories, not load an unknown list of accessories that's why I put them as elements.
But you're right, adding categories makes perfect sense, even if it is only to have a more organized structure.
You're right about redundancy, accessories should only be mentioned if they are supported, and the required="true" attribute is absolutely necessary.

Not so sure about this one though: <rating esrb="E" pegi="12+" />, it's not like there are more rating systems and the GUI should load both and be able to display the one matching a game's setting or user preference.

All of this being said, I do think a couple child elements will not hurt, it's just that I started with the minimum to see what was really necessary to add.

<!--quoteo(post=1928249:date=Apr 22 2009, 02:54 AM:name=AlexDP)--><div class='quotetop'>QUOTE(AlexDP @ Apr 22 2009, 02:54 AM) <a href="index.php?act=findpost&pid=1928249"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->I'm definitely willing to provide support for this idea in the next version of WBFS Manager (3.0) whether it's CSV files or XML (preferable) files. By support I mean using the XML files as input for copying over to WBFS drives as well as generating the XML from someone's WBFS drive (so they can just export it and post it quickly).
XML files are obviously preferable and I agree with McHaggis, the XML document layout needs to be nice and clean and most importantly standard and unified.<!--QuoteEnd--></div><!--QuoteEEnd-->Thanks for your support, I think those will be great features! It could also be used to verify iso checksums before games are sent to WBFS (and even to check games on WBFS with another checksum)
Yes, XML is the obvious choice and that's why I added preliminary support for it already <img src="style_emoticons/<#EMO_DIR#>/smile.gif" style="vertical-align:middle" emoid="" border="0" alt="smile.gif" />
The file can look nice and clean even with a simple and not so elegant structure because it will be formatted with identation, but I agree that maybe adding a slight level of complexity by introducing child elements is probably not a luxury.
Don't worry the file will be perfectly valid XML.

To make it standard and unified it's up to us to quickly decide on a draft and add support.
If people download several USB Loaders and see they can enjoy the benefit of this database, they will expect it in other loaders and it will be a unified standard.
It's only when support is added to the loaders that people can start contributing to improve the database and make it a standard.

<!--quoteo(post=1929060:date=Apr 22 2009, 08:51 AM:name=oggzee)--><div class='quotetop'>QUOTE(oggzee @ Apr 22 2009, 08:51 AM) <a href="index.php?act=findpost&pid=1929060"><{POST_SNAPBACK}></a></div><div class='quotemain'><!--quotec-->Sounds good.
If this takes off, I will include it in a future release of cfg usb loader.<!--QuoteEnd--></div><!--QuoteEEnd-->Thanks, you'll be glad to know it's in the final stages of ignition <img src="style_emoticons/<#EMO_DIR#>/wink.gif" style="vertical-align:middle" emoid="" border="0" alt="wink.gif" />
I think the reactions have been encouraging, and I hope it will only take until the end of next week to take more feedback, and prepare support for the various loaders.


<b>Summary of what needs to be done:</b>
- decide on a good structure that can be understood by anyone, what level of complexity is a good balance?
- add support for the XML file in the loaders (I've done that for the first draft for Configurable Loader, easy to improve when the final schema is decided)
- load the file in other programs such as WBFS managers, to easily rename all files on WBFS and verify iso checksums (AlexDP offered to add support to his manager)
- fill the XML with data <img src="style_emoticons/<#EMO_DIR#>/smile.gif" style="vertical-align:middle" emoid="" border="0" alt="smile.gif" />

 

[Deleted member 142963]

McHaggis' example is the best example. Use that one. It's clean, readable, and he knows what he's talking about.

 

[McHaggis]

What I meant about the 500+ games thing was, is it a great idea to download the entire .dat when you only have 20 games? Or is it a better idea to download the XML for the 20 games you have at 4% of the size of the larger .dat file?

I agree with you on the database purpose. But things are rarely perfect on the first draft. You may (er, probably will) find the need to add data or categories at a later date. Naturally, you wouldn't create an entire subset or extension to the XML just for these, it's unnecessary. Any software correctly implementing an older version of the spec would just ignore unknown data because a use wouldn't have been implemented.

...
- i placed accessories as element names because "nunchuck" is not the data that will be directly used, "true" is, and in hope to avoid people adding wrong elements by thinking they can add anything. when a new accessory is released it can be added.
- true/false or 0/1 is faster to type than "balanceboard" and less prone to error if edited by hand.
- minimal structure means less possibility of someone forgetting a quote somewhere
- minimal structure is more humanly readable in my opinion (but maybe I'm just thinking like a frakkin' machine?)
...

I think in order to get the best support from developers, the structure should be geared more towards the ease of parsing, rather than the ease of creating. If you're worried about unlearned users having difficulty adding to the database, the simple approach would be to provide a front-end that simplifies it. This way, the users with know-how can work with good, well structured and easy-to-read XML and the beginners can work with a front end form that has checkboxes for which accessories are used and input boxes for the title (and so on).

That is why I was leaning towards the "minimal XML" approach as described here http://xml.coverpages.org/minxmlspec20000413.html

from http://www.w3schools.com/DTD/dtd_el_vs_attr.asp
"
There are no rules about when to use attributes, and when to use child elements. My experience is that attributes are handy in HTML, but in XML you should try to avoid them. Use child elements if the information feels like data.
"
from http://www.ibm.com/developerworks/xml/library/x-eleatt.html
"
- If you consider the information in question to be part of the essential material that is being expressed or communicated in the XML, put it in an element. For human-readable documents this generally means the core content that is being communicated to the reader.
- data goes in elements, metadata in attributes
- If the information is intended to be read and understood by a person, use elements. In general this guideline places prose in element content.
"

I've never been a fan of minimal XML. It can be advantageous if you're working with a cut-down XML parser (which we might be, I'm not sure), but if your XML parser fully supports something like XPath, there's no excuse for not using attributes in the specification. I certainly wouldn't say that it makes the XML harder to understand. I would definitely disagree with the w3schools advice of avoiding them and I think this is poor advice to someone working with XPath.

I personally like to think of attributes as "switches". If you look at it from a programmatical aspect, an attribute (or multiple attributes) can be used to determine whether or not an element should be processed. As an example, imagine a developer wanted to create a multilingual loader, displaying titles in English on a UK/USA/AUS Wii and in French on a CA/FR Wii. The XPath would be something like this: 'titles/title[@lang="en"]' with the en being "switched" by the loader depending on the language detected.

Another thing - it's much easier to code a loop through a specific set of nodes (for example ) than manually checking the existence of many nodes (, ) not to mention the redundancy of specifying the text of those nodes as false.There is only a single type of main element, which is , and there is no need to load accessories only so it is really one pass to load all data about a game.
The need is to load true/false status for known accessories, not load an unknown list of accessories that's why I put them as elements.
But you're right, adding categories makes perfect sense, even if it is only to have a more organized structure.
You're right about redundancy, accessories should only be mentioned if they are supported, and the required="true" attribute is absolutely necessary.


I know I keep harping on about favouring developers but consider this scenario: I'm a developer and I want to list all the accessories for a game using images from a folder on the sd card, for example /images/accessories/nunchuck.png, /images/accessories/balanceboard.png, etc. The pseudo code, using your example would be:

CODE

IF nunchuck node value is true then load /images/accessories/nunchuck.png
IF balanceboard node value is true then load /images/accessories/balanceboard.png
IF classiccontroller node value is true then load /images/accessories/classiccontroller.png

And so on and so forth, for each accessory known at the time of development. Another IF statement would be needed for each node to check to see if it is a required accessory. Now consider the pseudo code for my example, with both required and optional images in place:

CODE

FOR EACH accessories/accessory node
ÂÂÂÂIF @required attribute load /images/accessories/_required.png
ÂÂÂÂELSE load /images/accessories/.png

Now consider that Nintendo launch a new controller, the scratch and sniff controller

With your example, I have to update my application so that it can see the new accessory in the XML. With my example, I merely have to download a new image for it. And looking at this from a user's point of view, what happens when the developer abandons support for his loader and stops updating it?

I'm far from suggesting my XML was a perfect example, I even struggled when deciding how to write the rating node (i'd quote it but I'm over my quote block quota, pun intended).

I think it's important to remember that if you get developer support, you automatically have end user support. If you just get end user support and no developers integrate your XML because they find it difficult to work with the structure, you just end up having a lot of users updating a database that's not really any good to anyone.

Thanks for being so understanding. It's easy to worry about everyone flaming new people on forums just for having a professional opinion

McH

 

[zorglub07]

Thanks for taking the time to explain in detail, I agree a bit of added complexity is worth it.

What I meant about the 500+ games thing was, is it a great idea to download the entire .dat when you only have 20 games? Or is it a better idea to download the XML for the 20 games you have at 4% of the size of the larger .dat file?It should be easy enough to add a download feature to download info for the installed games, so let's add that to the plan.
The URL should be user defined in config.txt (as well as cover URL), and would be a recent mirror of the Wiki.

This way, the users with know-how can work with good, well structured and easy-to-read XML and the beginners can work with a front end form that has checkboxes for which accessories are used and input boxes for the title (and so on).For beginners I wanted something that would look like a simple list when opened in notepad and look like a spreadsheet if opened with Excel, but providing the .xsd schema with a small tutorial for Excel should make it easy enough. I'll check if Excel doesn't mess things up when saving (which wouldn't surprise me) and will look for an existing front-end solution.

I've never been a fan of minimal XML. It can be advantageous if you're working with a cut-down XML parser (which we might be, I'm not sure), but if your XML parser fully supports something like XPath, there's no excuse for not using attributes in the specification. I certainly wouldn't say that it makes the XML harder to understand. I would definitely disagree with the w3schools advice of avoiding them and I think this is poor advice to someone working with XPath.Xpath is not available with this library on the Wii, but attributes are supported and should be no problem.

CODEFOR EACH accessories/accessory node
ÂÂÂÂIF @required attribute load /images/accessories/_required.png
ÂÂÂÂELSE load /images/accessories/.png

Absolutely right, this made me realize if there is one category that needs to remain "open" it's this one, and it will be implemented like your pseudo code.

QUOTE(McHaggis @ Apr 22 2009, 10:40 PM)

I'm far from suggesting my XML was a perfect example, I even struggled when deciding how to write the rating node (i'd quote it but I'm over my quote block quota, pun intended).

I thought about it and maybe this one can remain simple, there is only one official rating for each version of a game (one for PAL version, one for NTSC version).
The other one can be guessed from the official one for the version, and should not be a part of the database since it's not official data (maybe there are some cases where a game has been given a different rating between PAL/NTSC). The value that will be displayed will be the one matching the game's region, and there could be a setting to override this and always display a "converted" rating matching the user's region for all games. 3 or E is easy enough to guess which rating it is.

I will wait for a couple more days before the final draft to give time to more ideas.

Meanwhile I also got XML loading working with "USB Iso Loader GUI by dimok,nIxxx,cyrex et al" with the first draft, for now it only displays year and publisher under the game ID (under the cover), but this loader seems like the perfect one for nice accessories icons (of course this needs to be optional so users can choose not to display any of this).
It is really easy to upgrade to the new XML format for any loader based on Waninkoko's loader.

 

[Screemer]

i would really like to see this project evolve to a point where it is used in nearly any loader.

 

[McHaggis]

I can see a lot of potential in this from the GUIs. Things like hiding games with high ratings or that require an accessory that you don't have and sorting options (by release date, publisher, etc).

I'm looking forward to seeing the draft spec.

 

[zorglub07]

Thanks for your support Screemer and McHaggis,
Yes, hiding by rating will be one of the possible benefits

I think the final format will look pretty much like the example you showed, with maybe rating kept simple (is this a bad idea? I think it is easy to get ESRB from PEGI and vice versa), and publisher removed:
The last two digits of the game ID is the publisher code, too bad it's missing from the current titles.txt,
I think the new XML format should allow both 4 and 6 digits ID and remove the publisher category since with XML it will be easy to read the publisher name from the game ID
By including a list of "publishers codes/real names" at the top of the the XML structure, a loader that supports it will always be up to date if there is a new publisher, there is no need for a separate file.
CODE
ÂÂÂÂNintendo

This won't change anything for the USB Loaders since the game titles will keep being identified using only the first 4 characters of the game ID


Summary of USB Loader-related authors who showed interest in the new XML format:
- Configurable USB Loader (by oggzee, the author that implemented titles.txt in the beginning)
- Coverflow USB Loader (by Don Killah and Cdreams)
- WBFS Manager (AlexDP)

 

[Screemer]

i think PrixM would be interested, too. he has one of the best wbfs guis out there.

 

[redia]

i think PrixM would be interested, too. he has one of the best wbfs guis out there.

I second that...
you can find it in : http://gbatemp.net/index.php?showtopic=148613

Cheers,
R

 

[Knocks]

Were you able to get Unicode characters in your XML file? There has to be a way to display those ridiculously long Japanese titles in kanji. Right now the loader ignores a titles.txt if it's saved in Unicode.

 

[zorglub07]

XML supports UTF-8 so japanese could be displayed if the USB Loader supports UTF-8, this is bugging me as well because currently "é" and "è" accents are supported by the XML, but don't appear properly in the USB Loaders. The USB Loaders would also need a japanese font.

I was thinking about not including the Japanese as a title category because the English title would be the default title for NTSC games for both English and Japanese, but I will leave it, so that there could be an option in the Loaders to display romanized or translated English title for Japanese games, or the original Japanese title.

 

[Screemer]

what i don't get is why non of the coders of the libwiigui loader posted in here. this project would offer a lot what they could implement in their loader. maybe hungyip84 would be interested to implement it.

 

[ttabbal]

Looks like some good progress here. One thing I would add, and would be willing to put together if you like, is an XSD. That would allow developers to have a simple way to validate the XML data. It also allows you to use various tools that make editing XML a breeze. The XSD is a specification file that tells programs how to read/write the XML.

Another side thought... with this stuff we could add a webservice to query the titleID and return the XML for that game. So we could have a master list online, but you only download the ones you need/want in the loaders. Not too hard to do, but really nice for end users. And the loader could do it for you. It doesn't really need to be a full-out webservice, it could be as simple as XML files for each game stored in a directory on the webserver, much like how the cover downloading works now. On the user side, could keep them separate or join them together. I would think a simple online form to edit the master database would work well. Sort of a Wiki approach. So the end users never really need to edit the XML if they just want to use the info. A bulk update so that experienced users can upload a big chunk of XML all at once would be nice as well.

 

[zorglub07]

Yes, those two points will be part of the big plan.
The XSD schema will be provided (read my latest mammoth post) but you will be welcome to edit it if you find room for improvement which will probably be the case

About downloading info from the Loaders and an online form McHaggis suggested something like this and it will be a nice feature.
About downloading on a per-game basis, I expect the full file will be less than 1MB, more like 600KB and for now it's less than 100KB, so if bandwidth is not found to be a major issue, 600KB doesn't seem like it would take long to download, this is less than it takes to download 10 covers for the USB Loaders. and if it is zipped automatically then extracted on the SD by the Loaders, we're talking about 70KB, that's the equivalent of one cover image

Tantric ported Minizip library to the Wii so it should be really easy to add, I'll try to add it.
I first need to add support to the Loaders for the final version when it's decided (just one more day now just so that everyone interested has time to think about it), then fill the xml with a minimum of data so that Loaders can start using it.
But right after that, the real fun will begin because nice options could be added to the Loaders (sort by Year, by Publisher, by age rating, block game per age rating, accessory icons...), then we could race to fill the data as much as we can, maybe by hosting the file as text on the current GBAtemp Wiki or on Google doc maybe, then set up or find an existing solution for a wiki-like website. I'd prefer a simple existing solution, if only google doc supported XML it would be nice.

 

[McHaggis]

Having the publisher at the top is probably a good idea. It's one less field to add to each game and you won't get any instances of misspellings and typos.

I'm still unsure of merging the ratings. Although cases will be rare, you could find games that are rated completely differently by ESRB and PEGI. For completeness, I'd like to see the content descriptors each rating authority uses too, but I know it's not really important. It would just be nice to see the icons show up in the loaders.

the definition of irony: I just typo'd the word "typo" and had to correct it

 

[zorglub07]

Although cases will be rare, you could find games that are rated completely differently by ESRB and PEGI.

Exactly, I meant the only rating for each game would be the one matching the game's version for this reason, "rating conversion" would only be used as an option so that users can choose to display real ratings or converted ratings for their region, even if they are not 100% accurate. for PAL games the database would contain the PEGI rating (USK is virtually the same and PEGI is on German covers so to keep things simple PEGI is enough), for NTSC games ESRB, and for Japanese games CERO. There is also OFLC for PAL Australia/New Zealand and PEGI doesn't appear on their covers, but I suppose they don't have a different PAL version so PEGI will have to do.
Actually, icons for rating would be a bit redundant since they appear on the cover, but I think they would still be a nice touch, easier to read.
Content descriptors could be stored in a main category that would include conversion settings, under and before the list of games.

About checksums, should we use CRC or MD5? traditionally, CRC is used in the .dat for emulated systems, but I've only seen MD5 used for Wii games so I would lean towards MD5. The CRC in the Offlinelist dat for Wii are all fake (they are only placeholders), so by only allowing MD5 we can avoid any confusion.

 

[McHaggis]

Isn't the game ID be the same for each region? So PAL/NTSC would share the same section of XML code... my mistake if it differs. RSPE01 is Wii Sports in every region, right?

As for the checksums, I'm not really sure which is used most since I only play my own backups