Hacking wwt+wit: Wiimms WBFS+ISO Tools

[Wiimm]

2009-11-09 10:04 2,000,000,000 R8PE01.wbfs
2009-11-09 10:05 2,000,000,000 R8PE01.wbf1
2009-11-09 10:06 699,979,776 R8PE01.wbf2

is there anything wrong with my command ??
This is a bug. I remember that I set the output files size to the input file size, but WBFS should be an exception.


QUOTE(pely @ Nov 9 2009, 04:17 AM) BTW: I want to find a program can convert old wbf file to the splited wbfs files, any utils can do this? or you can add this in your todo list?

The next revision should do this with: wwt -p source.wbfs extract --dest dir --split --wbfs

 

[Wiimm]

wwt: Wiimms WBFS Tool v0.15a - 2009-11-10
A command line WBFS tool set for various os.

I have updated the tool. The first post with more details is updated too.

Download:

• wwt-v0.15a-r402-i386.tar.gz (linux/386 version)
• wwt-v0.15a-r402-cygwin.zip (cygwin version)

Highlights/bug fixes of this release:

• Because of some modification in the file subsystem creating of splitted and truncated WBFS files failed.
  .
• I have implemented a new wwt command: EDIT
  The undocumented command EDIT can manipulate the free blocks tables and the disc block assignments. I need this to test the CHECK command.
  .

Last changes:

wwt v0.14a r388 - 2009-11-08

- Progress info shows copy rate in MiB/sec too.
- Buf fix: damaged WDF file if copying from WDF to WDF with --raw.
- wwt CHECK: check WBFS for block errors (report only, no repair yet).

wwt v0.15a r402 - 2009-11-10

- Option --progeess has now a short brother: -P
- Bug fix: iso2wbfs had produced ISO images instead of WBFS.
- Bug fix: wwt EXTRACT --wbfs did not work.
- wwt EDIT (not documented): manipulate block assignments (EXPERIMENTAL!).
ÂÂ I need the EDIT command to make invalid WBFS for testing the CHECK command.
- wwt ID6 --long: print list in format WBFS_FILE/ID6 (wbfs+selector)
- wwt CHECK improved.

Sources comes with GPL2. Checkout the sources directly from the SVN repository or browse the source.

 

[Elfish]

that's not what it's supposed to be right? :x

stackdump content:

Stack trace:
FrameÂÂÂÂ FunctionÂÂArgs
0022C314ÂÂ76BDF003ÂÂ(0000019C, 0000EA60, 00000000, 0022C448)
0022C328ÂÂ76BDEFB2ÂÂ(0000019C, 0000EA60, 000000A4, 0022C370)
0022C448ÂÂ61097F54ÂÂ(00000000, 0000019C, 0022C468, 000000A4)
0022C538ÂÂ61095AEBÂÂ(00000000, 003B0023, 00230000, 0022CE68)
0022C598ÂÂ61095FCBÂÂ(0022C5B0, 00000000, 00000094, 61020C1B)
0022C658ÂÂ61096182ÂÂ(00000F9C, 00000006, 0022C688, 61096383)
0022C668ÂÂ610961ACÂÂ(00000006, 0022CE88, 00000000, 00000000)
0022C688ÂÂ61096383ÂÂ(0022C8E0, FFFFFFFF, 0022C770, 0040B332)
End of stack trace

 

[Wiimm]

I have never seen such error before, but I have made only some tests with cygwin.

Which OS do you use?

And what happens if you use slashes instead of backslash?

 

[Elfish]

seems to work now. pretty weird tbh...

i'm using windows 7

 

[Wiimm]

I have made a run time test:

CODETest System:
- Self made NAS based on VIA Esther processor 1200MHz.
- 1 GiB main memory
- System: Linux, Kernel 2.6.22.5
- Source harddisk: SAMSUNG HD154UI, 1.5 TiB, ext3
- Destination: USB Drive, WD Passport Essentia, 500 GiB, WBFS
- power consumption < 20 Watts
As you see it is not a very fast system. But it's my archive.

I copied 248 games with different sizes. Because I own only about 50 games I build clones of them and change the ID6 to WIM000 up to WIM199. All games are stored in WDF format and the disc usage on the source drive (ext3) is 431 GiB. I copied all games with one ADD command to a previous formatted (=clean) WBFS partition with 462 GiB. While I'm copying the NAS is stil doing other requests (storing and playing DVB-T streams).

$ ls source-dir/*.wdf | wc -l
248

$ du -msh source-dir
431G source-dir

$ date +%T
15:40:42

$ wwt -aP add source-dir
....

$ date +%T
20:50:08

$ wwt ll
....
Total: 248/500 discs, 444792 MiB ~ 434 GiB used, 28312 MiB ~ 28 GiB free.

I have copied 434 GiB = 444792 MiB = 466398 MB in 18566 seconds.
-> 23.96 MiB/sec = 84.2 GiB/hour (base 1024)
or
-> 25.12 MB/sec = 90.4 GB/hour (base 1000)

 

[Skizzo]

Thanks for the update Wiimm. I gave it a go and it appears to be working fine. Changed the big block (5858-6323 on the reports I sent) of the 'free marked as used' sectors which reduced the number of errors on my drive from 478 to 12. I'm going to leave those errors alone for the time being. All new games seem to be installing fine to the newly freed up sectors, with no disc errors reported. Fantastic work! Any plans for any type of automated defrag?

 

[Wiimm]

Is there a real need for defragmentation? There are only a *few* and *very large* games in a WBFS archive (WBFS is not a real file system).
500 GB drives have space for about 250 games and the block size is 8 MiB. This is total different from other filesystems with multiple of thousands small and medium files.

 

[Skizzo]

I guess you're right. I actually thought I had a few games that were heavily fragmented, but that was just from quickly skimming through the dump report. Upon closer inspection, they're not fragmented, just a lot of different entries because they were custom Guitar Hero discs.

On another note, whenever I add a game it always says something like [TITLEID, 2241M], or in the case of a DL game [TITLEID, 4052M]. What is the significance of those numbers? I figured it might tell you how big the game was going to be on the drive after being added, but that's not the case.

 

[W hat]

My most wanted feature is either easy to use Windows batch files or a GUI. Any devs of some other WBFS projects willing to add WDF support?

 

[soulbringer]

Thanks Wiimm this is exactly the app i needed.
just co'd the svn a few days ago
powerful, easy to use, and easy to incorporate into bash scripts.
seems almost perfect imho
geat work

 

[self_slaughter]

just wanted to say thanks for such a great tool.... compiled it on my ps3 and it worked flawlessly

great since my laptop died and i don't have a computer anymore

 

[Wiiwu]

Wiimm, using 0.15a cygwin I found mismatch in .wbfs files created by your tool and wbfs_file.exe.

I see 01fc of data dumped at .wbfs offset 0x2604. In wbfs_file's output this part is 00.

These data appeared to be copied from somewhere in the game partition's game data.

 

[Wiimm]

wwt: Wiimms WBFS Tool v0.16a - 2009-11-15
A command line WBFS tool set for various os.

I have updated the tool. The first post with more details is updated too.

Download this EXPERIMENTAL release:

• wwt-v0.16a-r428-i386.tar.gz (linux/386 version)
• wwt-v0.16a-r428-cygwin.zip (cygwin version)

Highlights/bug fixes of this release:

• While I'm implementing CHECK+REPAIR I have changed some calculations in libwbfs because the using of the 'free blocks table' was buggy. I have made a code view of my changes several times and I think it is absolut compatible in both directions. But because of this modifications I call this release EXPERIMENTAL!.
  .
• Because of the correction of the calculations in libwbfs I propagate libwbfs v1 (v0 prior). It is (should be) absolute compatible to v0 but now all blocks in the wbfs file are used.
  .
• The CHECK command may now fix the detected errors. Use the option --repair=mode to select which errors are fixed.
  This feature is EXPERIMENTAL!
  .
• The new command REPAIR is a short cut for "CHECK --rapair=fbt" -> repair the free blocks table but do not remove any game (non destructive repair).
  .
• The commands ADD, UPDATE. EXTRACT, REMOVE and TRUNCATE check the WBFS before modifying it. If there are any problematic errors detected the WBFS is ignored. If the option --force is set, the test is done but the result is ignored. The option --no-check disables this automatic check.
  .
• If the WBFS becomes full while adding a disc then a cleanup (removing the disc and freeing the blocks) is done.
  .
• "wit COPY" has a new feature: If none of --wdf, --iso and --wbfs is set, then the destinaton file type is taken from the destination filename. Examples:
  

  # copy source to WDF
  wit cp source dest.wdf
  
  # copy source to ISO
  wit cp source dest.iso
  
  # copy source to WBFS
  wit cp source dest.wbfs

Last changes:

wwt v0.15a r402 - 2009-11-10

- Option --progeess has now a short brother: -P
- Bug fix: iso2wbfs had produced ISO images instead of WBFS.
- Bug fix: wwt EXTRACT --wbfs did not work.
- wwt EDIT (not documented): manipulate block assignments (EXPERIMENTAL!).
ÂÂ I need the EDIT command to make invalid WBFS for testing the CHECK command.
- wwt ID6 --long: print list in format WBFS_FILE/ID6 (wbfs+selector).
- wwt CHECK improved.

wwt v0.16a r428 - 2009-11-15

- wwt DUMP layout redesigned.
- libwbfs: free blocks table bug fixes:
ÂÂÂÂ- Fix the free block table while formatting or loading
ÂÂÂÂ- Allocation of free blocks behind 32 bit boundaries
ÂÂÂÂ- Fix table if WBFS if full while adding a game.
ÂÂÂÂ- I propagate libwbfs v1 (v0 prior).
- bug fix: progress info failed if writing splitted files.
- wwt ADD/UPDATE/EXTRACT/REMOVE/TRUNCATE: A CHECK is done automatically.
ÂÂ The WBFS will be ignored if problematic errors are detected.
ÂÂÂÂ --force:ÂÂÂÂPrint messages but ignore result of the check and continue.
ÂÂÂÂ --no-check: Disable the automatic check.
- wwt ADD/UPDATE: Cleanup to previous state if the disc is full.
- wwt REMOVE: new option: --no-free: do not free blocks.
- wwt CHECK: new option: --repair=mode: fix the detected errors.
ÂÂ THIS OPTION IS EXPERIMENTAL!
- New wwt command: REPAIR: Shortcut for: CHECK --repair=fbt
ÂÂ (fbt = Free Blocks Table)ÂÂTHIS COMMAND IS EXPERIMENTAL!

Sources comes with GPL2. Checkout the sources directly from the SVN repository or browse the source.

 

[Wiimm]

Please do not use the REPAIR function on a WBFS without BACKUP..

It seems that there is a bug in it. The bug appeared in the CYGWIN version with a WBFS with a lot of errors (multiple assigned blocks in many files). The owner knows that some games are not playable. He works explicit with WBFS-Manager and USB Loader CFG before.

EDIT:
And this is the error message:

REPAIR /dev/sdf2
* Store fixed 'free blocks table' (7080 bytes).
!! wwt: ERROR #15 [WRITE FILE FAILED] in RepairWBFS() @ wbfs-interface.c#1714
!!ÂÂÂÂÂÂWrite failed [F,7fe400+1ba8]: /dev/sdf2
!!ÂÂÂÂÂÂ-> No space left on device

The writing address 0x7fe400 and the size 0x1ba8 are correct = the free blocks table. perhaps it is only a defect block on the harddrive. The message "No space left on device" comes from the system. But the device has >400 GB reserved for the WBFS .... mmmmhmmm

 

[Blue-K]

First: Amazing work again, Wiim

! Thanks for this great Tool!

Now, it looks like the repair mode worked for me...I had around 300 bad blocks, cleaned it, and the game works still after repairing...great job

!

 

[Wiiwu]

Wimm, I tested 0.16a and it still has the same corruption bug of writing wrong data in .wbfs files it creates, this time it does it at offset 4908.

 

[Wiimm]

It seems that there is a bug in it. The bug appeared in the CYGWIN version with a WBFS with a lot of errors (multiple assigned blocks in many files). The owner knows that some games are not playable. He works explicit with WBFS-Manager and USB Loader CFG before.

EDIT:
And this is the error message:

REPAIR /dev/sdf2
* Store fixed 'free blocks table' (7080 bytes).
!! wwt: ERROR #15 [WRITE FILE FAILED] in RepairWBFS() @ wbfs-interface.c#1714
!!ÂÂÂÂÂÂWrite failed [F,7fe400+1ba8]: /dev/sdf2
!!ÂÂÂÂÂÂ-> No space left on device

The writing address 0x7fe400 and the size 0x1ba8 are correct = the free blocks table. perhaps it is only a defect block on the harddrive. The message "No space left on device" comes from the system. But the device has >400 GB reserved for the WBFS .... mmmmhmmm
I have found the bug: Under cygwin only writing og a whole sector is possible => fixed.


QUOTE(Wiiwu @ Nov 15 2009, 06:24 PM) Wimm, I tested 0.16a and it still has the same corruption bug of writing wrong data in .wbfs files it creates, this time it does it at offset 4908.

This is the next job.

 

[Wiimm]

wwt: Wiimms WBFS Tool v0.16b - 2009-11-15
Bug fix release

Download this EXPERIMENTAL release:

• wwt-v0.16b-r431-i386.tar.gz (linux/386 version)
• wwt-v0.16b-r431-cygwin.zip (cygwin version)

Last changes:

wwt v0.16b r431 - 2009-11-15

- wwt EXTRACT and REMOVE denied options --force and --no-check.
- cygwin: REPAIR can't write unaligned free blocks tables.

 

[Wiimm]

Wiimm, using 0.15a cygwin I found mismatch in .wbfs files created by your tool and wbfs_file.exe.

I see 01fc of data dumped at .wbfs offset 0x2604. In wbfs_file's output this part is 00.

These data appeared to be copied from somewhere in the game partition's game data.

At offset 0x2604 there is the block assigning table of disc #0. It is entry 4482, the first not used entry. (see "wbfs sectors/disc"). That value is not used.

QUOTE(Wiiwu @ Nov 15 2009, 06:24 PM) Wimm, I tested 0.16a and it still has the same corruption bug of writing wrong data in .wbfs files it creates, this time it does it at offset 4908.

I have changed my libwbfs to use more sectors for small WBFS to get smaller wbfs files. This is full compatible.

For a real test you must extract the ISO of both WBFS and compare.