Files Documentation for PES/WE Games for N3DS

[marqisspes6]

Hello guys,

I'm a modder from the PES/WE community, mostly the old gen ps2 style games, I did some small mods on others games but not so relevant as with the ps2 engine games, however, I wanted to leave a little bit of documentation for the N3DS saga PES/WE games.

Declaimer:
I'm using a .3ds iso decrypted and citra, I don't have a physical N3DS, I wish, but they're quite expensive in my country

okay, so for unpacking im using the same set of tools for decrypt the .3ds iso and i notice that it creates 2 folders and a few binary files which seems to be useful for this tool to repack the iso, im gonna focus on the folder because they seem to be the more important now and i think we shouln't actually change the other binary files

okay, lets got with romfs inside there we have a folder cpk, which is the main folder that contains all the files that the game calls it will contains all the models, textures, database of players, etc, for those who came from pc, ps2 or psp modding those are our AFS files

gonna leave a little map

dt_00 <- seems to be equivalent to 0_sound
dt_01 <- seems to be equivalent to 0_text
dt_02_x <- seems to be equivalent to x_sound
dt_03_x <- seems to be equivalent to x_text
dt_03_0 <- this doesnt seems to have an equivalent on the pc, ps2 or psp games, must be something new that konami decided to add

now, for those who never edited pes/we on the other platform im gonna explain a little bit

dt_00 will contain the BGM for the game and the chants of the teams, seems to be in HCA format, you can find more info of the file format here http://wiki.xentax.com/index.php/HCA_Audio also there's a list of programs to encode and decode this format
dt_01 will contain the 3d models for anything use in the game being, face and hairs of players, body model, balls model, stadium models, trophy model, tunnel models, trophy, etc. also contains a textures, like kits, balls, faces and hairs of players, banners, etc the database of the game, opd files (which are the uv map for the textures on the dt_03_x ) i may be missing some files but that's basically what i remember there's in there
dt_02_x will contain the commentator sounds, it also has HCA files, but with this you're gonna be able to create your own commentary files
dt_03_x will contain textures used in the game for most menues and some game situations, also some of the strings we see on menues
dt_03_0 i dont know, but i did notice it contains some textures too

that's the most relevant folder inside roomfs, as for the other folders im not gonna dive into that, maybe someone else with more knowledge on N3DS than me could do that

exefs folder contains a few files, one of them is code.bin this file contains a some important stuff that some ps2/psp modders know very well is the equivalent of our SLXX or eboot file, with that we can incress the quantity of faces, hairs, kits, stadiums, balls in the game, but that's not the main target of this thread

MAGIC NUMBERS

*** At this point I hope that you have already decompress the .cpk files, so if you didn't done that then please get any cpk decompressor and decompress the .cpk files before reading below ***

the same as in the other platform of the game here we share the same magic numbers, so is not gonna be hard to locate some files, so im gonna explain a little bit from my experience with PES/WE games what we can find

Declaimer: im gonna skip dt_00 and dt_02_x because those only contains audio files, not any other file and i already left documentation about it

lets start with one of the easiest one dt_03_x, im gonna use dt_03_m to start with

I wrote a simple python script to fast check the magic numbers and easily find the files, just place it in the same folder as your IDXXXX files

Spoiler: python scrypt

import os

if __name__ == "__main__":
    for file_in_path in os.listdir("."):
        if file_in_path.endswith(".py"): continue
        file = open(file_in_path, "rb")
        magic_number = file.read(4)
        file.close()
        print("File with name: %s has magic number %s" % (file_in_path, magic_number))

and here you can find the result of my execution

Spoiler: results

File with name: ID00000 has magic number b'\x00\x06\x01\x00'
File with name: ID00001 has magic number b'\x00\x02\x00\x00'
File with name: ID00002 has magic number b'\x00\x02\x00\x00'
File with name: ID00003 has magic number b'\x00\x02\x00\x00'
File with name: ID00004 has magic number b'\x00\x02\x00\x00'
File with name: ID00005 has magic number b'\x00\x02\x00\x00'
File with name: ID00006 has magic number b'\x00\x02\x00\x00'
File with name: ID00007 has magic number b'\x00\x02\x00\x00'
File with name: ID00008 has magic number b'\x00\x02\x00\x00'
File with name: ID00009 has magic number b'\x00\x02\x00\x00'
File with name: ID00010 has magic number b'\x00\x02\x00\x00'
File with name: ID00011 has magic number b'\x00\x02\x00\x00'
File with name: ID00012 has magic number b'\x00\x02\x00\x00'
File with name: ID00013 has magic number b'\x00\x02\x00\x00'
File with name: ID00014 has magic number b'\x00\x02\x00\x00'
File with name: ID00015 has magic number b'\x00\x02\x00\x00'
File with name: ID00016 has magic number b'\x00\x02\x00\x00'
File with name: ID00017 has magic number b'\x00\x02\x00\x00'
File with name: ID00018 has magic number b'\x00\x02\x00\x00'
File with name: ID00019 has magic number b'\x00\x02\x00\x00'
File with name: ID00020 has magic number b'\x00\x02\x00\x00'
File with name: ID00021 has magic number b'\x00\x02\x00\x00'
File with name: ID00022 has magic number b'\x00\x02\x00\x00'
File with name: ID00023 has magic number b'\x00\x02\x00\x00'
File with name: ID00024 has magic number b'\x00\x02\x00\x00'
File with name: ID00025 has magic number b'\x00\x02\x00\x00'
File with name: ID00026 has magic number b'\x00\x02\x00\x00'
File with name: ID00027 has magic number b'\x00\x02\x00\x00'
File with name: ID00028 has magic number b'\x00\x02\x00\x00'
File with name: ID00029 has magic number b'\x00\x02\x00\x00'
File with name: ID00030 has magic number b'\x00\x02\x00\x00'
File with name: ID00031 has magic number b'\x00\x02\x00\x00'
File with name: ID00032 has magic number b'\x00\x02\x00\x00'
File with name: ID00033 has magic number b'\x00\x02\x00\x00'
File with name: ID00034 has magic number b'\x00\x02\x00\x00'
File with name: ID00035 has magic number b'\x00\x02\x00\x00'
File with name: ID00036 has magic number b'\x00\x02\x00\x00'
File with name: ID00037 has magic number b'\x00\x02\x00\x00'
File with name: ID00038 has magic number b'\x00\x02\x00\x00'
File with name: ID00039 has magic number b'\x00\x02\x00\x00'
File with name: ID00040 has magic number b'\x00\x02\x00\x00'
File with name: ID00041 has magic number b'\x00\x02\x00\x00'
File with name: ID00042 has magic number b'\x00\x02\x00\x00'
File with name: ID00043 has magic number b'\x00\x02\x00\x00'
File with name: ID00044 has magic number b'\x00\x02\x00\x00'
File with name: ID00045 has magic number b'\x00\x02\x00\x00'
File with name: ID00046 has magic number b'\x00\x02\x00\x00'
File with name: ID00047 has magic number b'\x00\x02\x00\x00'
File with name: ID00048 has magic number b'\x00\x02\x00\x00'
File with name: ID00049 has magic number b'\x00\x02\x00\x00'
File with name: ID00050 has magic number b'\x00\x02\x00\x00'
File with name: ID00051 has magic number b'\x00\x02\x00\x00'
File with name: ID00052 has magic number b'\x00\x02\x00\x00'
File with name: ID00053 has magic number b'\x00\x02\x00\x00'
File with name: ID00054 has magic number b'\x00\x02\x00\x00'
File with name: ID00055 has magic number b'\x00\x02\x00\x00'
File with name: ID00056 has magic number b'\x00\x02\x00\x00'
File with name: ID00057 has magic number b'\x00\x02\x00\x00'
File with name: ID00058 has magic number b'\x00\x02\x00\x00'
File with name: ID00059 has magic number b'\x00\x02\x00\x00'
File with name: ID00060 has magic number b'\x00\x02\x00\x00'
File with name: ID00061 has magic number b'\x00\x02\x00\x00'
File with name: ID00062 has magic number b'\x00\x02\x00\x00'

well in resume... we have 62 files with the magic number 00 02 00 00 and one with 00 06 01 00

00 02 00 00 = textures files with filename and zlib compression
00 06 01 00 = a compressed file with zlib, also i notice on other files that as long as we have a 01 on the 3 byte position so lets say XX XX 01 XX is always gonna be a zlib file

lets start with the easy one 00 06 01 00:

on this kind of files we have the following structure

DWORD MAGIC_NUMBER;
DWORD COMPRESSED_SIZE;
DWORD DECOMPRESSED_SIZE;
BYTE[20] ZERO_FILL_DATA;
BYTE[COMPRESSED_SIZE] ZLIB_DATA;

so basically a header of 32 bytes and then the zlib data

Now lets talk about the next file 00 02 00 00

DWORD MAGIC_NUMBER;
DWORD FILE_SIZE; //MINUS 32 BYTES WHICH IS THE HEADER
BYTE[24] ZERO_FILL_DATA;
DWORD TOTAL_FILES;
DWORD OFFSET_OF_FILENAMES_TABLE;
DWORD OFFSET_OF_FILE_DATA_TABLE;

FILENAMES TABLE
//depending on how many files we have we're gonna have more or less entries
DWORD OFFSET_FILENAME_x;
FILE DATA TABLE
//depending on how many files we have we're gonna have more or less entries
DWORD OFFSET_FILE_DATA_x;

BYTE[16] FILENAME; //although i think this can change depending on how we set up the offsets on the table

FILE DATA follows the same structure of 00 06 01 00 magic number but this magic number is 00 03 01 00 that's the only difference

Now lets write a script to read this type of files and decompress them

Spoiler: python script for txs files

import struct, io, zlib

class TextureFile():
    file_name = ""
    file_data = bytearray()
    file_data_size = 0
    file_data_decompress = 0
  
    def get_filename_from_offset(self, file, offset):
        file.seek(offset)
        tmp = file.read()
        self.file_name = tmp.partition(bytearray([0x00]))[0].decode("utf8")

    def get_file_data_from_offset(self, file, offset):
        file.seek(offset)
        tmp = file.read()
        magic_number, self.file_data_size, self.file_data_decompress = struct.unpack("<3I", tmp[:12])
        if magic_number != 0x00010300: return
        self.file_data = tmp[: HEADER_SIZE + self.file_data_size]

    def decompress(self, folder=None):
        if folder is None: folder = "."
        decompressed_bytes = zlib.decompress(self.file_data[HEADER_SIZE:])
        if self.file_data_decompress != len(decompressed_bytes): raise Exception("Error while decompressing!!! size doesn't match")
        file = open("%s/%s" % (folder, self.file_name), "wb")
        file.write(decompressed_bytes)
        file.close()

HEADER_SIZE = 32

if __name__ == "__main__":
  
    file_path = "ID00001" # Here we pass the file we want to decompress
    file = open(file_path, "rb")
  
    magic_number = file.read(4)
    if magic_number != bytearray([0x00, 0x02, 0x00, 0x00,]):
        file.close()
        raise Exception("Not a txs file")

    file_size = struct.unpack("<I", file.read(4))[0]
    file.seek(HEADER_SIZE, 0)
    file_with_no_header = io.BytesIO(file.read(file_size))
    file.close()
  
    total_files, offset_filename_table, offset_file_data_table = struct.unpack("<3I", file_with_no_header.read(12))
  
    i = 0
  
    texture_files = [TextureFile()] * total_files
  
    for i, texture_file in enumerate(texture_files):

        file_with_no_header.seek(offset_filename_table + i * 4)
        offset = struct.unpack("<I", file_with_no_header.read(4))[0]
        texture_file.get_filename_from_offset(file_with_no_header, offset)

        file_with_no_header.seek(offset_file_data_table + i * 4)
        offset = struct.unpack("<I", file_with_no_header.read(4))[0]
        texture_file.get_file_data_from_offset(file_with_no_header, offset)

        print(texture_file.file_name)
        print("Decompressing %s..." % (texture_file.file_name))
        texture_file.decompress()
        print("Finished!")

you can always improve my code work
okay now we have the files already decompressed lets check them!!!

this is one of the files inside ID00001

and yes, i also have documentation of this file

DWORD MAGIC_NUMBER;
DWORD TOTAL_TEXTURES; // I GUESS? COULD BE ALSO PLATFORM, ON PC, PSP AND PS2 IS ALWAYS 01 00 00 00 ON 3DS IS 01 73 64 33
DWORD FILE_SIZE;
DWORD TEXTURE_ID;
WORD PIXELS_OFFSET;
WORD PALLETE_OFFSET;
WORD WIDTH;
WORD HEIGHT;
BYTE[2] UNKNOWN_0;
BYTE STRETCHING_H;
BYTE STRETCHING_V;
BYTE[12] UNKNOWN_1;
WORD WIDTH_2;
WORD HEIGHT_2;
BYTE[16] UNKNOWN_3;
WORD PIXELS_OFFSET_2;
WORD PALLETE_OFFSET_2;

lets check the pallete bytes

how do we determinate how many pallete bytes we have? well from what i know with the files on pc, ps2 and psp... basically is always start of pallete until start of pixels on this case our size will be 1024 bytes for the pallete which i believe its gonna be a 256 colors pallete + alpha something like RGBA or any format i haven't dive into that yet

this is gonna be the start of our pallete

and this the end of the pallete and the unselected bytes will be the start of pixels

now... how do we know how many pixels we have?

welll i found two ways, one is doing a simple calculation width * height or FILE_SIZE - PIXELS_OFFSET both will work, in fact, i prefer the last one because i notice that on some textures this formula doesnt work, im not sure yet why it doesnt work... maybe is related to the type of texture

but on this case we have a texture with width 32 and height 128 which is gonna give us a total of 4096 bytes

well... now im stuck, because i can't actually found a way to convert this into a image format, being png, bmp, dds, whatever, if someone wants to dive into this and see where we can get its gonna be amazing


UPDATE:
seems like the pallete is in the format of ABRG after comparing many exported textures with png that i get from citra dumps i believe i can determinate that lets see about the pixels...


PS. Later i will update with information of 3D models files structure and how to convert them into obj