Thanks for the clarification, much appreciated. If you have some more spare time to waste on teaching a peasant, I wouldn't mind also having some brief explanation on why the "sparse" nsps can't be compressed to nsz (spitting out errors). An example of sparse nsp is the game Lost in Random.
Please correct me if I'm wrong but I don't think there is any legal way to obtain sparse games. How I assume spare games work is that the Nintendo eShop serves you a spare base game and the 0-day update at the same time and while installing they get combined. This is very similar to how NDV0 fragments get applied to an existing update by only downloading the difference between the update you have and the latest version. Maybe I'm misunderstanding something but I don't think you can just dump sparse games. I even tried MitM intercepting the Nintendo eShop download using the disable_ca_verification and disable_browser_ca_verification patches from misson20000 but unfortunately the Nintendo eShop seems to not be affected by those patches and still applies the CA check. So the only way I see would be to write CA patches to for the Nintendo eShop or use something like plato79's nut fork to directly download from Nintendo CDN which wouldn't be legal. But maybe I'm all wrong and you can just dump sparse games nowadays as I haven't tried dumping them for a long time. Can somebody try to dump one and let me know if dumping them is now possible?
There are also practical reasons for not supporting sparse games. A sparse game without it's update is pointless. Even if you could convert a sparse game to an NSZ the NSZ could never be played or installed without also providing the update linked to it. I might look into the possibility of adding support for sparse games but it's not on the top of my priority list but seams like a fun project if they are legally obtainable. If there is any documentation about the spare NSP file format please let me know.
Nintendo should officially support NCZ in horizon and completely replace the downloading of NCA files for updates and games. I feel like i'm missing something for how hostile some seem to be towards compression. Why would Nintendo want games people buy to take longer to play and update instead of using a open source library or implementing the standard themselves?
Compressing the games would actully save Nintendo a ton of money in CDN cost. Even better they could use block compressed NSZ files to allow players playing games without having to decompress them saving precious eMMC and CD Card storage. On the Nintendo Switch games are only allowed to use 3 out of the 4 cores so the 4th core could be easily used to decompress the data on the fly which would even improving load times as more data can be read using the same SD read speed. I would however expect them to compress things on an NCA level so they can leave the encryption layer unchanged. This is actually exactly what they started to do for some games as far I'm aware but it's extremely rare and I never looked into it. If Nintendo wants to use this file format the MIT license definitely allows them to do so.
I'm not sure on Nintendo's stance towards speeding up the update process, but I have noticed that various Yuzu and/or Ryujinx developers are kind of hostile towards the NSZ format. I did found one instance where someone explained that there are some technical hickups with supporting the ability to run Switch games in the emulators as NSZ files (read: not install and then play, but decompress on the fly). This much I can understand; this affects performance and resources required. What I don't understand however is - and you hear this one more often - is that this standard shouldn't supported because it's main purpose is piracy (file sharing) and that it could put the emulator in legal jeopardy. That's just ridiculous. Compression is just compression. By that logic they might as well not have bothered with the emulator at all, because it's no different than claiming that any point they make about the legality of the emulator just by claiming "most people use it for piracy".
With NDV0 Nintendo already has an awesome way to speed up updates in a much better way but definitely would be worth it for base games and updates where the prior update is missing.
It's so ironic. Back when I developed the NSPZ file format which was the precursor to the NSZ file format I did so with the help from the ReSwitched Discord and advice from the Yuzu team. Back in the NSPZ days they where not against game compression in any way. I feel it's really unfair that only because some pirates started to using the file format it's now considered unethical. Without NSZ pirates would just use NSP/XCI which most of them even do. The entire piracy scene on private trackers is using NSP/XCI as far I'm aware. I find it such a stupid argument that the only reason to compress a game is to illegally distribute it. Everyone that owns a huge game collection will understand that without compression it will get absolutely huge in storage size and I find it very understandable that someone buying games digitally wants to have a personal backup for when the Nintendo Switch eShop will inevitable stop existing at some point in the future. This is also why I mostly focus on NSZ and not XCZ as game cartridges seam for me a quite safe and convenient way to store my physical games.
I even went as far to keep the first 0x4000 bytes encrypted in the NSP/NSZ file format to ensure I’m not decrypting anything that could be considered DRM and to make sure real keys are still needed to play the games. I also made sure not to include any copyrighted material like keys inside my tool. It’s very important for me to keep my tool and the NSZ/XCZ file format totally legal. I even put the statement "This project shall only be used for legally purchased games." in the first paragraph of the README.md file to discourage pirates from using my file format.
The emulators can automatically convert incompatible or too demanding nsz files or tell people what to do for it to work. When I backup a Wii or Gamecube game from a disc I don't even want the uncompressed copy. I'm not sure nsz would actually slow anything down yuzu doesn't use much CPU but it would cap loading speeds at the speed of decompression so it would have to be setup to decompress in parallel. pzstd could be used in the emulators and on the console itself for parallel decompression.
There is absolutely no technical or legal reason for emulators not to support the NSZ file format. With the block compressed NSZ format it's absolutely no issue to directly play them without decompressing them first and solid compressed NSZ files could easily be decompressed automatically. Because the emulator community doesn't want anything to do with NSZ I already know that with NSZ v5.0 I will implement a way to mount block compressed NSZ files as NSP so users can play them on any emulator without decompressing them first.
What's even more interesting is that the same method to directly play block compressed games could be implemented on the Nintendo Switch if SciresM ever completes his FS reimplementation but with the speed of Atmosphere development having slowed down a lot in the past years I'm not sure if this will ever happen. In the ReSwitched Discord SciresM mentioned: "Even keeping in mind that FS reimpl work is kinda temporarily stalled until eol (besides research)," So indeed quite unfortunate.