The Mario 64 source is out there and fairly easy to compile. You could probably tweak things with minimal coding knowledge to increase damage outputs from enemies/hazards, reduce invincibility frames after being hit, maybe tighten frame counts to execute moves, possibly damage from drops, increase enemy boss health, increase enemy spawn rates/lower timers (be careful -- there was that amusing many many goombas video but that is a bit more specialist and too much might trouble the game), lower health caps if that works for you (might have to skip some water levels or tweak something, though playing level editor to remove some coins might be a thing to consider), enable the water does not heal you thing from one of the later levels in all of them, if there are any of those health hearts in later levels (they probably left it in the first as a quasi tutorial/can't fail this) and much more besides as most of those are going to be simple numbers and multipliers*, or cause and effect you can comment out/delete statements for. Adding something like a poison mode (or one up mushrooms remove life, possibly amusing in conjunction with the thing below), might be a bit harder but you do then have options in that sort for a bit more programming skill.
*seriously get the code, figure out how to compile it (a bit more fiddly than zipping up some files but not by much), open up the code and search for obvious things like health, life, mushrooms, air, damage, fall, drop, jump, coins... and you will probably find things add, subtract, multiply, set ranges/timers, check for certain conditions (the water does not heal for instance, might be a flag in level data but could also be in text or forced to be on all the time).
Set your own challenges, like speedruners do.
Better Mario 64 challenge