What is the most popular video game console that is legal to be modded and reverse engineered and preferably supported by Unity, Game Maker Studio or SDL2?
Insofar as a Steam Machine is a game console and is sold into that space, Steam Machines running plain SteamOS or Windows are supported by Unity and Game Maker Studio. As they are computers, they can be modded, and insofar as open source linux software like much of SteamOS goes, it can be "reverse-engineered" without much difficulty.
Where this kind of falls down is "popular" - there are really only two or three consoles that are actually "popular" - Xboxes, Playstations, and the Nintendo Wii U. Steam Boxes are a niche product, though they are widely popularly known.
In any case, one good example of a moddable steam machine would be the Maingear Drift Steam Machine. It's basically constructed out of off-the-shelf components and runs SteamOS without Windows, removing concerns one might have about inadvertently stepping on MS's toes.