There is no board that fits your exact requirements. Get an add-in capture card.
Assuming we allow 4K@30 output, getting 4K out is a pretty simple task since most every board supports HDMI 1.4. Unfortunately, encoding a 4K stream, even at 30 Hz, is an extremely demanding task, and putting that requirement on an SBC makes things... complicated. Let's run through some of the options.
x86 boards with video input
Intel embed-oriented processors—namely Apollo Lake and Cherry Trail—contain support for the MIPI-CSI interface, and there are a variety of boards with support for these processor lines. However, only a few actually have a the requisite connector on the board. From most to least powerful, they are the UP² and PICO-APL3, UP Core Plus, Up Core and ROCK Pi X.
Interfacing with the MIPI-CSI port can be accomplished through the use of an HDMI to CSI bridge. Toshiba manufactures three such chips: the TC358743XBG, TC358840XBG, and TC358870XBG, the latter two of which support 3840x2160@30. However, there is no reference or development board available to my knowledge, and the only consumer boards available—the Auvidea B100, B101, and B102—use the TC358743XBG. Additionally, these boards rely on a V4L2 driver to interface properly—something that obviously isn't available on Windows. While it's possible that they'd work just fine with Windows, it's hard to say. Outside of these, you'd be resorting to designing your own board or trolling AliExpress for an appropriate board.
Further, Apollo Lake and Cherry Trail processors are... not powerful. I wouldn't trust them to perform a high quality software encode, and given they only have HD Graphics 505 at best, I wouldn't trust them to handle it with Quick Sync Video either.
Powerful ARM boards with CSI video input
I know you've already discussed it to some extent, but the Jetson does make a fairly compelling case. It's got the most powerful GPU of any SBC I know, and 4K encode with NVENC should be will within reach—and NVENC looks good, even compared to a software encoder. Unfortunately, it's obviously not going to run standard Windows, so unless your application happens to run on Windows 10 on ARM, you'd have to refactor your application. Further, it uses MIPI-CSI for video input, not HDMI, which puts us back in the same situation as the x86 boards where there's (as of yet) no adapter.
ARM boards with HDMI video input
Similarly, there are a number of boards with HDMI video input. Unfortunately, most of these boards aren't particularly powerful, nor do they support standard desktop Windows, giving us many of the same problems as before.
- i.MX boards: Several i.MX SoCs include built-in support for MIPI-DSI video input, making them a popular and simple choice. Unfortunately, they're generally not powerful.
- Ventana GW5510: i.MX6 ARM 1c A9 @ 800 MHz. Only lists support for 1080p@60 input and output.
- Geniatech NXP i.MX 8M: 4c i.MX 8M ARM 4c A53 + M4 @ 1.5 GHz. Supports 2160p@60 out, but only 1080p@60 in.
- Boundary Devices BD_HDMI_MIPI: Isn't an SBC itself, but allows any i.MX6 or i.MX8 board to use an HDMI input over CSI. However, since it uses the TC358743XBG, it only supports 1080p@60.
- INFORCE 6420: 4c Snapdragon 600E ARM 4c @ 1.7 GHz. Only supports 1080p in and out.
- Orange Pi RK3399: 4c RK3399 ARM A53 + 2c ARM @ 2.0 GHz. Supports 2160p@60 out, but unknown in.
- Tronsmart PAVO M9: Some random settop box that happens to have HDMI in. No idea what the specs are, technically not an SBC, but someone did manage to get something going with it.
Powerful x86 boards without video input
This is probably the most reasonable option. Given the availability of USB and PCI-e based capture cards, using a capture card with a powerful board would allow for everything to work—just with some extra effort and hardware. Given the widespread availability of this type of SBC (and similar NUCs), I won't attempt to enumerate every possible board. However, using software, Intel's Quick Sync Video, Nvidia's NVENC, or AMD's VCE are all possible depending on which exact hardware you get (though you should note that VCE in particular falls behind the others.) Some manufacturers you may want to look at include Udoo, Up, Aaeon, and Ibase.