Is there a plugin or a way to open an external program, like mpv (video player) to open YouTube links from TT-RSS instead a new tab in the browser?
This has nothing to do with TT-RSS, but depends on your browser, so please check your browser’s documentation.
Theoretically you could do it with a custom protocol, would would require adjusting links on the browser side (though I’d suggest a userscript or browser extension rather than modifying tt-rss). If your video player does not register a custom protocol handler then you’d need to create a bridge (on the native, non-browser side) that does so and passes it on to the video player.