You have a few options for capping your fps:
The -frameLimit 1 in launch options should do the trick, the 1 is for a 1 to 1 ratio of your monitor refresh rate, any value above is exponential, 2 for 1/2, etc.
Use RTSS (Riva Tuner Statistics Server) to cap your fps, this is a component of MSI AfterBurner, a graphics card overclocking utility. Alternatively you could use EVGA Precision, which has its own on-screen statistics display utility built in.
You could try using the nVidia Inspector software to cap your in-game fps by setting up a game profile for GTA V.
Doing a quick search I found this:
-refreshrate 60 -framelimit 1
Try that in place of just the -framelimit 1 launch option, hopefully that will define the refresh rate parameter as well as lock it to 60 with the -framelimit 1 launch command.
The best thing you can do is to try and limit your fps, this will first and foremost help your recordings/streams via OBS by reducing system load when running your game/s. Lowering your graphic settings a tad if just limiting your fps doesn't help, will help a bit.
Turning off the preview window once you verify OBS has hooked into the game should help a little as well.
Your gtx760 is a little like my r9 290x, getting a bit long in the tooth yet still strong.
A more powerful GPU will most certainly assist in regards to performance overall, especially when using the NVENC encoder to record/stream. Another limiting factor for you is that with your i5 7500 you only have 4 cores, even when using the NVENC encoder your CPU has to deal with the load of the data transfer from and to GPU, disk, etc. When streaming it can add another 10-20% load, dependent upon settings and bandwidth usage. For streaming and recording games the more cores you have the better the performance and more quality you can push out.
If you still have your AMD setup, you might be able to do a 2 PC stream setup, where you have one PC dedicated to doing the stream and one PC dedicated to the game.
https://help.twitch.tv/customer/portal/articles/1988680-broadcasting-with-two-computers#hardware
I could not find a guide with for a capture card setup, whether it be a single or dual PC environment, sorry.