I found an odd fix for anyone who has the same problem.
The map was actually Rp_downtown_v2_fiend (something)
Everytime I would start gmod up fresh it would crash if trying to connect, even on singleplayer (on that specific map)
I loaded construct, disconnected, loaded rp_downtown_v2_fiend (singleplayer)
then i connected to the server.
Seems weird, but I've had to do it 5 or so times now. If i don't do that order it doesn't work for me.
I think it has somethign to do with the added material on that map conflicting with certain things.
No I do not believe taht it is a hardware issue