What are your pc components?
What are your target settings (including resolution and frame rate)?
What fps are you getting in good, bad, and normal situations?
What are the performance statistics of your hardware (when using OSD software such as MSI afterburner)?
For me, there seems to be a bug when starting a new race where the fps is occasionally rubbish - it goes from 60fps to mid to low 30s, and I've even had it as low as 8-10fps! Simply restart the race, maybe even twice and this restores the frame rate.
Also, perhaps the game is running on the limit regarding hardware setup and software settings so a relatively undemanding mode like time trial can get by for acceptable performance and more demanding situations like actual races push things a bit too far.
At the very least, know what fps you are getting in good and bad circumstances, which can be shown via Steam itself (you'll have to enable it in Steams games options) - or much better yet, use a program such as MSI Afterburner that can show OSD (on screen display) information of your current hardware while you're playing games. That way you'll know if you've hit the limit with your hardware or if it's a software based optimisation issue or bug.