FPS drops with low frametimes

[Luhketa]

Hey everyone, this is a technical question I couldn't find the answer to. I'm running a R9 5900x and 3080 on a HP Reverb G2 and I'm getting fps drops even though my frametimes always stays below 11,1ms. From what I understand, FPS should always be at least 1000/frametime. Am I missing something? It happens in iRacing too. I'll get FPS drops with my frame times being in the 6-8ms range, shouldn't that be good for 125-166 FPS? Why can't I maintain 90 FPS without frametime spikes? I'm using fpsVR to monitor this. I've attached a video to demonstrate the issue.