Thanks for all the input. Switching between windowed and fullscreen mode didn't significantly affect performance for me. While Shadow Quality on High, I turned Shadow Map Resolution all the way down to 1024 (with SQ on low it doesn't really make any difference no matter how I adjust it). FPS counter is normally green for me, flicking around 60, turning red when FPS drops.
As far as I remember, with everything maxed out I ended up getting like 30 FPS downs, too (quite often though). Lowering Shadow Map Resolution and AA helped a bit, but not much.
Should I just accept that it won't go any better with the current gear?