RX580 8GB Pulse framedrops

I am getting random FPS drops with my i5 4460/RX580 in place with 2GB of ram allocated, it started happening as soon as I swapped my old 2GB GTX950 card with the RX580. I tested it both on Linux and Windows with the adrenalin driver.

Results were the exact same, sometimes I am able to reach 400/15 fps at average with OptiFine, effect is exactly the same with vanilla too, I can feel stutters every now and then and feel the frames jump from as low as 18 to 120 fps, even worse with vsync.

1.8.9 is a whole different story, I get stable and smooth frames.