Huge FPS drops when in the inventory or the menu

Fairly recently this has been a problem where when I go into any menu (inventory, main menu, pause menu) my FPS drops to about 5 making the game unenjoyable. However, hen I am in-game it runs fine. I tried changing somethings in settings he only thing that did something was windowed mode is slightly better than full-screen but besides that nothing else changed it.