Java 64bit

When i use java 64bit minecraft lags from the menu and when i try to play. Have allocated 4Gb of 8Gb ram and it gives me 2-3 fps. When i use 32bit java, there is no lag, and i can play. But i want to play with mods, so i need more more memory than 32bit can give me.