The speed increase of efficiency ONLY applies to blocks that the tool is appropriate for.

An efficiency enchantment only applies to blocks that the tool is appropriate for. For example: mining a wooden blok with an efficiency IV pickaxe is as fast as mining it with your hand.