Raids stop for no apparent reason after a few waves

When starting a raid with a functional user-made village (beds and villager job stations exist), most times the raid will randomly stop after a few waves (the bar at the top disappears and no further waves spawn, currently experienced after the second, third and fifth wave). 

 

The issue has currently only occurred while already having the "Hero of the Village" effect, playing in hard mode.