Ok then put a timer on it

Or maybe have them also roam around the maze looking for you and if they find you and you defeat them they must head over to the gate

That way everyones busy doing something