Track a visited set to prevent clustered samples from being revisited