Fix IsomorphismPermGroupOrFailFpGroup to honor its second argument, which limits the coset table size that gets used before it gives up

Fix IsomorphismPermGroupOrFailFpGroup to honor its second argument, which limits the coset table size that gets used before it gives up #6110