remove the duplicate results

[gibran-abdillah]

when the tool has finished searching, remove the duplicate urls. if the result is less than "amount" do the search again until it reaches the inputted amount

[naufalardhani]

by default domhttpx will return output like below,
-https://domain-123.com
-https://domain-321.com
-http://domain-789.com
-https://domain-132.com

but actually if you set the output by adding --real-path it will be like below,
-https://domain-123.com/about/program/5
-https://domain-321.com/news/article.php?id=1
-http://domain-789.com/watch?v=KnZ8h3MRuYg
-https://domain-123.com/about/program/6