Velen is a crawler written in Go for Hunter. It analyzes millions of public internet pages every month. We only crawl pages that are publicly accessible. Any page behind a login or any other authentication won't be crawled. Our goal with this crawler is to build business datasets and machine learning models to better understand the web.
This crawler follows robots.txt and meta instructions. We make sure our crawler doesn't add any noticeable load on websites we visit (usually, this means we'll never crawl more than one page at a time and one page every 2 seconds).
If for whatever reason, you wish to ban this crawler in particular, you should target the User Agent VelenPublicWebCrawler in your robots.txt. We should pick up on this change within a few hours.
If you notice bugs or want to send us any feedback regarding the crawler, contact us at: engineering@velen.io.