About our crawler
Companies Radar lists technology companies. To keep company profiles accurate, our crawler (the “Data Feeder”) occasionally reads the public homepage of a company's website. Everything it finds is reviewed by our team before anything is published.
How to recognise it
Our requests identify themselves with this User-Agent:
Mozilla/5.0 (compatible; CompaniesRadarBot/<version>; +https://companiesradar.com/about-our-crawler)
What it collects – and what it doesn't
- Business information only: company name, website, description, country and address, general business contact (such as info@ or sales@ addresses and the main phone number), links to official social media profiles, and the company's own logo.
- It does not collect personal data about employees, photos, screenshots, or content from other directories and review sites.
- Descriptions found on websites are kept only as a reference; profiles are published only with text written by our team or supplied by the company.
How it behaves
- It follows your
robots.txt, includingCrawl-delay. - It makes one request at a time to your site, waits between requests, and backs off when your server is busy.
- It reads the homepage and your logo – it does not crawl your whole site.
Website health checks
For listed companies we also check that the website still works – about once a month, and daily for a few days while it is failing. A check is one request for your homepage with the same User-Agent (like an uptime monitor: we look at the status and the security headers, not the content), a look at your HTTPS certificate, and public DNS and domain-registration records. It also checks that the profile's social and product links still exist. The result feeds the profile's Website Score; the details are shown only to the company and our team. Because these checks do not read your pages, a robots.txt rule does not stop them; they end when the profile is removed.
How to block it
Add these lines to the robots.txt file at the root of your website:
User-agent: CompaniesRadarBot Disallow: /
Corrections and removal
Company representatives can ask us to correct or remove a profile. A contact address will be published here soon. Once a removal is approved, your domain is added to a do-not-list, so our crawler and imports never add it again.