Comment by cmeacham98

Comment by cmeacham98 3 months ago

I work for Amazon, but not directly on web crawling.

Based on the internal information I have been able to gather, it is highly unlikely this is actually Amazon. Amazonbot is supposed to respect robots.txt and should always come from an Amazon-owned IP address (You can see verification steps here: https://developer.amazon.com/en/amazonbot).

I've forwarded this internally just in case there is some crazy internal team I'm not aware of pulling this stunt, but I would strongly suggest the author treats this traffic as malicious and lying about its user agent.

xena 3 months ago

Randomly selected IPs from my logs show that 80% of them have the matching that forward confirming reverse DNS domain. The most aggressive ones were from the amazonbot domain.

Believe what you want though. Search for `xeiaso.net` in ticketing if you want proof.

Reply View 6 replies

guardiangod 3 months ago

So you said the IPs are residential IP, but their reverse DNS points to a amazonbot domain? Does that even make sense?

Reply View | 0 replies
petee 3 months ago

Reverse DNS doesn't mean much, they can set it to anything; can you forward match them to any amazon domain?

Reply View | 4 replies
- xena 3 months ago
  
  It's forward confirming reverse DNS. I assumed that everyone does that by default.
  
  Reply View | 3 replies
  
  petee 2 months ago
  
  What everyone does by default doesn't matter really here, it's that an IP owner/user can literally set the reverse to any arbitrary domain regardless if the actual domain has a record for that IP. What matters is both match, thats all I meant
  
  Reply View | 2 replies