miscoranda - User-Agent Abuse

source: above

According to RFC 2616, the User-Agent header is a statistical datapoint and capability preference, allowing the receiving site to serve pages based on what the client is known to be able to receive: "This [header] is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations." So if the limitations of your user agent change, you can modify the User-Agent field that you send appropriately.

With this in mind, I often set my User-Agent header to "Mozilla/5.0 (Something)" when I'm using wget, curl, or urllib in Python, but I'm often told that this is a bad thing, even an abuse of the header. That's absurd; the abuse is usually on the server side, not the client. I fake the User-Agent because many sites don't allow download via curl or wget?two that spring immediately to mind are google.com and f2o.org. These sites have a legitimate practical reason to do so: presumably a high percentage of the hits they receive from these user agents are crawlers and bots. With Google especially, this is going to cost them a lot of money, so blocking is prudent.

But bots should adhere to robots.txt, and I'll bet that a significant portion of the requests that curl and wget banning sites receive from those clients are legitimate. Their filtering is, therefore, a technical solution to a societal problem. It's a bit like banning Firefox on a framed site because Firefox can display the content unframed. So whilst I realise that banning the clients server side is something that pragmatically just has to be done, a hack to save a lot of money and bandwidth, it's an abuse of the User-Agent header, and it's taking place on the server. Getting around that by faking the User-Agent header client side is abuse by neither morals nor specification, as long as the client is being used legitimately.

well a user who access the sever with some wierd tool which lies about it's identity is almost always up to no good.  Faking your user agent is acting like those criminals.   The point is not whether you think you are doing something that is legitimate.  The point is whether the server has provided for that kind of access ... and if it has, then it will allow user agents accordingly. It seems to me that you are arguing for acting like a criminal.

I filter bots and some mischief from my Chat Who's Here page because otherwise the page would be useless to me.  That page forms the basis of my awareness of what is happening to my domain in real time.  If people are honest about their user agent, like most of the legitimate bots, then i can make that page more useful.  If people lie about the tools with which they access my site, then it makes it more difficult to make awareness pages like this one useful. 


  1. sbp
  2. user agent
  3. abuse
  4. chat-who-is-here

See Also

  1. Thought Browsing the Semantic Web with 77 viewings related by tag "sbp".
  2. Thought [title (2415)] with 4 viewings related by tag "chat-who-is-here".
  3. Thought swhack with 3 viewings related by tag "sbp".
  4. Thought [title (2416)] with 3 viewings related by tag "chat-who-is-here".
  5. Thought Chat Who's Here with 3 viewings related by tag "chat-who-is-here".
  6. Thought There are two sides to this Privacy Issue with 0 viewings related by tag "chat-who-is-here".
  7. Thought [title (3126)] with 0 viewings related by tag "chat-who-is-here".
  8. Thought I wanted to see ... with 0 viewings related by tag "chat-who-is-here".
  9. Thought about: Validate With Logos with 0 viewings related by tag "sbp".
  10. Thought Chat Rooms with 0 viewings related by tag "chat-who-is-here".
  11. Thought [title (2412)] with 0 viewings related by tag "chat-who-is-here".
  12. Thought [title (2414)] with 0 viewings related by tag "chat-who-is-here".
  13. Thought [title (2426)] with 0 viewings related by tag "chat-who-is-here".
  14. Thought Do you want to be a sperm? with 0 viewings related by tag "chat-who-is-here".
  15. Thought work in progress for Chat Who's Here with 0 viewings related by tag "chat-who-is-here".
  16. Thought i met pineau christophe on fastblogit Chat Who's Here with 0 viewings related by tag "chat-who-is-here".
  17. Thought Flashing Chat Room with 0 viewings related by tag "chat-who-is-here".
  18. Thought Caught a hacker with hands in the cookie jar with 0 viewings related by tag "chat-who-is-here".