Bots

You can see what a bot sees for a thought by adding ?bot=1 to the end of a thought url.

Bots are detected in config.php and the php function isBot() identifies bots according to a regular expression which is manually updated periodically as new bots are discovered.

Detected bots are recorded in the bots table with allow=1 for bots that were allowed through and 0 for ones that were rejected.

Allowed bots are served the content of td/php/botthought.php  on thoughts. Bots are only allowed to access thoughts.

The sitemap.txt file is updated with the last 2000 thoughts whenever a new thought is created. This file is referenced in robots.txt so that all bots can find it to index new thoughts from. Bots are also allowed to find thoughts from the home page tag cloud. Note that to be included in the sitemap a thought must have statuspublish” (not private or muffle) and not be a draft or in the back room.

Bots attempting to follow links to anything other than a thought will receive an instant 403 Forbidden error with very little code having been run and only the bot tracking database query. Very efficient.

Currently widget content does not appear to bots, only thought body and comment body content.
 

Tags

  1. google bot
  2. bots
  3. robots
  4. search engine
  5. google
  6. bing
  7. yahoo

Comments


Seth says

See Also

  1. Thought Will the GoogleBot index this thought? with 140 viewings related by tag "google".
  2. Thought Watching our indexing at Google with 121 viewings related by tag "google".
  3. Thought about: How some Losers play the RWG - comment 67990 - comment 68201 with 92 viewings related by tag "google".
  4. Thought My Google Saves with 60 viewings related by tag "google".
  5. Thought about: Tutorvista.com - Online Tutoring, Homework Help in Math, Science & English By Expert Tutors with 26 viewings related by tag "bots".
  6. Thought Hmmmmm..... with 26 viewings related by tag "robots".
  7. Thought Email Chatbots with SparkPost with 25 viewings related by tag "bots".
  8. Thought [title (23165)] with 12 viewings related by tag "robots".
  9. Thought Google Animations with 6 viewings related by tag "google".
  10. Thought about: research blog: inceptionism: going deeper into neural networks with 4 viewings related by tag "google".
  11. Thought Google Offline Areas with 4 viewings related by tag "google".
  12. Thought Good Girl with 4 viewings related by tag "bots".
  13. Thought about: alphabet aka abc.xyz with 3 viewings related by tag "google".
  14. Thought Obviously i'm gonnna haf to get it ! with 3 viewings related by tag "google".
  15. Thought Very Kewl street level maps on Google with 2 viewings related by tag "google".
  16. Thought Testing Google Docs & Spreadsheets with 2 viewings related by tag "google".
  17. Thought about: why wont adsense remove these click fraud sites | Threadwatch.org with 1 viewings related by tag "google".
  18. Thought about: google is letting artificial intelligence run search - bloomberg business with 1 viewings related by tag "google".
  19. Thought about: Search engine's sense of humour crashes as it fires off warning letters over use of name as a verb with 1 viewings related by tag "google".
  20. Thought Googling The Great Work with 1 viewings related by tag "google".
  21. Thought The machines do the translating from Google Blog with 1 viewings related by tag "google".
  22. Thought about: Google Sitemaps (BETA) Help with 0 viewings related by tag "google".
  23. Thought about: Official Google Blog: Rumor of the day with 0 viewings related by tag "google".
  24. Thought Google's new pages editor/publisher with 0 viewings related by tag "google".
  25. Thought about: Google Image Labeler with 0 viewings related by tag "google".
  26. Thought about: Official Google Blog: Setting trends with 0 viewings related by tag "google".
  27. Thought Is google evil ? with 0 viewings related by tag "google".
  28. Thought about: Google Press Center: The Google Podium with 0 viewings related by tag "google".
  29. Thought about: google green blog: project sunroof: mapping the planet’s solar energy potential, one rooftop at a time with 0 viewings related by tag "google".
  30. Thought Hmmm... just when I began to want to switch with 0 viewings related by tag "google".
  31. Thought my Google+ with 0 viewings related by tag "google".
  32. Thought google bashing with 0 viewings related by tag "google".
  33. Thought about: Official Google Blog: An update on payments with 0 viewings related by tag "google".
  34. Thought about: google base products for Speak To Me Catalog with 0 viewings related by tag "google".
  35. Thought Announcement: [google x] as a wiki reference with 0 viewings related by tag "google".
  36. Thought Something New ? with 0 viewings related by tag "google".
  37. Thought Google Web Toolkit with 0 viewings related by tag "google".
  38. Thought I just noticed Google's recommended stories with 0 viewings related by tag "google".
  39. Thought Building stufftalks.com with 0 viewings related by tag "yahoo".
  40. Thought about: Google Help : Add to Google button with 0 viewings related by tag "google".
  41. Thought about: Google Webmaster Tools - Site Status with 0 viewings related by tag "google".
  42. Thought Will the real Web2.0 stand up? with 0 viewings related by tag "google".
  43. Thought about: Better Living Through Software - Who's the Master? with 0 viewings related by tag "google".
  44. Thought Google NEWS - HA! with 0 viewings related by tag "google".
  45. Thought Google is watching with 0 viewings related by tag "google".
  46. Thought about: Google Operating System: Open Gmail's Attachments in Google Docs with 0 viewings related by tag "google".
  47. Thought New Interface with 0 viewings related by tag "google".
  48. Thought Copyrights with 0 viewings related by tag "google".
  49. Thought I dont know how to feel about this yet ... with 0 viewings related by tag "google".
  50. Thought about: Google Click-to-Call with 0 viewings related by tag "google".