DragonPrime - LoGD Resource Community
Welcome Guest
  • Good morning, Guest.
    Please log in, or register.
  • October 22, 2016, 06:33:12 AM
Home Forums News Links Downloads Login Register Advanced Search
* * *
DragonPrime Menu
Resource Pages
IRC Channels

Pages: [1]   Go Down
Author Topic: Keep an eye on gooooooogle  (Read 3127 times)
0 Members and 1 Guest are viewing this topic.
« on: August 30, 2005, 12:46:17 AM »

Just thought I'd tell you, as it made me nibble on my desk this month.
All of a sudden google caused enormously high traffic on my server, 300MB a day where it used to be 30 a month. A look in the logfile nearly made me drop from my chair.

They indexed the news.php of my LoGD installation 22,000 times in a 4-day period, and the bandwidth had summed up tp 4,5GB google-traffic in August before I noticed. (And I only have a dozen active players, so the news don't change that often.)
Luckily I have a very high bandwidth allowance, but those who run on a shared server or have to pay for bandwidth anyway should definitely give it a thought to check for google in their logs if they notice a sudden increase in traffic.

This just started this month, and news.php is the only of the ~10,000 files on my server that gets that insane amount of hits, so I guess they might have changed their bots' behaviour concerning "news". Hopefully they will stop that again some time soon, I'd like to unblock them from the site again.
Global Moderator
Mod God
Offline Offline

Posts: 4647

Be aware of the squirrel!

View Profile WWW
« Reply #1 on: August 30, 2005, 01:05:32 AM »

Create a file: "robots.txt"

User-agent: *
Disallow: /

This will disallow all search engines to index your site.

you can add thing like:
Allow: home.php
Allow: index.php
To allow those engines to at least index you login screen... Maybe even about.php depends on you...
But be aware: The Allow-Command is not understood by all engines... But at least that'll keep traffic low...

Running for more than three years now:
« Reply #2 on: August 30, 2005, 02:38:31 AM »

I did, but others may not have. Therefore the warning.
« Reply #3 on: August 30, 2005, 03:57:47 AM »

I think they are just reindexing.  It wasn't only news pages they were hitting pretty hard.
« Reply #4 on: August 31, 2005, 07:47:18 PM »

Yep, I just noticed this last night as I was going through the server logs.  Not only did we get hit for the news, but for the player list as well.  Doesn't make much difference to us though as we have 1000Gb a month to play with.
Pages: [1]   Go Up
Jump to:  

DragonPrime Notices
Welcome to DragonPrime - The LoGD Resource Community!

Support Us
No funds raised yet this year
Your help is greatly appreciated!
Who's Online
46 Guests, 0 Users
DragonPrime LoGD
Recent Topics
Home Forums News Links Downloads Login Register Advanced Search