DragonPrime - LoGD Resource Community
Welcome Guest
  • Good evening, Guest.
    Please log in, or register.
  • September 25, 2016, 12:07:25 AM
Home Forums News Links Downloads Login Register Advanced Search
* * *
DragonPrime Menu
Login
 
 
Resource Pages
IRC Channels
Search

Pages: [1]   Go Down
  Print  
Author Topic: Keep an eye on gooooooogle  (Read 3100 times)
0 Members and 1 Guest are viewing this topic.
eph
Guest
« on: August 30, 2005, 12:46:17 AM »

Just thought I'd tell you, as it made me nibble on my desk this month.
All of a sudden google caused enormously high traffic on my server, 300MB a day where it used to be 30 a month. A look in the logfile nearly made me drop from my chair.

They indexed the news.php of my LoGD installation 22,000 times in a 4-day period, and the bandwidth had summed up tp 4,5GB google-traffic in August before I noticed. (And I only have a dozen active players, so the news don't change that often.)
Luckily I have a very high bandwidth allowance, but those who run on a shared server or have to pay for bandwidth anyway should definitely give it a thought to check for google in their logs if they notice a sudden increase in traffic.

This just started this month, and news.php is the only of the ~10,000 files on my server that gets that insane amount of hits, so I guess they might have changed their bots' behaviour concerning "news". Hopefully they will stop that again some time soon, I'd like to unblock them from the site again.
Logged
XChrisX
Global Moderator
Mod God
*****
Offline Offline

Posts: 4647

Be aware of the squirrel!


View Profile WWW
« Reply #1 on: August 30, 2005, 01:05:32 AM »

Create a file: "robots.txt"

Content:
Code:
User-agent: *
Disallow: /

This will disallow all search engines to index your site.

you can add thing like:
Code:
Allow: home.php
Allow: index.php
To allow those engines to at least index you login screen... Maybe even about.php depends on you...
But be aware: The Allow-Command is not understood by all engines... But at least that'll keep traffic low...
Logged

Running for more than three years now:
eph
Guest
« Reply #2 on: August 30, 2005, 02:38:31 AM »

I did, but others may not have. Therefore the warning.
Logged
Dannic
Guest
« Reply #3 on: August 30, 2005, 03:57:47 AM »

I think they are just reindexing.  It wasn't only news pages they were hitting pretty hard.
Logged
Turock
Guest
« Reply #4 on: August 31, 2005, 07:47:18 PM »

Yep, I just noticed this last night as I was going through the server logs.  Not only did we get hit for the news, but for the player list as well.  Doesn't make much difference to us though as we have 1000Gb a month to play with.
Logged
Pages: [1]   Go Up
  Print  
 
Jump to:  


*
DragonPrime Notices
Please take the time to read the FAQ and browse the DragonPedia

Support Us
No funds raised yet this year
Your help is greatly appreciated!
Who's Online
45 Guests, 0 Users
DragonPrime LoGD
Recent Topics
Home Forums News Links Downloads Login Register Advanced Search