|
¸®´ª½º Tech °Ô½ÃÆÇ
|
2024. 05. 05. |
|
|
|
|
|
|
|
[Tip] °Ë»ö¿£ÁøÀÌ ±Ü¾î°¡´Â °Í ¸·±â |
|
|
|
|
|
³¯Â¥: 2006.05.22. 21:37:18 Á¶È¸: 332 |
|
|
|
|
|
|
|
|
|
|
|
|
¨ç ¾Æ·¡¿Í °°ÀÌ ÇÏ¿© robots.txt À̸§À¸·Î ÀúÀåÇÕ´Ï´Ù.
User-agent: *
Disallow: /
À§ ³»¿ëÀ» ¼³¸íÇÏÀÚ¸é ¸ðµç °Ë»ö¿£ÁøÀÌ ±Ü¾î°¡´Â °ÍÀ» ¸·´Â °Ì´Ï´Ù.
User-agent: * =>*Àº ¸ðµç °Ë»ö¿£ÁøÀ» ¶æÇÔ.
Disallow: / => /´Â ¸ðµç µð·ºÅ丮
[robots.txt »ó¼¼ ¿É¼Ç ¼³¸í]
¸ðµç °Ë»ö¿£ÁøÀÌ ±Ü¾î°¡´Â °Í ¸ðµÎ ¸·±â
User-agent: *
Disallow: /
¸ðµÎ Çã¿ëÇϱâ
User-agent: *
Disallow:
cgi-bin µð·ºÅ丮, tmp µð·ºÅ丮, private µð·ºÅ丮 ±Ü¾î°¡´Â °Í¸¸ ¸·±â
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/
¹èµåº¿ °Ë»ö·Îº¿¸¸ ±Ü¾î°¡±â Á¦¿Ü
User-agent: BadBot
Disallow: /
À¥Å©·Ñ·¯ °Ë»ö·Îº¿¸¸ ±Ü¾î°¡±â Çã¶ô
User-agent: WebCrawler
Disallow:
¨è robots.txt ÆÄÀÏÀ» ftp·Î ¿Ã¸³´Ï´Ù. ÃÖ»óÀ§ µð·ºÅ丮¿¡ ¿Ã·Á¾ßÇÕ´Ï´Ù.
¿¹) http://www.tagin.net/robots.txt ( O )
http://www.tagin.net/test/robots.txt ( X )
* robots.txt À¯ÀÇ»çÇ×
1. »çÀÌÆ®´Â ÇϳªÀÇ "robots.txt" ¸¸À» °¡Áú ¼ö ÀÖ´Ù
2. "robots.txt" ¹®ÀÚ¿Àº ¸ðµÎ ¼Ò¹®ÀÚÀ̾î¾ß Çϸç, °ø¹éÀº Çã¿ë µÇÁö ¾Ê´Â´Ù.
3. ÃÖ»óÀ§ µð·ºÅ丮ÀÇ robots.txt¸¸ Àд´Ù. (´Ù¸¥ µð·ºÅ丮ÀÇ robots.txt´Â ¾Æ¹«¼Ò¿ëÀ̾ø´Ù)
ÀÌ ¹æ¹ý ¿Ü¿¡µµ html ÆäÀÌÁö¿¡ ¸ÞŸ ű׸¦ ½á¼ °Ë»ö¿£ÁøÀ» ¸·À» ¼ö ÀÖ½À´Ï´Ù.
À§ ¼Ò½º¸¦ ¿Í »çÀÌ¿¡ ³Ö¾îÁÖ¸é µË´Ï´Ù.
À¯ÀÇÇÒ Á¡Àº À§¿Í °°Àº ¹æ¹ýÀÌ ¸ðµç °Ë»ö¿£ÁøÀ» ¿ÏÀü Â÷´ÜÇÏÁø ¸øÇÑ´Ù´Â °É ¿°µÎÇصμ¼¿ä.
ÀÚ¼¼ÇÑ »çÇ×Àº http://www.robotstxt.org/ Âü°íÇÏ½Ã±æ ¹Ù¶ø´Ï´Ù. |
|
|
|
|