抓取网址为302重定向，但实际上没有重定向为什么会发生这种情况？

时间：2015-05-27 11:04:19

标签： php .htaccess redirect web-crawler magento-1.7 

.htaccess
    ##### Add support for SVG Images and CSS3 Pie #####

    AddType image/svg+xml svg svgz
    AddEncoding gzip svgz
    AddType text/x-component .htc
    DirectoryIndex index.php 

##### PHP Settings for your domain #####

<IfModule mod_php5.c> 
    php_value memory_limit 512M 
    php_value max_execution_time 18000 
    php_flag magic_quotes_gpc off 
    php_flag session.auto_start off 
    php_flag suhosin.session.cryptua off 
    php_flag zend.ze1_compatibility_mode Off 
</IfModule>

##### Search Engine redirects and rewrites for SEO purposes #####

<IfModule mod_rewrite.c>
    #RewriteCond %{HTTP_HOST} !^www.alshop.com$ [NC]
    #RewriteRule ^(.*)$ http://www.alshop.com/$1 [R=301,L]

    ##### Redirect away from /index.php and /home   
    ##### Warning: This index.php rewrite will prevent Magento 
    ##### Connect from working. Simply comment out the  
    ##### following two lines of code when using Connect.
    ##### Please note - http://www. if not using www simply use http://

    RewriteCond %{THE_REQUEST} ^.*/index.php
    RewriteRule ^(.*)index.php$ http://www.alshop.com/$1 [R=301,L]

    ##### Please note - http://www. if not using www simply use http://
    redirect 301 /home http://www.alshop.com

    Options +FollowSymLinks
    RewriteEngine on
    RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
    RewriteCond %{REQUEST_URI} !^/(media|skin|js)/
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !-l
    RewriteRule .* index.php [L]

    RewriteRule ^(.*)$ $1 [NS,E=no-gzip:1,E=dont-vary:1]
</IfModule>


##### mod_deflate compresses your output to lower the file size being sent to the client #####

<IfModule mod_deflate.c>

    php_flag zlib.output_compression off
    SetEnvIfNoCase Request_URI \.(html?|txt|css|js|php|pl)$$ no-gzip dont-vary
</IfModule>


<IfModule mod_ssl.c>
    SSLOptions StdEnvVars 
</IfModule>


##### Header Directives #####

<ifModule mod_headers.c>
    Header unset ETag
    Header unset Last-Modified
</ifModule>



 ##### Disable ETags http://developer.yahoo.com/performance/rules.html#etags #####

    FileETag None


##### Prevent character encoding issues from server overrides #####

    AddDefaultCharset Off
    #AddDefaultCharset UTF-8


##### By default allow all access #####

    #Order allow,deny
    #Allow from all





## robots.txt

User-agent: *


## Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
## Set a custom crawl rate if you're experiencing traffic problems with your server.
# Crawl-delay: 30

## DEVELOPMENT RELATED SETTINGS`

## Do not crawl development files and folders: CVS, svn directories and dump files
Disallow: /CVS
Disallow: /*.svn$
Disallow: /*.idea$
Disallow: /*.sql$
Disallow: /*.tgz$

## GENERAL MAGENTO SETTINGS

## Do not crawl Magento admin page
Disallow: /admin/

## Do not crawl common Magento technical folders
Disallow: /app/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /shell/
Disallow: /var/

## Do not crawl common Magento files
Disallow: /api.php
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /get.php
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /README.txt
Disallow: /RELEASE_NOTES.txt

## MAGENTO SEO IMPROVEMENTS

## Do not crawl sub category pages that are sorted or filtered.
Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /*?mode*
## Disallow: /*?*

## Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
Disallow: /index.php/

## Do not crawl links with session IDs
Disallow: /*?SID=

## Do not crawl checkout and user account pages
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/

## Do not crawl seach pages and not-SEO optimized catalog links
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/

## SERVER SETTINGS

## Do not crawl common server technical folders and files
Disallow: /cgi-bin/
Disallow: /cleanup.php
Disallow: /apc.php
Disallow: /memcache.php
Disallow: /phpinfo.php

## IMAGE CRAWLERS SETTINGS

## Extra: Uncomment if you do not wish Google and Bing to index your images
# User-agent: Googlebot-Image
# Disallow: /
# User-agent: msnbot-media
# Disallow: /

On Fri, May 29, 2015 at 9:51 AM, tech alshop <tech.alshop@gmail.com> wrote:
   .htaccess
    ##### Add support for SVG Images and CSS3 Pie #####

    AddType image/svg+xml svg svgz
    AddEncoding gzip svgz
    AddType text/x-component .htc
    DirectoryIndex index.php 

##### PHP Settings for your domain #####

<IfModule mod_php5.c> 
    php_value memory_limit 512M 
    php_value max_execution_time 18000 
    php_flag magic_quotes_gpc off 
    php_flag session.auto_start off 
    php_flag suhosin.session.cryptua off 
    php_flag zend.ze1_compatibility_mode Off 
</IfModule>

##### Search Engine redirects and rewrites for SEO purposes #####

<IfModule mod_rewrite.c>
    #RewriteCond %{HTTP_HOST} !^www.alshop.com$ [NC]
    #RewriteRule ^(.*)$ http://www.alshop.com/$1 [R=301,L]

    ##### Redirect away from /index.php and /home   
    ##### Warning: This index.php rewrite will prevent Magento 
    ##### Connect from working. Simply comment out the  
    ##### following two lines of code when using Connect.
    ##### Please note - http://www. if not using www simply use http://

    RewriteCond %{THE_REQUEST} ^.*/index.php
    RewriteRule ^(.*)index.php$ http://www.alshop.com/$1 [R=301,L]

    ##### Please note - http://www. if not using www simply use http://
    redirect 301 /home http://www.alshop.com

    Options +FollowSymLinks
    RewriteEngine on
    RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
    RewriteCond %{REQUEST_URI} !^/(media|skin|js)/
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !-l
    RewriteRule .* index.php [L]

    RewriteRule ^(.*)$ $1 [NS,E=no-gzip:1,E=dont-vary:1]
</IfModule>


##### mod_deflate compresses your output to lower the file size being sent to the client #####

<IfModule mod_deflate.c>

    php_flag zlib.output_compression off
    SetEnvIfNoCase Request_URI \.(html?|txt|css|js|php|pl)$$ no-gzip dont-vary
</IfModule>


<IfModule mod_ssl.c>
    SSLOptions StdEnvVars 
</IfModule>


##### Header Directives #####

<ifModule mod_headers.c>
    Header unset ETag
    Header unset Last-Modified
</ifModule>



 ##### Disable ETags http://developer.yahoo.com/performance/rules.html#etags #####

    FileETag None


##### Prevent character encoding issues from server overrides #####

    AddDefaultCharset Off
    #AddDefaultCharset UTF-8


##### By default allow all access #####

    #Order allow,deny
    #Allow from all

我正在使用magento后端的电子商务网站工作。过去一切都很好，但突然间不知道从哪里开始。我们正在创建的一些旧网址和所有新网址都没有编入索引，因为抓取工具将链接视为302重定向，但实际上在浏览器中，如果您看到它是200 ok。当我们使用seo工具时，所有与seo相关的东西都很好。当我们仔细检查时，我们发现这些302状态链接中的一些正在达到200 ok，但在一段时间后它再次回到302。有人可以帮助我解决这个问题。因此，我们新创建的页面都没有被编入索引。

0 个答案:

没有答案
相关问题
最新问题