.htaccess用于域和子域机器人以防止抓取

时间:2018-05-02 20:21:50

标签: .htaccess robots.txt

domain.com - 实时服务器

dev.domain.com - 开发服务器

两者都有:

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTPS} off [OR]
    RewriteCond %{HTTP_HOST} ^www\. [NC]
    RewriteCond %{HTTP_HOST} ^(?:www\.)?(.+)$ [NC]
    RewriteRule ^ https://%1%{REQUEST_URI} [L,NE,R=301]
    RewriteRule ^(.*)$ public/$1 [L]
    RewriteRule ^robots\.txt$ /robots-%1\.txt [L] #this one
</IfModule>

如何使用相同的.htaccess选择特定的机器人,具体取决于实时开发

我试过这个,但它没有用(htaccess的一部分是将非https&amp; www重定向到非www的https,请忽略它):

# GLOBAL RULES
# ============
# Directories Index protection
Options -Indexes

# Remove index.php from URL
<IfModule mod_rewrite.c>
    RewriteCond %{THE_REQUEST} ^GET.*index\.php [NC]
    RewriteRule (.*?)index\.php/*(.*) /$1$2 [R=301,NE,L]
</IfModule>

# Default Apache rules
<IfModule mod_rewrite.c>
    <IfModule mod_negotiation.c>
        Options -MultiViews
    </IfModule>
    RewriteEngine On

    # Redirect Trailing Slashes If Not A Folder...
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule ^(.*)/$ /$1 [L,R=301]

    # Handle Front Controller...
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule ^ index.php [L]

    # Handle Authorization Header
    RewriteCond %{HTTP:Authorization} .
    RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
</IfModule>

# Robots.txt
RewriteRule  ^(robots\.txt)$ /robots [NC,L]

# Iframe calling
<IfModule mod_headers.c>
    Header unset X-Frame-Options
</IfModule>

# GTMetrix - Enable gzip compression
<IfModule mod_deflate.c>
    # Compress HTML, CSS, JavaScript, Text, XML and fonts
    AddOutputFilterByType DEFLATE application/javascript
    AddOutputFilterByType DEFLATE application/rss+xml
    AddOutputFilterByType DEFLATE application/vnd.ms-fontobject
    AddOutputFilterByType DEFLATE application/x-font
    AddOutputFilterByType DEFLATE application/x-font-opentype
    AddOutputFilterByType DEFLATE application/x-font-otf
    AddOutputFilterByType DEFLATE application/x-font-truetype
    AddOutputFilterByType DEFLATE application/x-font-ttf
    AddOutputFilterByType DEFLATE application/x-javascript
    AddOutputFilterByType DEFLATE application/xhtml+xml
    AddOutputFilterByType DEFLATE application/xml
    AddOutputFilterByType DEFLATE application/json
    AddOutputFilterByType DEFLATE font/opentype
    AddOutputFilterByType DEFLATE font/otf
    AddOutputFilterByType DEFLATE font/ttf
    AddOutputFilterByType DEFLATE image/svg+xml
    AddOutputFilterByType DEFLATE image/x-icon
    AddOutputFilterByType DEFLATE text/css
    AddOutputFilterByType DEFLATE text/html
    AddOutputFilterByType DEFLATE text/javascript
    AddOutputFilterByType DEFLATE text/plain
    AddOutputFilterByType DEFLATE text/xml

    # Remove browser bugs (only needed for really old browsers)
    BrowserMatch ^Mozilla/4 gzip-only-text/html
    BrowserMatch ^Mozilla/4\.0[678] no-gzip
    BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
    Header append Vary User-Agent
</IfModule>
# GTMetrix - Enable gzip compression

# GTMetrix Browser Caching
<IfModule mod_expires.c>
    ExpiresActive On
    ExpiresByType image/jpg "access plus 1 week"
    ExpiresByType image/jpeg "access plus 1 week"
    ExpiresByType image/gif "access plus 1 week"
    ExpiresByType image/png "access plus 1 week"
    ExpiresByType image/svg "access plus 1 month"
    ExpiresByType text/css "access plus 1 week"
    ExpiresByType text/javascript "access plus 1 week"
    ExpiresByType application/pdf "access plus 1 month"
    ExpiresByType application/json "access plus 1 week"
    ExpiresByType application/javascript "access plus 1 week"
    ExpiresByType text/x-javascript "access plus 1 week"
    ExpiresByType application/x-shockwave-flash "access plus 1 month"
    ExpiresByType image/x-icon "access plus 1 year"
    ExpiresDefault "access plus 1 month"
</IfModule>
# GTMetrix Browser Caching

更新:以下是公开/ .htaccess

private void createNotificationChannel() {
// Create the NotificationChannel, but only on API 26+ because
// the NotificationChannel class is new and not in the support library
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
    CharSequence name = getString(R.string.channel_name);
    String description = getString(R.string.channel_description);
    int importance = NotificationManager.IMPORTANCE_DEFAULT;
    NotificationChannel channel = new NotificationChannel(CHANNEL_ID, name, importance);
    channel.setDescription(description);
    // Register the channel with the system; you can't change the importance
    // or other notification behaviors after this
    NotificationManager notificationManager = getSystemService(NotificationManager.class);
    notificationManager.createNotificationChannel(channel);
}

1 个答案:

答案 0 :(得分:0)

您可以使用以下规则:

RewriteEngine On

RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} ^www\. [NC]
RewriteCond %{HTTP_HOST} ^(?:www\.)?(.+)$ [NC]
RewriteRule ^ https://%1%{REQUEST_URI} [L,NE,R=301]

RewriteCond %{HTTP_HOST} ^dev\. [NC]
RewriteRule ^robots\.txt$ public/robots-dev.txt [L,NC]

RewriteRule ^(?!public/)(.*)$ public/$1 [L,NC]