Question

我完全糊涂了。

它建立在CodeIgniter平台上。

我在Google Search Console中发现，我的网站只有1页在Google上编入索引。所有其他页面都有404错误，因此谷歌没有列出它们。

因此我认为这是一个错误的站点地图，所以前往https://www.xml-sitemaps.com/生成一个新站点地图。我放入了根网址，令我惊讶的是，只有博客条目包含在生成的xml站点地图中 - 我网站的主要页面没有！!!

因此，我访问了另一个网站，检查链接是否已损坏http://www.brokenlinkcheck.com/，令我惊讶的是，我网站上的每个网页都有404个断开状态的状态。但是，单击这些链接会显示一个有效页面。因此，它们不会断开链接，我可以很好地浏览网站。

因此，我不明白为什么自动机器人带有404列表并且不会将网站编入索引，当所有链接看起来都有效时！???

有什么想法吗？

感谢

更新：我也尝试从搜索控制台进行抓取和渲染，浏览器上显示的有效页面会显示Not Found错误！

更新2：完成网站后：google中的citylightstours.com我注意到索引的唯一网页是博客页面。所有其他页面都退出了索引 - 任何想法为什么???

更新3：其中一条评论表明它可能是.htaccess的一个问题，所以我在这里发帖，希望有人发现一些东西。感谢

更新4：阅读完这篇文章enter link description here后，我认为可能是服务器返回404错误，实际页面代码为客户404人类可读消息!!正如我所说，我使用codeigniter所以它必须与自定义404页面和路由有关。我不知道如何调试这个，甚至不知道要看什么。任何人都可以帮忙吗？...谢谢！

<IfModule mod_rewrite.c>
# Development
    RewriteEngine On
    RewriteBase /
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond $1 !^(index\.php|images|scripts|styles|vendor|robots\.txt)
    RewriteRule ^(.*)$ index.php/$1 [L]
</IfModule>

DirectoryIndex index.php
RewriteEngine on
RewriteCond $1 !^(index\.php|images|css|js|robots\.txt|favicon\.ico)
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ ./index.php/$1 [L,QSA]


# ----------------------------------------------------------------------
# Better website experience for IE users
# ----------------------------------------------------------------------

<IfModule mod_setenvif.c>
  <IfModule mod_headers.c>
    BrowserMatch MSIE ie
    Header set X-UA-Compatible "IE=Edge,chrome=1" env=ie
  </IfModule>
</IfModule>

<IfModule mod_headers.c>
  Header append Vary User-Agent
</IfModule>


# ----------------------------------------------------------------------
# Webfont access
# ----------------------------------------------------------------------

<FilesMatch "\.(ttf|otf|eot|woff|font.css)$">
  <IfModule mod_headers.c>
    Header set Access-Control-Allow-Origin "*"
  </IfModule>
</FilesMatch>

# ----------------------------------------------------------------------
# Proper MIME type for all files
# ----------------------------------------------------------------------

# audio
AddType audio/ogg                      oga ogg

# video
AddType video/ogg                      .ogv
AddType video/mp4                      .mp4
AddType video/webm                     .webm

# Proper svg serving. Required for svg webfonts on iPad
#   twitter.com/FontSquirrel/status/14855840545
AddType     image/svg+xml              svg svgz 
AddEncoding gzip                       svgz

# webfonts                             
AddType application/vnd.ms-fontobject  eot
AddType font/truetype                  ttf
AddType font/opentype                  otf
AddType application/x-font-woff        woff

# assorted types                                      
AddType image/x-icon                   ico
AddType image/webp                     webp
AddType text/cache-manifest            appcache manifest
AddType text/x-component               htc
AddType application/x-chrome-extension crx
AddType application/x-xpinstall        xpi
AddType application/octet-stream       safariextz

# ----------------------------------------------------------------------
# gzip compression
# ----------------------------------------------------------------------

<IfModule mod_deflate.c>

<IfModule mod_setenvif.c>
  <IfModule mod_headers.c>
    SetEnvIfNoCase ^(Accept-EncodXng|X-cept-Encoding|X{15}|~{15}|-{15})$ ^((gzip|deflate)\s,?\s(gzip|deflate)?|X{4,13}|~{4,13}|-{4,13})$ HAVE_Accept-Encoding
    RequestHeader append Accept-Encoding "gzip,deflate" env=HAVE_Accept-Encoding
  </IfModule>
</IfModule>

<FilesMatch "^(?!.*\.ogg$|.*\.ogv$|.*\.mp4$).+" >

# html, txt, css, js, json, xml, htc:
<IfModule filter_module>
  FilterDeclare   COMPRESS
  FilterProvider  COMPRESS  DEFLATE resp=Content-Type /text/(html|css|javascript|plain|x(ml|-component))/
  FilterProvider  COMPRESS  DEFLATE resp=Content-Type /application/(javascript|json|xml|x-javascript)/
  FilterChain     COMPRESS
  FilterProtocol  COMPRESS  change=yes;byteranges=no
</IfModule>
</FilesMatch>

# webfonts and svg:
  <FilesMatch "\.(ttf|otf|eot|svg)$" >
    SetOutputFilter DEFLATE
  </FilesMatch>
</IfModule>

# ----------------------------------------------------------------------
# Expires headers (for better cache control)
# ----------------------------------------------------------------------

<IfModule mod_expires.c>
  ExpiresActive on

# Perhaps better to whitelist expires rules? Perhaps.
  ExpiresDefault                          "access plus 1 month"

# cache.appcache needs re-requests in FF 3.6 (thx Remy ~Introducing HTML5)
  ExpiresByType text/cache-manifest       "access plus 0 seconds"

# your document html 
  ExpiresByType text/html                 "access plus 0 seconds"

# data
  ExpiresByType text/xml                  "access plus 0 seconds"
  ExpiresByType application/xml           "access plus 0 seconds"
  ExpiresByType application/json          "access plus 0 seconds"

# rss feed
  ExpiresByType application/rss+xml       "access plus 1 hour"

# favicon (cannot be renamed)
  ExpiresByType image/x-icon              "access plus 1 week" 

# media: images, video, audio
  ExpiresByType image/gif                 "access plus 1 month"
  ExpiresByType image/png                 "access plus 1 month"
  ExpiresByType image/jpg                 "access plus 1 month"
  ExpiresByType image/jpeg                "access plus 1 month"
  ExpiresByType video/ogg                 "access plus 1 month"
  ExpiresByType audio/ogg                 "access plus 1 month"
  ExpiresByType video/mp4                 "access plus 1 month"
  ExpiresByType video/webm                "access plus 1 month"

# htc files  (css3pie)
  ExpiresByType text/x-component          "access plus 1 month"

# webfonts
  ExpiresByType font/truetype             "access plus 1 month"
  ExpiresByType font/opentype             "access plus 1 month"
  ExpiresByType application/x-font-woff   "access plus 1 month"
  ExpiresByType image/svg+xml             "access plus 1 month"
  ExpiresByType application/vnd.ms-fontobject "access plus 1 month"

# css and javascript
  ExpiresByType text/css                  "access plus 2 months"
  ExpiresByType application/javascript    "access plus 2 months"
  ExpiresByType text/javascript           "access plus 2 months"

  <IfModule mod_headers.c>
    Header append Cache-Control "public"
  </IfModule>

</IfModule>

# ----------------------------------------------------------------------
# ETag removal
# ----------------------------------------------------------------------

FileETag None

# ----------------------------------------------------------------------
# Stop screen flicker in IE on CSS rollovers
# ----------------------------------------------------------------------

# The following directives stop screen flicker in IE on CSS rollovers - in
# combination with the "ExpiresByType" rules for images (see above). If
# needed, un-comment the following rules.

# BrowserMatch "MSIE" brokenvary=1
# BrowserMatch "Mozilla/4.[0-9]{2}" brokenvary=1
# BrowserMatch "Opera" !brokenvary
# SetEnvIf brokenvary 1 force-no-vary

RewriteEngine On
RewriteCond %{HTTP_HOST} !^citylightstours\.com$ [NC]
RewriteRule ^(.*)$ http://citylightstours.com/$1 [R=301,L]
RewriteCond %{HTTP_USER_AGENT} libwww-perl.* 
RewriteRule .* ? [F,L]

Answer 1

已解决 - 网站中集成的wordpress博客正在为所有非wordpress页面设置404状态，即codeigniter页面

CI的index.php有以下代码需要注释掉

/*
 *---------------------------------------------------------------
 * WORDPRESS INTEGRATION
 *---------------------------------------------------------------
 * The ci_site_url function helps to avoid collision between WP & CI.
 */

 //header("HTTP/1.0 200 OK");

 define('WP_USE_THEMES', false);
 require_once './blog/wp-blog-header.php';

 add_filter('site_url', 'ci_site_url', 1);

    function ci_site_url()
    {
  include(APPPATH.'/config/config.php');
  return $config['base_url'];
    }

CodeIgniter 404路由 - 404断开的链接和谷歌无法爬网站点但一切看起来都不错

1 个答案: