Solr 5的配置(Umlaute,特殊字符和字符串长度)

时间:2016-06-08 14:50:03

标签: ruby-on-rails solr full-text-search

过去几个月在德国网站上运行Solr 5,处理Umlaute似乎是一场噩梦。我不是Solr的专家,也不是我运行弹性的所有其他项目。通过Solr文档找到自己的方式是一场艰苦的战斗。

我想知道是否可以通过schema.xml轻松配置以下两件事:

1。)UMLAUTE和特殊字符

特殊字符以HTML代码存储在数据库中。例如: “einerAußenwand.EineBrandschutztürsorgtfürmaximaleSicherheit。”

现在Solr并不知道如何处理它。因此,如果用户搜索“für”,则不会出现任何问题。我也试图搜索“für”和“fr” - 没有任何东西返回预期的结果。

如果我输入“Regelungs-App”,同样没有任何结果 - 如果我输入“Regelungs App”,我会受到点击。为什么一个简单的破折号会抛出它的轨道?什么设置,或者我可以做些什么来忽略这个?

2.。)搜索字符串的长度

如果我在索引内容中搜索字符串,则可能会限制为一定数量的字符 - 例如:

“Erreicht als einziger Staubemissionen” - 没有结果 “als einziger Staubemissionen” - 没有结果 “einziger Staubemissionen” - 正确的结果 “Staubemissionen” - 正确的结果

我该如何设置?

我当前的schema.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!--
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-->
<!--
 This is the Solr schema file. This file should be named "schema.xml" and
 should be in the conf directory under the solr home
 (i.e. ./solr/conf/schema.xml by default)
 or located where the classloader for the Solr webapp can find it.

 This example schema is the recommended starting point for users.
 It should be kept correct and concise, usable out-of-the-box.

 For more information, on how to customize this file, please see
 http://wiki.apache.org/solr/SchemaXml

 PERFORMANCE NOTE: this schema includes many optional features and should not
 be used for benchmarking.  To improve performance one could
  - set stored="false" for all fields possible (esp large fields) when you
    only need to search on the field but don't need to return the original
    value.
  - set indexed="false" if you don't need to search on the field, but only
    return the field as a result of searching on other indexed fields.
  - remove all unneeded copyField statements
  - for best index size and searching performance, set "index" to false
    for all general text fields, use copyField to copy them to the
    catchall "text" field, and use that for searching.
  - For maximum indexing performance, use the StreamingUpdateSolrServer
    java client.
  - Remember to run the JVM in server mode, and use a higher logging level
    that avoids logging every request
-->
<schema name="sunspot" version="1.0">
  <types>
    <!-- field type definitions. The "name" attribute is
       just a label to be used by field definitions.  The "class"
       attribute and any other attributes determine the real
       behavior of the fieldType.
         Class names starting with "solr" refer to java classes in the
       org.apache.solr.analysis package.
    -->
    <!-- *** This fieldType is used by Sunspot! *** -->
    <fieldType name="string" class="solr.StrField" omitNorms="true"/>
    <!-- *** This fieldType is used by Sunspot! *** -->
    <fieldType name="tdouble" class="solr.TrieDoubleField" omitNorms="true"/>
    <!-- *** This fieldType is used by Sunspot! *** -->
    <fieldType name="rand" class="solr.RandomSortField" omitNorms="true"/>
    <!-- *** This fieldType is used by Sunspot! *** -->
    <fieldType name="text" class="solr.TextField" omitNorms="false">
      <analyzer type="index">
        <tokenizer class="solr.NGramTokenizerFactory" minGramSize="3" maxGramSize="15" />
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.StandardFilterFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.PorterStemFilterFactory"/>
        <filter class="solr.WordDelimiterFilterFactory" stemEnglishPossessive="1" splitOnNumerics="1" splitOnCaseChange="1" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="1" preserveOriginal="1"/>
      </analyzer>
    </fieldType>
    <!-- *** This fieldType is used by Sunspot! *** -->
    <fieldType name="boolean" class="solr.BoolField" omitNorms="true"/>
    <!-- *** This fieldType is used by Sunspot! *** -->
    <fieldType name="tint" class="solr.TrieIntField" omitNorms="true"/>
    <!-- *** This fieldType is used by Sunspot! *** -->
    <fieldType name="tlong" class="solr.TrieLongField" omitNorms="true"/>
    <!-- *** This fieldType is used by Sunspot! *** -->
    <fieldType name="tfloat" class="solr.TrieFloatField" omitNorms="true"/>
    <!-- *** This fieldType is used by Sunspot! *** -->
    <fieldType name="tdate" class="solr.TrieDateField"
               omitNorms="true"/>

    <fieldType name="daterange" class="solr.DateRangeField" omitNorms="true" />

    <!-- Special field type for spell correction. Be careful about
         adding filters here, as they apply *before* your values go in
         the spellcheck. For example, the lowercase filter here means
         all spelling suggestions will be lower case (without it,
         though, you'd have duplicate suggestions for lower and proper
         cased words). -->
    <fieldType name="textSpell" class="solr.TextField" positionIncrementGap="100" omitNorms="true">
      <analyzer>
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.StandardFilterFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
    </fieldType>
    <!-- A specialized field for geospatial search. If indexed, this fieldType must not be multivalued. -->
    <fieldType name="location" class="solr.LatLonType" subFieldSuffix="_coordinate"/>
  </types>
  <fields>
    <!-- Valid attributes for fields:
     name: mandatory - the name for the field
     type: mandatory - the name of a previously defined type from the
       <types> section
     indexed: true if this field should be indexed (searchable or sortable)
     stored: true if this field should be retrievable
     compressed: [false] if this field should be stored using gzip compression
       (this will only apply if the field type is compressable; among
       the standard field types, only TextField and StrField are)
     multiValued: true if this field may contain multiple values per document
     omitNorms: (expert) set to true to omit the norms associated with
       this field (this disables length normalization and index-time
       boosting for the field, and saves some memory).  Only full-text
       fields or fields that need an index-time boost need norms.
     termVectors: [false] set to true to store the term vector for a
       given field.
       When using MoreLikeThis, fields used for similarity should be
       stored for best performance.
     termPositions: Store position information with the term vector.
       This will increase storage costs.
     termOffsets: Store offset information with the term vector. This
       will increase storage costs.
     default: a value that should be used if no value is specified
       when adding a document.
   -->
    <!-- *** This field is used by Sunspot! *** -->
    <field name="id" stored="true" type="string" multiValued="false" indexed="true"/>
    <!-- *** This field is used by Sunspot! *** -->
    <field name="type" stored="false" type="string" multiValued="true" indexed="true"/>
    <!-- *** This field is used by Sunspot! *** -->
    <field name="class_name" stored="false" type="string" multiValued="false" indexed="true"/>
    <!-- *** This field is used by Sunspot! *** -->
    <field name="text" stored="false" type="string" multiValued="true" indexed="true"/>
    <!-- *** This field is used by Sunspot! *** -->
    <field name="lat" stored="true" type="tdouble" multiValued="false" indexed="true"/>
    <!-- *** This field is used by Sunspot! *** -->
    <field name="lng" stored="true" type="tdouble" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="random_*" stored="false" type="rand" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="_local*" stored="false" type="tdouble" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_text" stored="false" type="text" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_texts" stored="true" type="text" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_b" stored="false" type="boolean" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_bm" stored="false" type="boolean" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_bs" stored="true" type="boolean" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_bms" stored="true" type="boolean" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_d" stored="false" type="tdate" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_dm" stored="false" type="tdate" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_ds" stored="true" type="tdate" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_dms" stored="true" type="tdate" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_e" stored="false" type="tdouble" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_em" stored="false" type="tdouble" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_es" stored="true" type="tdouble" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_ems" stored="true" type="tdouble" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_f" stored="false" type="tfloat" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_fm" stored="false" type="tfloat" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_fs" stored="true" type="tfloat" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_fms" stored="true" type="tfloat" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_i" stored="false" type="tint" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_im" stored="false" type="tint" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_is" stored="true" type="tint" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_ims" stored="true" type="tint" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_l" stored="false" type="tlong" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_lm" stored="false" type="tlong" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_ls" stored="true" type="tlong" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_lms" stored="true" type="tlong" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_s" stored="false" type="string" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_sm" stored="false" type="string" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_ss" stored="true" type="string" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_sms" stored="true" type="string" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_it" stored="false" type="tint" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_itm" stored="false" type="tint" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_its" stored="true" type="tint" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_itms" stored="true" type="tint" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_ft" stored="false" type="tfloat" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_ftm" stored="false" type="tfloat" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_fts" stored="true" type="tfloat" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_ftms" stored="true" type="tfloat" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_dt" stored="false" type="tdate" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_dtm" stored="false" type="tdate" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_dts" stored="true" type="tdate" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_dtms" stored="true" type="tdate" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_textv" stored="false" termVectors="true" type="text" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_textsv" stored="true" termVectors="true" type="text" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_et" stored="false" termVectors="true" type="tdouble" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_etm" stored="false" termVectors="true" type="tdouble" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_ets" stored="true" termVectors="true" type="tdouble" multiValued="false" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_etms" stored="true" termVectors="true" type="tdouble" multiValued="true" indexed="true"/>
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_dr" stored="false" type="daterange" multiValued="false" indexed="true" />
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_drm" stored="false" type="daterange" multiValued="true" indexed="true" />
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_drs" stored="true" type="daterange" multiValued="false" indexed="true" />
    <!-- *** This dynamicField is used by Sunspot! *** -->
    <dynamicField name="*_drms" stored="true" type="daterange" multiValued="true" indexed="true" />

    <!-- Type used to index the lat and lon components for the "location" FieldType -->
    <dynamicField name="*_coordinate"  type="tdouble" indexed="true"  stored="false" multiValued="false"/>
    <dynamicField name="*_p" type="location" indexed="true" stored="true" multiValued="false"/>

    <dynamicField name="*_ll" stored="false" type="location" multiValued="false" indexed="true"/>
    <dynamicField name="*_llm" stored="false" type="location" multiValued="true" indexed="true"/>
    <dynamicField name="*_lls" stored="true" type="location" multiValued="false" indexed="true"/>
    <dynamicField name="*_llms" stored="true" type="location" multiValued="true" indexed="true"/>
    <field name="textSpell" stored="false" type="textSpell" multiValued="true" indexed="true"/>

    <!-- required by Solr 4 -->
    <field name="_version_" type="string" indexed="true" stored="true" multiValued="false" />
  </fields>

  <!-- Field to use to determine and enforce document uniqueness.
      Unless this field is marked with required="false", it will be a required field
   -->
  <uniqueKey>id</uniqueKey>
  <!-- field for the QueryParser to use when an explicit fieldname is absent -->
  <defaultSearchField>text</defaultSearchField>
  <!-- SolrQueryParser configuration: defaultOperator="AND|OR" -->
  <solrQueryParser defaultOperator="AND"/>
  <!-- copyField commands copy one field to another at the time a document
        is added to the index.  It's used either to index the same field differently,
        or to add multiple fields to the same field for easier/faster
        searching.  -->

  <!-- Use copyField to copy the fields you want to run spell checking
       on into one field. For example: -->
  <copyField source="*_text"  dest="textSpell" />
  <copyField source="*_s"  dest="textSpell" />
</schema>

2 个答案:

答案 0 :(得分:1)

您没有对您正在搜索的字段类型进行任何说明,但如果它的类型为“text”,则分析链看起来不适合您要执行的操作。输入(NGramTokenizer)和只是小写,不会给出你在查询端与StandardTokenizer一起期待的结果。

创建一个具有更简化定义的新字段(现在对于索引和查询可能都是相同的),它只包含一个空格标记器或另一个更标准的标记化器 - 请参阅差异的the reference manual for examples。你可能也想要一个小写的过滤器。

您可能会遇到变音符号和其他特定德语术语的问题,但ICU * - 过滤器和标记符的范围比其他更加国际化。还有一个过滤器可以将单词分成它们的组件(因为你和我们的挪威语有同样的问题,那里的单词是一起写的,而不是英语的分割方式)。

Solr Admin下的“分析”页面是一个开始调试的好地方 - 它会准确显示在索引和查询方面进行的转换,让您可以看到条款不匹配的原因和每个步骤的条款是什么样的。

答案 1 :(得分:0)

对于拉丁语重音字符,请使用

<filter class="solr.ASCIIFoldingFilterFactory"/>

这将以ASCII格式存储所有单词并删除重音符。

您的文本类型在查询方面看起来非常不平衡。尝试这样的事情,然后重新索引你的数据(我目前正在法国网站上工作)。我不认为PorterStemFilterFactor对德语有好处,还有其他的词干分析器更好用: -

<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.ASCIIFoldingFilterFactory"/>
        <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="40" />
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.ASCIIFoldingFilterFactory"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>   
      </analyzer>
    </fieldType>