过去几个月在德国网站上运行Solr 5,处理Umlaute似乎是一场噩梦。我不是Solr的专家,也不是我运行弹性的所有其他项目。通过Solr文档找到自己的方式是一场艰苦的战斗。
我想知道是否可以通过schema.xml轻松配置以下两件事:
1。)UMLAUTE和特殊字符
特殊字符以HTML代码存储在数据库中。例如: “einerAußenwand.EineBrandschutztürsorgtfürmaximaleSicherheit。”
现在Solr并不知道如何处理它。因此,如果用户搜索“für”,则不会出现任何问题。我也试图搜索“für”和“fr” - 没有任何东西返回预期的结果。
如果我输入“Regelungs-App”,同样没有任何结果 - 如果我输入“Regelungs App”,我会受到点击。为什么一个简单的破折号会抛出它的轨道?什么设置,或者我可以做些什么来忽略这个?
2.。)搜索字符串的长度
如果我在索引内容中搜索字符串,则可能会限制为一定数量的字符 - 例如:
“Erreicht als einziger Staubemissionen” - 没有结果 “als einziger Staubemissionen” - 没有结果 “einziger Staubemissionen” - 正确的结果 “Staubemissionen” - 正确的结果
我该如何设置?
我当前的schema.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!--
This is the Solr schema file. This file should be named "schema.xml" and
should be in the conf directory under the solr home
(i.e. ./solr/conf/schema.xml by default)
or located where the classloader for the Solr webapp can find it.
This example schema is the recommended starting point for users.
It should be kept correct and concise, usable out-of-the-box.
For more information, on how to customize this file, please see
http://wiki.apache.org/solr/SchemaXml
PERFORMANCE NOTE: this schema includes many optional features and should not
be used for benchmarking. To improve performance one could
- set stored="false" for all fields possible (esp large fields) when you
only need to search on the field but don't need to return the original
value.
- set indexed="false" if you don't need to search on the field, but only
return the field as a result of searching on other indexed fields.
- remove all unneeded copyField statements
- for best index size and searching performance, set "index" to false
for all general text fields, use copyField to copy them to the
catchall "text" field, and use that for searching.
- For maximum indexing performance, use the StreamingUpdateSolrServer
java client.
- Remember to run the JVM in server mode, and use a higher logging level
that avoids logging every request
-->
<schema name="sunspot" version="1.0">
<types>
<!-- field type definitions. The "name" attribute is
just a label to be used by field definitions. The "class"
attribute and any other attributes determine the real
behavior of the fieldType.
Class names starting with "solr" refer to java classes in the
org.apache.solr.analysis package.
-->
<!-- *** This fieldType is used by Sunspot! *** -->
<fieldType name="string" class="solr.StrField" omitNorms="true"/>
<!-- *** This fieldType is used by Sunspot! *** -->
<fieldType name="tdouble" class="solr.TrieDoubleField" omitNorms="true"/>
<!-- *** This fieldType is used by Sunspot! *** -->
<fieldType name="rand" class="solr.RandomSortField" omitNorms="true"/>
<!-- *** This fieldType is used by Sunspot! *** -->
<fieldType name="text" class="solr.TextField" omitNorms="false">
<analyzer type="index">
<tokenizer class="solr.NGramTokenizerFactory" minGramSize="3" maxGramSize="15" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.PorterStemFilterFactory"/>
<filter class="solr.WordDelimiterFilterFactory" stemEnglishPossessive="1" splitOnNumerics="1" splitOnCaseChange="1" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="1" preserveOriginal="1"/>
</analyzer>
</fieldType>
<!-- *** This fieldType is used by Sunspot! *** -->
<fieldType name="boolean" class="solr.BoolField" omitNorms="true"/>
<!-- *** This fieldType is used by Sunspot! *** -->
<fieldType name="tint" class="solr.TrieIntField" omitNorms="true"/>
<!-- *** This fieldType is used by Sunspot! *** -->
<fieldType name="tlong" class="solr.TrieLongField" omitNorms="true"/>
<!-- *** This fieldType is used by Sunspot! *** -->
<fieldType name="tfloat" class="solr.TrieFloatField" omitNorms="true"/>
<!-- *** This fieldType is used by Sunspot! *** -->
<fieldType name="tdate" class="solr.TrieDateField"
omitNorms="true"/>
<fieldType name="daterange" class="solr.DateRangeField" omitNorms="true" />
<!-- Special field type for spell correction. Be careful about
adding filters here, as they apply *before* your values go in
the spellcheck. For example, the lowercase filter here means
all spelling suggestions will be lower case (without it,
though, you'd have duplicate suggestions for lower and proper
cased words). -->
<fieldType name="textSpell" class="solr.TextField" positionIncrementGap="100" omitNorms="true">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- A specialized field for geospatial search. If indexed, this fieldType must not be multivalued. -->
<fieldType name="location" class="solr.LatLonType" subFieldSuffix="_coordinate"/>
</types>
<fields>
<!-- Valid attributes for fields:
name: mandatory - the name for the field
type: mandatory - the name of a previously defined type from the
<types> section
indexed: true if this field should be indexed (searchable or sortable)
stored: true if this field should be retrievable
compressed: [false] if this field should be stored using gzip compression
(this will only apply if the field type is compressable; among
the standard field types, only TextField and StrField are)
multiValued: true if this field may contain multiple values per document
omitNorms: (expert) set to true to omit the norms associated with
this field (this disables length normalization and index-time
boosting for the field, and saves some memory). Only full-text
fields or fields that need an index-time boost need norms.
termVectors: [false] set to true to store the term vector for a
given field.
When using MoreLikeThis, fields used for similarity should be
stored for best performance.
termPositions: Store position information with the term vector.
This will increase storage costs.
termOffsets: Store offset information with the term vector. This
will increase storage costs.
default: a value that should be used if no value is specified
when adding a document.
-->
<!-- *** This field is used by Sunspot! *** -->
<field name="id" stored="true" type="string" multiValued="false" indexed="true"/>
<!-- *** This field is used by Sunspot! *** -->
<field name="type" stored="false" type="string" multiValued="true" indexed="true"/>
<!-- *** This field is used by Sunspot! *** -->
<field name="class_name" stored="false" type="string" multiValued="false" indexed="true"/>
<!-- *** This field is used by Sunspot! *** -->
<field name="text" stored="false" type="string" multiValued="true" indexed="true"/>
<!-- *** This field is used by Sunspot! *** -->
<field name="lat" stored="true" type="tdouble" multiValued="false" indexed="true"/>
<!-- *** This field is used by Sunspot! *** -->
<field name="lng" stored="true" type="tdouble" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="random_*" stored="false" type="rand" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="_local*" stored="false" type="tdouble" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_text" stored="false" type="text" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_texts" stored="true" type="text" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_b" stored="false" type="boolean" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_bm" stored="false" type="boolean" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_bs" stored="true" type="boolean" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_bms" stored="true" type="boolean" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_d" stored="false" type="tdate" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_dm" stored="false" type="tdate" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_ds" stored="true" type="tdate" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_dms" stored="true" type="tdate" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_e" stored="false" type="tdouble" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_em" stored="false" type="tdouble" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_es" stored="true" type="tdouble" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_ems" stored="true" type="tdouble" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_f" stored="false" type="tfloat" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_fm" stored="false" type="tfloat" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_fs" stored="true" type="tfloat" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_fms" stored="true" type="tfloat" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_i" stored="false" type="tint" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_im" stored="false" type="tint" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_is" stored="true" type="tint" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_ims" stored="true" type="tint" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_l" stored="false" type="tlong" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_lm" stored="false" type="tlong" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_ls" stored="true" type="tlong" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_lms" stored="true" type="tlong" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_s" stored="false" type="string" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_sm" stored="false" type="string" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_ss" stored="true" type="string" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_sms" stored="true" type="string" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_it" stored="false" type="tint" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_itm" stored="false" type="tint" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_its" stored="true" type="tint" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_itms" stored="true" type="tint" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_ft" stored="false" type="tfloat" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_ftm" stored="false" type="tfloat" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_fts" stored="true" type="tfloat" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_ftms" stored="true" type="tfloat" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_dt" stored="false" type="tdate" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_dtm" stored="false" type="tdate" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_dts" stored="true" type="tdate" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_dtms" stored="true" type="tdate" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_textv" stored="false" termVectors="true" type="text" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_textsv" stored="true" termVectors="true" type="text" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_et" stored="false" termVectors="true" type="tdouble" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_etm" stored="false" termVectors="true" type="tdouble" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_ets" stored="true" termVectors="true" type="tdouble" multiValued="false" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_etms" stored="true" termVectors="true" type="tdouble" multiValued="true" indexed="true"/>
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_dr" stored="false" type="daterange" multiValued="false" indexed="true" />
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_drm" stored="false" type="daterange" multiValued="true" indexed="true" />
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_drs" stored="true" type="daterange" multiValued="false" indexed="true" />
<!-- *** This dynamicField is used by Sunspot! *** -->
<dynamicField name="*_drms" stored="true" type="daterange" multiValued="true" indexed="true" />
<!-- Type used to index the lat and lon components for the "location" FieldType -->
<dynamicField name="*_coordinate" type="tdouble" indexed="true" stored="false" multiValued="false"/>
<dynamicField name="*_p" type="location" indexed="true" stored="true" multiValued="false"/>
<dynamicField name="*_ll" stored="false" type="location" multiValued="false" indexed="true"/>
<dynamicField name="*_llm" stored="false" type="location" multiValued="true" indexed="true"/>
<dynamicField name="*_lls" stored="true" type="location" multiValued="false" indexed="true"/>
<dynamicField name="*_llms" stored="true" type="location" multiValued="true" indexed="true"/>
<field name="textSpell" stored="false" type="textSpell" multiValued="true" indexed="true"/>
<!-- required by Solr 4 -->
<field name="_version_" type="string" indexed="true" stored="true" multiValued="false" />
</fields>
<!-- Field to use to determine and enforce document uniqueness.
Unless this field is marked with required="false", it will be a required field
-->
<uniqueKey>id</uniqueKey>
<!-- field for the QueryParser to use when an explicit fieldname is absent -->
<defaultSearchField>text</defaultSearchField>
<!-- SolrQueryParser configuration: defaultOperator="AND|OR" -->
<solrQueryParser defaultOperator="AND"/>
<!-- copyField commands copy one field to another at the time a document
is added to the index. It's used either to index the same field differently,
or to add multiple fields to the same field for easier/faster
searching. -->
<!-- Use copyField to copy the fields you want to run spell checking
on into one field. For example: -->
<copyField source="*_text" dest="textSpell" />
<copyField source="*_s" dest="textSpell" />
</schema>
答案 0 :(得分:1)
您没有对您正在搜索的字段类型进行任何说明,但如果它的类型为“text”,则分析链看起来不适合您要执行的操作。输入(NGramTokenizer)和只是小写,不会给出你在查询端与StandardTokenizer一起期待的结果。
创建一个具有更简化定义的新字段(现在对于索引和查询可能都是相同的),它只包含一个空格标记器或另一个更标准的标记化器 - 请参阅差异的the reference manual for examples。你可能也想要一个小写的过滤器。
您可能会遇到变音符号和其他特定德语术语的问题,但ICU * - 过滤器和标记符的范围比其他更加国际化。还有一个过滤器可以将单词分成它们的组件(因为你和我们的挪威语有同样的问题,那里的单词是一起写的,而不是英语的分割方式)。
Solr Admin下的“分析”页面是一个开始调试的好地方 - 它会准确显示在索引和查询方面进行的转换,让您可以看到条款不匹配的原因和每个步骤的条款是什么样的。
答案 1 :(得分:0)
对于拉丁语重音字符,请使用
<filter class="solr.ASCIIFoldingFilterFactory"/>
这将以ASCII格式存储所有单词并删除重音符。
您的文本类型在查询方面看起来非常不平衡。尝试这样的事情,然后重新索引你的数据(我目前正在法国网站上工作)。我不认为PorterStemFilterFactor对德语有好处,还有其他的词干分析器更好用: -
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="40" />
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>