Dynamic Fields: Title Author Keywords Body Catchall Solrconfig - XML Catchall

field when users or clients do not specify a field to query.
For example, title, author,

keywords, and body
may all be fields that should be searched by default, with copy field rules for each field to
copy to a catchall
field (for example, it could be named anything). Later you can set a rule in solrconfig.xml
to search the
catchall field by default. One caveat to this is your index will grow when using copy
fields. However,
whether this becomes problematic for you and the final size will depend on the number of
fields being
copied, the number of destination fields being copied to, the analysis in use, and the
available disk space.
The maxChars parameter, an int parameter, establishes an upper limit for the number of
characters to be
copied from the source value when constructing the value added to the destination field. This
limit is useful
for situations in which you want to copy some data from the source field, but also control
the size of index
files.
Both the source and the destination of copyField can contain either leading or trailing
asterisks, which will
match anything. For example, the following line will copy the contents of all incoming fields
that match the
wildcard pattern *_t to the text field.:
<copyField source="*_t" dest="text" maxChars="25000" />
The copyField command can use a wildcard (*) character in the dest parameter only if the
source parameter contains one as well. copyField uses the matching glob from the source
field for the dest field name into which the source content is copied.
Copying is done at the stream source level and no copy feeds into another copy. This means
that copy fields
cannot be chained i.e., you cannot copy from here to there and then from there to
elsewhere. However, the
same source field can be copied to multiple destination fields:
Apache Solr Reference Guide 7.7 Page 177 of 1426
c 2019, Apache Software Foundation Guide Version 7.7 - Published: 2019-03-04
<copyField source="here" dest="there"/>
<copyField source="here" dest="elsewhere"/>
Page 178 of 1426 Apache Solr Reference Guide 7.7
Guide Version 7.7 - Published: 2019-03-04 c 2019, Apache Software Foundation
Dynamic Fields
Dynamic fields allow Solr to index fields that you did not explicitly define in your schema.
This is useful if you discover you have forgotten to define one or more fields. Dynamic
fields can make your
application less brittle by providing some flexibility in the documents you can add to Solr.
A dynamic field is just like a regular field except it has a name with a wildcard in it. When
you are indexing
documents, a field that does not match any explicitly defined fields can be matched with a
dynamic field.
For example, suppose your schema includes a dynamic field with a name of *_i. If you attempt
to index a
document with a cost_i field, but no explicit cost_i field is defined in the schema, then
the cost_i field will
have the field type and analysis defined for *_i.
Like regular fields, dynamic fields have a name, a field type, and options.
<dynamicField name="*_i" type="int" indexed="true" stored="true"/>
It is recommended that you include basic dynamic field mappings (like that shown above) in
your
schema.xml. The mappings can be very useful.
Other Schema Elements

This section describes several other important elements of schema.xml not covered in earlier
sections.
Unique Key
The uniqueKey element specifies which field is a unique identifier for documents. Although
uniqueKey is not
required, it is nearly always warranted by your application design. For example, uniqueKey
should be used if
you will ever update a document in the index.
You can define the unique key field by naming it:
<uniqueKey>id</uniqueKey>
Schema defaults and copyFields cannot be used to populate the uniqueKey field. The
fieldType of
uniqueKey must not be analyzed and must not be any of the *PointField types. You can use
UUIDUpdateProcessorFactory to have uniqueKey values generated automatically.
Further, the operation will fail if the uniqueKey field is used, but is multivalued (or
inherits the multivalueness
from the fieldtype). However, uniqueKey will continue to work, as long as the field is
properly used.
Similarity
Similarity is a Lucene class used to score a document in searching.
Each collection has one "global" Similarity, and by default Solr uses an implicit
SchemaSimilarityFactory
which allows individual field types to be configured with a "per-type" specific Similarity
and implicitly uses
BM25Similarity for any field type which does not have an explicit Similarity.
This default behavior can be overridden by declaring a top level <similarity/> element in
your schema.xml,
outside of any single field type. This similarity declaration can either refer directly to
the name of a class with
a no-argument constructor, such as in this example showing BM25Similarity:
<similarity class="solr.BM25SimilarityFactory"/>
or by referencing a SimilarityFactory implementation, which may take optional
initialization parameters:
<similarity class="solr.DFRSimilarityFactory">
<str name="basicModel">P</str>
<str name="afterEffect">L</str>
<str name="normalization">H2</str>
<float name="c">7</float>
</similarity>
In most cases, specifying global level similarity like this will cause an error if your
schema.xml also includes
field type specific <similarity/> declarations. One key exception to this is that you may
explicitly declare a
SchemaSimilarityFactory and specify what that default behavior will be for all field types
that do not
declare an explicit Similarity using the name of field type (specified by
defaultSimFromFieldType) that is
configured with a specific similarity:
<similarity class="solr.SchemaSimilarityFactory">
<str name="defaultSimFromFieldType">text_dfr</str>
</similarity>
<fieldType name="text_dfr" class="solr.TextField">
<analyzer ... />
<similarity class="solr.DFRSimilarityFactory">
<str name="basicModel">I(F)</str>
<str name="afterEffect">B</str>
<float name="mu">900</float>
</similarity>
</fieldType>
<fieldType name="text_ib" class="solr.TextField">
<analyzer ... />
<similarity class="solr.IBSimilarityFactory">
<str name="distribution">SPL</str>
<str name="lambda">DF</str>
</similarity>
</fieldType>
<fieldType name="text_other" class="solr.TextField">
<analyzer ... />
</fieldType>
In the example above IBSimilarityFactory (using the Information-Based model) will be used
for any fields
of type text_ib, while DFRSimilarityFactory (divergence from random) will be used for any
fields of type
text_dfr, as well as any fields using a type that does not explicitly specify a
<similarity/>.
If SchemaSimilarityFactory is explicitly declared without configuring a
defaultSimFromFieldType, then
BM25Similarity is implicitly used as the default.
In addition to the various factories mentioned on this page, there are several other
similarity
implementations that can be used such as the SweetSpotSimilarityFactory,
ClassicSimilarityFactory,
etc. For details, see the Solr Javadocs for the similarity factories.
Schema API
The Schema API allows you to use an HTTP API to manage many of the elements of your schema.
The Schema API utilizes the ManagedIndexSchemaFactory class, which is the default schema
factory in
modern Solr versions. See the section Schema Factory Definition in SolrConfig for more
information about
choosing a schema factory for your index.
This API provides read and write access to the Solr schema for each collection (or core, when
using
standalone Solr). Read access to all schema elements is supported. Fields, dynamic fields,
field types and
copyField rules may be added, removed or replaced. Future Solr releases will extend write
access to allow
more schema elements to be modified.
Why is hand editing of the managed schema discouraged?

The file named "managed-schema" in the example configurations may include a note that
recommends never hand-editing the file. Before the Schema API existed, such edits were
the only way to make changes to the schema, and users may have a strong desire to
continue making changes this way.
The reason that this is discouraged is because hand-edits of the schema may be lost if the
Schema API described here is later used to make a change, unless the core or collection is
reloaded or Solr is restarted before using the Schema API. If care is taken to always reload
or restart after a manual edit, then there is no problem at all with doing those edits.
The API allows two output modes for all calls: JSON or XML. When requesting the complete
schema, there is
another output mode which is XML modeled after the managed-schema file itself, which is in
XML format.
When modifying the schema with the API, a core reload will automatically occur in order for
the changes to
be available immediately for documents indexed thereafter. Previously indexed documents will
not be
automatically updated - they must be re-indexed if existing index data uses schema elements
that you
changed.
Re-index after schema modifications!

If you modify your schema, you will likely need to re-index all documents. If you do not, you
may lose access to documents, or not be able to interpret them properly, e.g., after
replacing a field type.
Modifying your schema will never modify any documents that are already indexed. You
must re-index documents in order to apply schema changes to them. Queries and updates
made after the change may encounter errors that were not present before the change.
Completely deleting the index and rebuilding it is usually the only option to fix such
errors.
Modify the Schema
To add, remove or replace fields, dynamic field rules, copy field rules, or new field types,
you can send a
POST request to the /collection/schema/ endpoint with a sequence of commands in JSON format
to
perform the requested actions. The following commands are supported:
• add-field: add a new field with parameters you provide.
• delete-field: delete a field.
• replace-field: replace an existing field with one that is differently configured.
• add-dynamic-field: add a new dynamic field rule with parameters you provide.
• delete-dynamic-field: delete a dynamic field rule.
• replace-dynamic-field: replace an existing dynamic field rule with one that is
differently configured.
• add-field-type: add a new field type with parameters you provide.
• delete-field-type: delete a field type.
• replace-field-type: replace an existing field type with one that is differently
configured.
• add-copy-field: add a new copy field rule.
• delete-copy-field: delete a copy field rule.
These commands can be issued in separate POST requests or in the same POST request. Commands
are
executed in the order in which they are specified.
In each case, the response will include the status and the time to process the request, but
will not include
the entire schema.
When modifying the schema with the API, a core reload will automatically occur in order for
the changes to
be available immediately for documents indexed thereafter. Previously indexed documents will
not be
automatically handled - they must be re-indexed if they used schema elements that you
changed.
Add a New Field
The add-field command adds a new field definition to your schema. If a field with the same
name exists an
error is thrown.
All of the properties available when defining a field with manual schema.xml edits can be
passed via the API.
These request attributes are described in detail in the section Defining Fields.
For example, to define a new stored field named "sell_by", of type "pdate", you would POST
the following
request:
V1 API
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":{
"name":"sell_by",
"type":"pdate",
"stored":true }
}' http://localhost:8983/solr/gettingstarted/schema
V2 API
"add-field":{
"name":"sell_by",
"type":"pdate",
"stored":true }
}' http://localhost:8983/api/cores/gettingstarted/schema
Delete a Field
The delete-field command removes a field definition from your schema. If the field does not
exist in the
schema, or if the field is the source or destination of a copy field rule, an error is
thrown.
For example, to delete a field named "sell_by", you would POST the following request:
V1 API
"delete-field" : { "name":"sell_by" }
V2 API
"delete-field" : { "name":"sell_by" }
Replace a Field
The replace-field command replaces a field’s definition. Note that you must supply the full
definition for a
field - this command will not partially modify a field’s definition. If the field does not
exist in the schema an
error is thrown.
All of the properties available when defining a field with manual schema.xml edits can be
passed via the API.
These request attributes are described in detail in the section Defining Fields.
For example, to replace the definition of an existing field "sell_by", to make it be of type
"date" and to not be
stored, you would POST the following request:
V1 API
"replace-field":{
"name":"sell_by",
"type":"date",
"stored":false }
V2 API
"replace-field":{
"name":"sell_by",
"type":"date",
"stored":false }
Add a Dynamic Field Rule
The add-dynamic-field command adds a new dynamic field rule to your schema.
All of the properties available when editing schema.xml can be passed with the POST request.
The section
Dynamic Fields has details on all of the attributes that can be defined for a dynamic field
rule.
For example, to create a new dynamic field rule where all incoming fields ending with "_s"
would be stored
and have field type "string", you can POST a request like this:
V1 API
"add-dynamic-field":{
"name":"*_s",
"type":"string",
"stored":true }
V2 API
"add-dynamic-field":{
"name":"*_s",
"type":"string",
"stored":true }
Delete a Dynamic Field Rule
The delete-dynamic-field command deletes a dynamic field rule from your schema. If the
dynamic field
rule does not exist in the schema, or if the schema contains a copy field rule with a target
or destination that
matches only this dynamic field rule, an error is thrown.
For example, to delete a dynamic field rule matching "*_s", you can POST a request like this:
V1 API
"delete-dynamic-field":{ "name":"*_s" }
V2 API
"delete-dynamic-field":{ "name":"*_s" }
Replace a Dynamic Field Rule
The replace-dynamic-field command replaces a dynamic field rule in your schema. Note that
you must
supply the full definition for a dynamic field rule - this command will not partially modify
a dynamic field
rule’s definition. If the dynamic field rule does not exist in the schema an error is
thrown.
All of the properties available when editing schema.xml can be passed with the POST request.
The section
Dynamic Fields has details on all of the attributes that can be defined for a dynamic field
rule.
For example, to replace the definition of the "*_s" dynamic field rule with one where the
field type is
"text_general" and it’s not stored, you can POST a request like this:
V1 API
"replace-dynamic-field":{
"name":"*_s",
"type":"text_general",
"stored":false }
V2 API
"replace-dynamic-field":{
"name":"*_s",
"type":"text_general",
"stored":false }
Add a New Field Type
The add-field-type command adds a new field type to your schema.
All of the field type properties available when editing schema.xml by hand are available for
use in a POST
request. The structure of the command is a JSON mapping of the standard field type
definition, including the
name, class, index and query analyzer definitions, etc. Details of all of the available
options are described in
the section Solr Field Types.
For example, to create a new field type named "myNewTxtField", you can POST a request as
follows:
V1 API with Single Analysis
"add-field-type" : {
"name":"myNewTxtField",
"class":"solr.TextField",
"positionIncrementGap":"100",
"analyzer" : {
"charFilters":[{
"class":"solr.PatternReplaceCharFilterFactory",
"replacement":"$1$1",
"pattern":"([a-zA-Z])\\\\1+" }],
"tokenizer":{
"class":"solr.WhitespaceTokenizerFactory" },
"filters":[{
"class":"solr.WordDelimiterFilterFactory",
"preserveOriginal":"0" }]}}
Note in this example that we have only defined a single analyzer section that will apply to
index analysis
and query analysis.
V1 API with Two Analyzers
If we wanted to define separate analysis, we would replace the analyzer section in the above
example
with separate sections for indexAnalyzer and queryAnalyzer. As in this example:
"add-field-type":{
"name":"myNewTextField",
"indexAnalyzer":{
"tokenizer":{
"class":"solr.PathHierarchyTokenizerFactory",
"delimiter":"/" }},
"queryAnalyzer":{
"tokenizer":{
"class":"solr.KeywordTokenizerFactory" }}}
V2 API with Two Analyzers
To define two analyzers with the V2 API, we just use a different endpoint:
"add-field-type":{
"name":"myNewTextField",
"indexAnalyzer":{
"tokenizer":{
"class":"solr.PathHierarchyTokenizerFactory",
"delimiter":"/" }},
"queryAnalyzer":{
"tokenizer":{
"class":"solr.KeywordTokenizerFactory" }}}
Delete a Field Type
The delete-field-type command removes a field type from your schema. If the field type does
not exist in
the schema, or if any field or dynamic field rule in the schema uses the field type, an error
is thrown.
For example, to delete the field type named "myNewTxtField", you can make a POST request as
follows:
V1 API
"delete-field-type":{ "name":"myNewTxtField" }
V2 API
"delete-field-type":{ "name":"myNewTxtField" }
Replace a Field Type
The replace-field-type command replaces a field type in your schema. Note that you must
supply the full
definition for a field type - this command will not partially modify a field type’s
definition. If the field type
does not exist in the schema an error is thrown.
All of the field type properties available when editing schema.xml by hand are available for
use in a POST
request. The structure of the command is a JSON mapping of the standard field type
definition, including the
name, class, index and query analyzer definitions, etc. Details of all of the available
options are described in
the section Solr Field Types.
For example, to replace the definition of a field type named "myNewTxtField", you can make a
POST request
as follows:
V1 API
"replace-field-type":{
"name":"myNewTxtField",
"positionIncrementGap":"100",
"analyzer":{
"tokenizer":{
"class":"solr.StandardTokenizerFactory" }}}

Dynamic Fields: Title Author Keywords Body Catchall Solrconfig - XML Catchall

Uploaded by

Copyright:

Available Formats

Dynamic Fields: Title Author Keywords Body Catchall Solrconfig - XML Catchall

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Dynamic Fields: Title Author Keywords Body Catchall Solrconfig - XML Catchall

Uploaded by

Copyright:

Available Formats

field when users or clients do not specify a field to query.

For example, title, author,

Other Schema Elements

Why is hand editing of the managed schema discouraged?

Re-index after schema modifications!

You might also like