Improve support of Hunspell in ispell dictionary.

Now it's possible to load recent version of Hunspell for several languages.
To handle these dictionaries Hunspell patch adds support for:
* FLAG long - sets the double extended ASCII character flag type
* FLAG num - sets the decimal number flag type (from 1 to 65535)
* AF parameter - alias for flag's set

Also it moves test dictionaries into separate directory.

Author: Artur Zakirov with editorization by me

Author: feodor

doc/src/sgml/textsearch.sgml

@@ -2615,18 +2615,41 @@ SELECT plainto_tsquery('supernova star');
    </para>
 
    <para>
-    To create an <application>Ispell</> dictionary, use the built-in
-    <literal>ispell</literal> template and specify several parameters:
+    To create an <application>Ispell</> dictionary perform these steps:
    </para>
-
+   <itemizedlist spacing="compact" mark="bullet">
+    <listitem>
+     <para>
+      download dictionary configuration files. <productname>OpenOffice</>
+      extension files have the <filename>.oxt</> extension. It is necessary
+      to extract <filename>.aff</> and <filename>.dic</> files, change
+      extensions to <filename>.affix</> and <filename>.dict</>. For some
+      dictionary files it is also needed to convert characters to the UTF-8
+      encoding with commands (for example, for norwegian language dictionary):
 <programlisting>
-CREATE TEXT SEARCH DICTIONARY english_ispell (
+iconv -f ISO_8859-1 -t UTF-8 -o nn_no.affix nn_NO.aff
+iconv -f ISO_8859-1 -t UTF-8 -o nn_no.dict nn_NO.dic
+</programlisting>
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      copy files to the <filename>$SHAREDIR/tsearch_data</> directory
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      load files into PostgreSQL with the following command:
+<programlisting>
+CREATE TEXT SEARCH DICTIONARY english_hunspell (
     TEMPLATE = ispell,
-    DictFile = english,
-    AffFile = english,
-    StopWords = english
-);
+    DictFile = en_us,
+    AffFile = en_us,
+    Stopwords = english);
 </programlisting>
+     </para>
+    </listitem>
+   </itemizedlist>
 
    <para>
     Here, <literal>DictFile</>, <literal>AffFile</>, and <literal>StopWords</>
@@ -2642,6 +2665,56 @@ CREATE TEXT SEARCH DICTIONARY english_ispell (
     example, a Snowball dictionary, which recognizes everything.
    </para>
 
+   <para>
+    The <filename>.affix</> file of <application>Ispell</> has the following
+    structure:
+<programlisting>
+prefixes
+flag *A:
+    .           >   RE      # As in enter > reenter
+suffixes
+flag T:
+    E           >   ST      # As in late > latest
+    [^AEIOU]Y   >   -Y,IEST # As in dirty > dirtiest
+    [AEIOU]Y    >   EST     # As in gray > grayest
+    [^EY]       >   EST     # As in small > smallest
+</programlisting>
+   </para>
+   <para>
+    And the <filename>.dict</> file has the following structure:
+<programlisting>
+lapse/ADGRS
+lard/DGRS
+large/PRTY
+lark/MRS
+</programlisting>
+   </para>
+
+   <para>
+    Format of the <filename>.dict</> file is:
+<programlisting>
+basic_form/affix_class_name
+</programlisting>
+   </para>
+
+   <para>
+    In the <filename>.affix</> file every affix flag is described in the
+    following format:
+<programlisting>
+condition > [-stripping_letters,] adding_affix
+</programlisting>
+   </para>
+
+   <para>
+    Here, condition has a format similar to the format of regular expressions.
+    It can use groupings <literal>[...]</> and <literal>[^...]</>.
+    For example, <literal>[AEIOU]Y</> means that the last letter of the word
+    is <literal>"y"</> and the penultimate letter is <literal>"a"</>,
+    <literal>"e"</>, <literal>"i"</>, <literal>"o"</> or <literal>"u"</>.
+    <literal>[^EY]</> means that the last letter is neither <literal>"e"</>
+    nor <literal>"y"</>.
+   </para>
+
    <para>
     Ispell dictionaries support splitting compound words;
     a useful feature.
@@ -2663,6 +2736,65 @@ SELECT ts_lexize('norwegian_ispell', 'sjokoladefabrikk');
 </programlisting>
    </para>
 
+   <para>
+    <application>MySpell</> format is a subset of <application>Hunspell</>.
+    The <filename>.affix</> file of <application>Hunspell</> has the following
+    structure:
+<programlisting>
+PFX A Y 1
+PFX A   0     re         .
+SFX T N 4
+SFX T   0     st         e
+SFX T   y     iest       [^aeiou]y
+SFX T   0     est        [aeiou]y
+SFX T   0     est        [^ey]
+</programlisting>
+   </para>
+
+   <para>
+    The first line of an affix class is the header. Fields of an affix rules are
+    listed after the header:
+   </para>
+   <itemizedlist spacing="compact" mark="bullet">
+    <listitem>
+     <para>
+      parameter name (PFX or SFX)
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      flag (name of the affix class)
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      stripping characters from beginning (at prefix) or end (at suffix) of the
+      word
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      adding affix
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      condition that has a format similar to the format of regular expressions.
+     </para>
+    </listitem>
+   </itemizedlist>
+
+   <para>
+    The <filename>.dict</> file looks like the <filename>.dict</> file of
+    <application>Ispell</>:
+<programlisting>
+larder/M
+lardy/RT
+large/RSPMYT
+largehearted
+</programlisting>
+   </para>
+
    <note>
     <para>
      <application>MySpell</> does not support compound words.

src/backend/tsearch/Makefile

@@ -13,8 +13,11 @@ include $(top_builddir)/src/Makefile.global
 
 DICTDIR=tsearch_data
 
-DICTFILES=synonym_sample.syn thesaurus_sample.ths hunspell_sample.affix \
-	ispell_sample.affix ispell_sample.dict
+DICTFILES=dicts/synonym_sample.syn dicts/thesaurus_sample.ths \
+	dicts/hunspell_sample.affix \
+	dicts/ispell_sample.affix dicts/ispell_sample.dict \
+	dicts/hunspell_sample_long.affix dicts/hunspell_sample_long.dict \
+	dicts/hunspell_sample_num.affix dicts/hunspell_sample_num.dict
 
 OBJS = ts_locale.o ts_parse.o wparser.o wparser_def.o dict.o \
 	dict_simple.o dict_synonym.o dict_thesaurus.o \

src/backend/tsearch/hunspell_sample.affix renamed to src/backend/tsearch/dicts/hunspell_sample.affix

@@ -0,0 +1,35 @@
+FLAG long
+
+AF 7
+AF cZ		#1
+AF cL		#2
+AF sGsJpUsS	#3
+AF sSpB		#4
+AF cZsS		#5
+AF sScZs\	#6
+AF sA		#7
+
+COMPOUNDFLAG cZ
+ONLYINCOMPOUND cL
+
+PFX pB Y 1
+PFX pB   0	re	.
+
+PFX pU N 1
+PFX pU   0	un	.
+
+SFX sJ Y 1
+SFX sJ   0	INGS	[^E]
+
+SFX sG Y 1
+SFX sG   0	ING		[^E]
+
+SFX sS Y 1
+SFX sS   0	S	[^SXZHY]
+
+SFX sA Y 1
+SFX sA   Y	IES	[^AEIOU]Y
+
+SFX s\ N 1
+SFX s\   0	Y/2	[^Y]
+

src/backend/tsearch/dicts/hunspell_sample_long.affix

@@ -0,0 +1,8 @@
+book/3
+booking/4
+footballklubber
+foot/5
+football/1
+ball/6
+klubber/1
+sky/7

src/backend/tsearch/dicts/hunspell_sample_long.dict

@@ -0,0 +1,26 @@
+FLAG num
+
+COMPOUNDFLAG 101
+ONLYINCOMPOUND 102
+
+PFX 201 Y 1
+PFX 201   0	re	.
+
+PFX 202 N 1
+PFX 202   0	un	.
+
+SFX 301 Y 1
+SFX 301   0	INGS	[^E]
+
+SFX 302 Y 1
+SFX 302   0	ING		[^E]
+
+SFX 303 Y 1
+SFX 303   0	S	[^SXZHY]
+
+SFX 304 Y 1
+SFX 304   Y	IES	[^AEIOU]Y
+
+SFX 305 N 1
+SFX 305   0	Y/102	[^Y]
+

src/backend/tsearch/dicts/hunspell_sample_num.affix

@@ -0,0 +1,8 @@
+book/302,301,202,303
+booking/303,201
+footballklubber
+foot/101,303
+football/101
+ball/303,101,305
+klubber/101
+sky/304