Using Lucene Search Text Queries

Using Lucene Search Text Queries

The Geoportal uses a sophisticated search engine that provides many search options, ranking options, fast performance, and extensibility. The search engine is based on the open source search engine Apache Lucene. To make the most of the Geoportal search page, keep in mind the following features that Lucene provides for search syntax:

1. Terms

   A query is broken up into terms and operators. There are two types of terms: Single Terms and Phrases. A Single Term is a single word such as air or quality. A Phrase is a group of words surrounded by double quotes such as "air quality". Multiple terms can be combined together with Boolean operators to form a more complex query. Examples:

   • Searching for air could result in 35 hits (items contain the word air)
   • Searching for quality results in 123 hits (items contain the word quality)
   • Searching for air quality (without quotes) results in 148 hits (items contain the words air or quality or both)
   • Searching for air AND quality results in 10 hits (results contain both words air and quality)
   • Searching for "air quality" (with quotes) results in 7 hits (items contain the words air and quality directly after each other)
   • Searching for title:air results in 5 hits (items contain the word air in the title)
   • Searching for title:quality results in 14 hits (items contain the word quality in the title)
   • Searching for +title:air +title:quality or title:"air quality" results in 2 hits (both items contain both words air and quality in the title)

2. Special Characters

   The Geoportal supports escaping special characters that are part of the query syntax. The current list special characters are + - && || ! ( ) { } [ ] ^ " ~ * ? : \ To escape these character use the \ before the character. For example to search for items that contain the scale hit 1:250k use the query: \1\:250k.

3. Fields

   Lucene supports fielded data. When performing a search you can either specify a field, or use the default field. The field names and default field is implementation-specific. You can search any field by typing the field name followed by a colon and then the term for which you are looking. Targeting a specific field in the query can be more accurate than just searching with terms. Keep in mind that some fields are case sensitive. Remember that certain special characters must be escaped in the query by using a back-slash (\) character or embraced within quotation ("") whenever they are a part of text to search. Examples:

   • title:"The Right Way" AND text:"don't go this way"
   • uuid:"{550E8400-E29B-41D4-A716-446655440000}"
   • uuid:\{550E8400\-E29B\-41D4\-A716\-446655440000\}
   • resource.url:"http://server.arcgisonline.com/ArcGIS/rest/services/ESRI_StreetMap_World_2D/MapServer"

   Note:

   The field is only valid for the term that it directly precedes, so the query title:Do it right will only find "Do" in the title field.

4. Wildcard Searches

   The Geoportal supports single and multiple character wildcard searches within single terms (not within phrase queries).

   Caution:

   You cannot use a * or ? symbol as the first character of a search.

   • To perform a single character wildcard search use the "?" symbol. The single character wildcard search looks for terms that match that with the single character replaced. For example, to search for text or test you can use the search:te?t
   • To perform a multiple character wildcard search use the "*" symbol. Multiple character wildcard searches looks for 0 or more characters. For example, to search for test, tests or tester, you can use the search: test* . You can also use the wildcard searches in the middle of a term: te*t
5. Fuzzy Searches

   The Geoportal supports fuzzy searches based on the Levenshtein Distance, or Edit Distance algorithm. To do a fuzzy search use the tilde, "~", symbol at the end of a Single Term. For example to search for a term similar in spelling to air use the fuzzy search: air~. This search will not only find items containing terms like air and airplane, but also aid. The Geoportal supports specifying the required similarity. The value is between 0 and 1, with a value closer to 1 only terms with a higher similarity will be matched. For example: air~0.8 The default that is used if the parameter is not given is 0.5.

6. Proximity Searches

   The Geoportal supports finding words are a within a specific distance away. To do a proximity search use the tilde, "~", symbol at the end of a Phrase. For example to search for air and quality within 10 words of each other in a document use the search: air quality"~10

7. Range Searches

   The Geoportal supports range queries for envelope and timestamp. This allows the user to match documents whose field(s) values are between the lower and upper bound specified by the Range Query. Range Queries can be inclusive or exclusive of the upper and lower bounds.

   • Envelope examples:
     • envelope:[-80,-70 TO +30,+70] This search would result in returned documents that intersect a spatial envelope with a Southwest bounding coordinate of -80° W and -70° S, and a Northeast bounding coordinate of 30° W and 70° N.
     • envelope:{-80,-70 TO +30,+70} This search would result in returned documents that fall exactly within the range of a spatial envelope with a Southwest bounding coordinate of -80° W and -70° S, and a Northeast bounding coordinate of 30° W and 70° N.

     Note:

     The syntax here is important. It is the field name (envelope) followed by colon (:), and then either an inclusive range definition or exclusive range definition. For an inclusive search square brackets ([ ]) are used and for an exclusive search curly braces are used ({ }). The difference is that exclusive range searches will select only those resources in which the envelope falls exactly within the range specified, while an inclusive range search will select resources that intersect the range specified but also fall outside. The first pair of values are the lower left corner coordinates followed by TO keyword (uppercase), than the upper right corner coordinates. Coordinates are always given in WGS 1984 (4236) projection system. Wildcards can also be used in place of a single coordinate or an entire pair of corner coordinates, for example envelope:[*,-70 TO +30,*] or envelope:{-80,-70 TO *}

   • Timestamp examples:
     • dateModified:[2009-10-11 TO 2009-11-10]
     • dateModified:[2006 TO 2010]
     • dateModified:2009-12
     For the timestamp syntax, the field name specified is dateModifiedfollowed by colon and then an inclusive range definition. The first example will select resources with dateModified between 2009-10-11 to 2009-11-10 including these range dates. The second example will select resources with dateModified from the year 2006 to 2010. The last example will select resources with dateModified in December of 2009 (no brackets required).
8. Boosting a Term

   The Geoportal provides the relevance level of matching documents based on the terms found. To boost a term use the caret, ^, symbol with a boost factor (a number) at the end of the term you are searching. The higher the boost factor, the more relevant the term will be. Boosting allows you to control the relevance of a document by boosting its term. For example, if you are searching for air quality and you want the term air to be more relevant, boost it using the ^ symbol along with the boost factor next to the term. You would type: air^4 quality. This will make documents with the term air appear more relevant. You can also boost Phrase Terms as in the example: "air quality"^4 "water quality". By default, the boost factor is 1. Although the boost factor must be positive, it can be less than 1 (e.g. 0.2)

9. Boolean Operators

   Boolean operators allow terms to be combined through logic operators. The Geoportal supports AND, +, OR, NOT and - as Boolean operators.

   Note:

   Boolean operators must be ALL CAPS

   • The OR operator is the default conjunction operator. This means that if there is no Boolean operator between two terms, the OR operator is used. The OR operator links two terms and finds a matching document if either of the terms exist in a document. This is equivalent to a union using sets. The symbol || can be used in place of the word OR.
   • The AND operator matches documents where both terms exist anywhere in the text of a single document. This is equivalent to an intersection using sets. The symbol && can be used in place of the word AND.
   • The + or required operator requires that the term after the + symbol exist somewhere in a field of a single document.
   • The NOT operator excludes documents that contain the term after NOT. This is equivalent to a difference using sets. The symbol ! can be used in place of the word NOT.

     Note:

     The NOT operator cannot be used with just one term.

10. Grouping

    The Geoportal supports using parentheses to group clauses to form sub queries. This can be very useful if you want to control the boolean logic for a query. For example: (air OR water) AND quality will find documents containing the words air and quality or the words water and quality.

11. Field Grouping

    The Geoportal supports using parentheses to group multiple clauses to a single field. For example: title:(air OR water) finds items that contain the words air or water in the title.

Fore more information on how to specifically leverage Lucene search syntax for powerful searching in your Geoportal, please see the Lucene website.