# FlexSearch

### Superfast lightweight full text search engine for JavaScript.

Searching full text with FlexSearch is up to 1,000 times faster than with ElasticSearch.
Benchmark: <a href="https://jsperf.com/flexsearch" target="_blank">https://jsperf.com/flexsearch</a>

All Features:
<ul>
    <li>Partial Words</li>
    <li>Multiple Words</li>
    <li>Flexible Word Order</li>
    <li><a href="#phonetic">Phonetic Search</a></li>
    <li>Limit Results</li>
    <li>Caching</li>
    <li>Asynchronous Mode</li>
    <li>Custom Matchers</li>
    <li>Custom Encoders</li>
</ul>

## Installation

##### Node.js

```npm
npm install flexsearch
```

In your code include as follows:

```javascript
var FlexSearch = require("FlexSearch");
```

##### HTML / Javascript

```html
<html>
<head>
    <script src="https://cdn.rawgit.com/nextapps-de/flexsearch/master/flexsearch.min.js"></script>
</head>
...
```

__AMD__

```javascript
var FlexSearch = require("FlexSearch");
```

## Usage (API)

#### Create a new index

```js
var index = new FlexSearch();
```

alternatively you can also use:

```js
var index = FlexSearch.create();
```

##### Create a new index with custom options

> FlexSearch.__create__(options)

```js
var index = new FlexSearch({

    // default values:

    type: "integer",
    encode: "icase",
    boolean: "and",
    size: 4000,
    depth: 3,
    multi: false,
    strict: false,
    ordered: false,
    async: false,
    cache: false
});
```

__Read more:__ <a href="#phonetic">Phonetic Search</a>, <a href="#compare">Phonetic Comparison</a>, <a href="#memory">Improve Memory Usage</a>

#### Add items to an index

> Index.__add___(id, string)

```js
index.add(10025, "John Doe");
```

#### Search items

> Index.__search__(string, limit, callback)

```js
index.search("John");
```

##### Limit the result

```js
index.search("John", 10);
```

##### Perform queries asynchronously

```js
index.search("John", function(result){
    
    // array of results
});
```

#### Update item to the index

> Index.__update__(id, string)

```js
index.update(10025, "Road Runner");
```

#### Remove item to the index

> Index.__remove__(id)

```js
index.remove(10025);
```

#### Destroy the index

```js
index.destroy();
```

#### Initialize the index

> Index.__init__(options)

```js
index.init();
```

#### Add custom matcher

> Index.__addMatcher__(_KEY_VALUE_PAIRS_)

```js
index.addMatcher({

    'ä': 'a', // replaces all 'ä' to 'a'
    'ö': 'o',
    'Ü': 'u'
});
```

#### Add custom encoder

```js
var index = new FlexSearch({

    encode: function(str){
    
        // do something with str ...
        
        return str;
    }
});
```

#### Get info

```js
index.info();
```

Returns information about the index, e.g.:

```json
{
    "bytes": 3600356288,
    "chunks": 9,
    "fragmentation": 0,
    "fragments": 0,
    "id": 0,
    "length": 7798,
    "matchers": 0,
    "size": 10000,
    "status": false
}
```

__Note:__ When the fragmentation value is about 50% or higher, your should consider using _cleanup()_ to free all fragmented available memory.

#### Optimize / Cleanup index

```js
index.cleanup();
```

<a name="options" id="options"></a>
## Options

<table>
    <tr>
        <th align="left">Option</th>
        <th align="left">Values</th>
        <th align="left">Description</th>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">type</td>
        <td vertical="top" vertical-align="top">
            "byte"<br>
            "short"<br>
            "integer"<br>
            "float"<br>
            "string"
        </td>
        <td vertical-align="top">The data type of passed IDs has to be specified on creation. It is recommended to uses to most lowest possible data range here, e.g. use "short" when IDs are not higher than 65,535.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">encode</td>
        <td>
            false<br>
            "icase"<br>
            "simple"<br>
            "advanced"<br>
            "extra"<br>
            {function}
        </td>
        <td>The encoding type. Choose one of the built-ins or pass a custom encoding function.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">boolean</td>
        <td>
            "and"<br>
            "or"
        </td>
        <td>The applied boolean model when comparing multiple words. <b>Note:</b> When using "or" the first word is also compared with "and". Example: a query with 3 words, results has either: matched word 1 & 2 and matched word 1 & 3.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">size</td>
        <td>2500 - 10000</td>
        <td>The size of chunks. It depends on content length which value fits best. Short content length (e.g. User names) are faster with a chunk size of 2,500. Bigger text runs faster with a chunk size of 10,000. <b>Note:</b> It is recommended to use a minimum chunk size of the maximum content length which has to be indexed to prevent fragmentation.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">depth</td>
        <td>0 - 6</td>
        <td>Set the depth of register. It is recommended to use a value in relation to the number of stored index and content length for an optimum performance-memory value. <b>Note:</b> Increase this options carefully!</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">multi</td>
        <td>
            true<br>
            false
        </td>
        <td>Enable multi word processing.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">ordered</td>
        <td>
            true<br>
            false
        </td>
        <td>Multiple words has to be the same order as the matched entry.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">strict</td>
        <td>
            true<br>
            false
        </td>
        <td>Matches exactly needs to be started with the query.</td>
    </tr>
    <tr></tr>
    <tr>
        <td align="top">cache</td>
        <td>
            true<br>
            false
        </td>
        <td>Enable caching.</td>
    </tr>
</table>

<a name="phonetic" id="phonetic"></a>
## Phonetic Encoding

<table>
    <tr>
        <th align="left">Option</th>
        <th align="left">Description</th>
        <th align="left">Example</th>
        <th align="left">False Positives</th>
        <th align="left">Compression Level</th>
    </tr>
    <tr></tr>
    <tr>
        <td><b>false</b></td>
        <td>Turn off encoding</td>
        <td>
            Reference: "Björn-Phillipp Mayer"<br>
            Matches: "Phil"
        </td>
        <td>no</td>
        <td>no</td>
    </tr>
    <tr></tr>
    <tr>
        <td><b>icase</b></td>
        <td>Case in-sensitive encoding</td>
        <td>
            Reference: "Björn-Phillipp Mayer"<br>
            Matches: "phil"
        </td>
        <td>no</td>
        <td>no</td>
    </tr>
    <tr></tr>
    <tr>
        <td><b>simple</b></td>
        <td>Phonetic normalizations</td>
        <td>
            Reference: "Björn-Phillipp Mayer"<br>
            Matches: "bjoern fillip"
        </td>
        <td>no</td>
        <td>~ 3%</td>
    </tr>
    <tr></tr>
    <tr>
        <td><b>advanced</b></td>
        <td>Phonetic normalizations + Literal transformations</td>
        <td>
            Reference: "Björn-Phillipp Mayer"<br>
            Matches: "filip meier"
        </td>
        <td>no</td>
        <td>~ 25%</td>
    </tr>
    <tr></tr>
    <tr>
        <td><b>extra</b></td>
        <td>Phonetic normalizations + Soundex transformations</td>
        <td>
            Reference: "Björn-Phillipp Mayer"<br>
            Matches: "byorn mair"
        </td>
        <td>yes</td>
        <td>~ 50%</td>
    </tr>
</table>

<a name="compare" id="compare"></a>
### Compare Phonetic Search Results

Reference String: __"Björn-Phillipp Mayer"__

<table>
    <tr>
        <th align="left">Query</th>
        <th align="left">ElasticSearch</th>
        <th align="left">FlexSearch (iCase)</th>
        <th align="left">FlexSearch (Simple)</th>
        <th align="left">FlexSearch (Adv.)</th>
        <th align="left">FlexSearch (Extra)</th>
    </tr>
    <tr></tr>
    <tr>
        <td>björn</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>björ</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>bjorn</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>bjoern</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>philipp</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>filip</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>björnphillip</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>meier</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>björn meier</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>meier fhilip</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
        <td><b>yes</b></td>
    </tr>
    <tr></tr>
    <tr>
        <td>byorn mair</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td>no</td>
        <td><b>yes</b></td>
    </tr>
    <tr>
        <td><i>(false positives)</i></td>
        <td>yes</td>
        <td><b>no</b></td>
        <td><b>no</b></td>
        <td><b>no</b></td>
        <td>yes</td>
    </tr>
</table>

<a name="memory" id="memory"></a>
## Improve Memory Usage

__Note:__ The data type of passed IDs has to be specified on creation. It is recommended to uses to most lowest possible data range here, e.g. use "short" when IDs are not higher than 65,535. 

<table>
    <tr>
        <th align="left">ID Type</th>
        <th align="left">Range of Values</th>
        <th align="left">Memory usage of every ~ 100,000 indexed words (Index + Content)</th>
    </tr>
    <tr></tr>
    <tr>
        <td>Byte</td>
        <td>0 - 255</td>
        <td>683 kb + 2.4 Mb</td>
    </tr>
    <tr></tr>
    <tr>
        <td>Short</td>
        <td>0 - 65,535</td>
        <td>1.3 Mb + 2.4 Mb</td>
    </tr>
    <tr></tr>
    <tr>
        <td>Integer</td>
        <td>0 - 4,294,967,295</td>
        <td>2.7 Mb + 2.4 Mb</td>
    </tr>
    <tr></tr>
    <tr>
        <td>Float</td>
        <td>0 - * (16 digits)</td>
        <td>5.3 Mb + 2.4 Mb</td>
    </tr>
    <tr></tr>
    <tr>
        <td>String</td>
        <td>* (unlimited)</td>
        <td>1.3 Mb <b>* <u>char length of IDs</u></b> + 2.4 Mb</td>
    </tr>
</table>

The required RAM per instance can be calculated as follow:

`BYTES = CONTENT_CHAR_COUNT * (BYTES_OF_ID + 2) + CONTENT_WORD_COUNT * 8`

__Note:__ Character count may be less related to the phonetic settings, e.g. when using "extra" the count of chars shrinks to ~ 50%.

---

Author FlexSearch: Thomas Wilkerling<br>
License: <a href="http://www.apache.org/licenses/LICENSE-2.0.html" target="_blank">Apache 2.0 License</a><br>
