<!DOCTYPE html>
<html>
  <head>
  </head>
  <body>
    <dl style="line-height: 1.5; margin-left: 2em">
      <dt style="clear: both; float: left; font-weight: bold; width: 8em">Author</dt>
      <dd style="margin-left: 8.5em">Kouhei Sutou &lt;kou****@clear*****&gt;</dd>
      <dt style="clear: both; float: left; font-weight: bold; width: 8em">Date</dt>
      <dd style="margin-left: 8.5em">2018-09-20 17:15:55 +0900 (Thu, 20 Sep 2018)</dd>
      <dt style="clear: both; float: left; font-weight: bold; width: 8em">Revision</dt>
      <dd style="margin-left: 8.5em"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1">95f8fd3600bc16c640808bb30c9dd49cae23ded1</a></dd>
      <dt style="clear: both; float: left; font-weight: bold; width: 8em">Message</dt>
      <dd style="margin-left: 8.5em"><pre style="border: 1px solid #aaa; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; padding: 0.5em; width: auto">TokenNgram: add unify_digit option

TokenNgram(&quot;unify_digit&quot;, false) == TokenBigramSplitDigit (not exist ;p)</pre></dd>
      <dt style="clear: both; float: left; font-weight: bold; width: 8em">Added files</dt>
      <dd style="margin-left: 8.5em">
        <ul>
          <li><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366">test/command/suite/tokenizers/ngram/unify_digit.expected</a></li>
          <li><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-537bb19e93f3d4eb2c7baafe2e0eb733">test/command/suite/tokenizers/ngram/unify_digit.test</a></li>
        </ul>
      </dd>
      <dt style="clear: both; float: left; font-weight: bold; width: 8em">Modified files</dt>
      <dd style="margin-left: 8.5em">
        <ul>
          <li><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8e">lib/tokenizers.c</a></li>
        </ul>
      </dd>
    </dl>

    <div class="diff-section" style="clear: both">
      <table style="border-collapse: collapse; border: 1px solid #aaa">
        <thead>
          <tr class="diff-header" style="border: 1px solid #aaa">
            <td colspan="3">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-header" style="background-color: #eaf2f5; color: #999999; display: block; white-space: pre">  Modified: lib/tokenizers.c (+6 -0)</span>
<span class="diff-header-mark" style="background-color: #eaf2f5; color: #999999; display: block; white-space: pre">===================================================================</span>
</pre>
            </td>
          </tr>
        </thead>
        <tbody>
          <tr>
            <th class="diff-line-number" style="border: 1px solid #aaa">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-line-number-hunk-header" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eL735">...</a></span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eL736">736</a></span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eL737">737</a></span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eL738">738</a></span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eL739">739</a></span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eL740">740</a></span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eL741">741</a></span>
</pre>
            </th>
            <th class="diff-line-number" style="border: 1px solid #aaa">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-line-number-hunk-header" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR735">...</a></span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR736">736</a></span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR737">737</a></span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR738">738</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR739">739</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR740">740</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR741">741</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR742">742</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR743">743</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR744">744</a></span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR745">745</a></span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR746">746</a></span>
<span class="diff-line-number-not-changed" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-bb0ba8ce6a2347aa780a747cf6452b8eR747">747</a></span>
</pre>
            </th>
            <td class="diff-content" style="border: 1px solid #aaa">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-hunk-header" style="background-color: #eaf2f5; color: #999999; display: block; white-space: pre">@@ -736,6 +736,12 @@ <span class="diff-context" style="background-color: #ffffaa; color: #000000">ngram_open_options(grn_ctx *ctx,</span></span>
<span class="diff-not-changed" style="display: block; white-space: pre">                                     raw_options,</span>
<span class="diff-not-changed" style="display: block; white-space: pre">                                     i,</span>
<span class="diff-not-changed" style="display: block; white-space: pre">                                     options-&gt;unify_alphabet);</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    } else if (GRN_RAW_STRING_EQUAL_CSTRING(name_raw, &quot;unify_digit&quot;)) {</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      options-&gt;unify_digit =</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+        grn_vector_get_element_bool(ctx,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+                                    raw_options,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+                                    i,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+                                    options-&gt;unify_digit);</span>
<span class="diff-not-changed" style="display: block; white-space: pre">     } else if (GRN_RAW_STRING_EQUAL_CSTRING(name_raw, &quot;unify_symbol&quot;)) {</span>
<span class="diff-not-changed" style="display: block; white-space: pre">       options-&gt;unify_symbol =</span>
<span class="diff-not-changed" style="display: block; white-space: pre">         grn_vector_get_element_bool(ctx,</span>
</pre>
            </td>
          </tr>
        </tbody>
      </table>
      <table style="border-collapse: collapse; border: 1px solid #aaa">
        <thead>
          <tr class="diff-header" style="border: 1px solid #aaa">
            <td colspan="3">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-header" style="background-color: #eaf2f5; color: #999999; display: block; white-space: pre">  Added: test/command/suite/tokenizers/ngram/unify_digit.expected (+40 -0) 100644</span>
<span class="diff-header-mark" style="background-color: #eaf2f5; color: #999999; display: block; white-space: pre">===================================================================</span>
</pre>
            </td>
          </tr>
        </thead>
        <tbody>
          <tr>
            <th class="diff-line-number" style="border: 1px solid #aaa">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-line-number-hunk-header" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366">...</a></span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
</pre>
            </th>
            <th class="diff-line-number" style="border: 1px solid #aaa">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-line-number-hunk-header" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366">...</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R1">1</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R2">2</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R3">3</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R4">4</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R5">5</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R6">6</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R7">7</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R8">8</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R9">9</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R10">10</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R11">11</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R12">12</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R13">13</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R14">14</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R15">15</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R16">16</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R17">17</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R18">18</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R19">19</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R20">20</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R21">21</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R22">22</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R23">23</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R24">24</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R25">25</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R26">26</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R27">27</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R28">28</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R29">29</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R30">30</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R31">31</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R32">32</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R33">33</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R34">34</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R35">35</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R36">36</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R37">37</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R38">38</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R39">39</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-06cad7655cc50c0d774e5b4bd5860366R40">40</a></span>
</pre>
            </th>
            <td class="diff-content" style="border: 1px solid #aaa">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-hunk-header" style="background-color: #eaf2f5; color: #999999; display: block; white-space: pre">@@ -0,0 +1,40 @@</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+tokenize   &#39;TokenNgram(&quot;unify_digit&quot;, false)&#39;   &quot;012345&quot;   NormalizerAuto</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+[</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+  [</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    0,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    0.0,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    0.0</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+  ],</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+  [</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    {</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;value&quot;: &quot;01&quot;,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;position&quot;: 0,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;force_prefix&quot;: false</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    },</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    {</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;value&quot;: &quot;12&quot;,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;position&quot;: 1,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;force_prefix&quot;: false</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    },</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    {</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;value&quot;: &quot;23&quot;,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;position&quot;: 2,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;force_prefix&quot;: false</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    },</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    {</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;value&quot;: &quot;34&quot;,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;position&quot;: 3,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;force_prefix&quot;: false</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    },</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    {</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;value&quot;: &quot;45&quot;,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;position&quot;: 4,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;force_prefix&quot;: false</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    },</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    {</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;value&quot;: &quot;5&quot;,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;position&quot;: 5,</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+      &quot;force_prefix&quot;: false</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+    }</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+  ]</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+]</span>
</pre>
            </td>
          </tr>
        </tbody>
      </table>
      <table style="border-collapse: collapse; border: 1px solid #aaa">
        <thead>
          <tr class="diff-header" style="border: 1px solid #aaa">
            <td colspan="3">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-header" style="background-color: #eaf2f5; color: #999999; display: block; white-space: pre">  Added: test/command/suite/tokenizers/ngram/unify_digit.test (+4 -0) 100644</span>
<span class="diff-header-mark" style="background-color: #eaf2f5; color: #999999; display: block; white-space: pre">===================================================================</span>
</pre>
            </td>
          </tr>
        </thead>
        <tbody>
          <tr>
            <th class="diff-line-number" style="border: 1px solid #aaa">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-line-number-hunk-header" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-537bb19e93f3d4eb2c7baafe2e0eb733">...</a></span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
<span class="diff-line-number-nothing" style="display: block; white-space: pre">&nbsp;</span>
</pre>
            </th>
            <th class="diff-line-number" style="border: 1px solid #aaa">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-line-number-hunk-header" style="display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-537bb19e93f3d4eb2c7baafe2e0eb733">...</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-537bb19e93f3d4eb2c7baafe2e0eb733R1">1</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-537bb19e93f3d4eb2c7baafe2e0eb733R2">2</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-537bb19e93f3d4eb2c7baafe2e0eb733R3">3</a></span>
<span class="diff-line-number-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre"><a href="https://github.com/groonga/groonga/commit/95f8fd3600bc16c640808bb30c9dd49cae23ded1#diff-537bb19e93f3d4eb2c7baafe2e0eb733R4">4</a></span>
</pre>
            </th>
            <td class="diff-content" style="border: 1px solid #aaa">
<pre style="border: 0; font-family: Consolas, Menlo, &quot;Liberation Mono&quot;, Courier, monospace; line-height: 1.2; margin: 0; padding: 0.5em; white-space: normal; width: auto"><span class="diff-hunk-header" style="background-color: #eaf2f5; color: #999999; display: block; white-space: pre">@@ -0,0 +1,4 @@</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+tokenize \</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+  &#39;TokenNgram(&quot;unify_digit&quot;, false)&#39; \</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+  &quot;012345&quot; \</span>
<span class="diff-added" style="background-color: #eaffea; color: #000000; display: block; white-space: pre">+  NormalizerAuto</span>
</pre>
            </td>
          </tr>
        </tbody>
      </table>
    </div>
  </body>
</html>