Kouhei Sutou
null+****@clear*****
Thu Apr 14 19:01:03 JST 2016
Kouhei Sutou 2016-04-14 19:01:03 +0900 (Thu, 14 Apr 2016) New Revision: 7002bac4ab8dc9ead4e423de6a365e698c89ccf2 https://github.com/groonga/groonga/commit/7002bac4ab8dc9ead4e423de6a365e698c89ccf2 Message: doc groonga-suggest-httpd: updating... (not completed) groonga-suggest-httpd and groonga-suggest-learner may be buggy... Added files: doc/source/example/reference/executables/groonga-suggest-httpd-launch-lerner.log doc/source/example/reference/executables/groonga-suggest-httpd-learn.log doc/source/example/reference/executables/groonga-suggest-httpd-request.log Modified files: doc/source/example/reference/executables/groonga-suggest-httpd-launch.log doc/source/example/reference/executables/groonga-suggest-httpd-setup.log doc/source/reference/executables/groonga-suggest-httpd.rst Added: doc/source/example/reference/executables/groonga-suggest-httpd-launch-lerner.log (+4 -0) 100644 =================================================================== --- /dev/null +++ doc/source/example/reference/executables/groonga-suggest-httpd-launch-lerner.log 2016-04-14 19:01:03 +0900 (0f4dd88) @@ -0,0 +1,4 @@ +Execution example:: + + % groonga-suggest-learner --daemon ${DB_PATH} + Modified: doc/source/example/reference/executables/groonga-suggest-httpd-launch.log (+2 -1) =================================================================== --- doc/source/example/reference/executables/groonga-suggest-httpd-launch.log 2016-04-14 16:30:01 +0900 (8941510) +++ doc/source/example/reference/executables/groonga-suggest-httpd-launch.log 2016-04-14 19:01:03 +0900 (cd1b588) @@ -1,3 +1,4 @@ Execution example:: - % groonga-suggest-httpd /tmp/groonga-databases/groonga-suggest-httpd + % groonga-suggest-httpd --send-endpoint 'tcp://127.0.0.1:1234' --receive-endpoint 'tcp://127.0.0.1:1235' --daemon ${DB_PATH} + Added: doc/source/example/reference/executables/groonga-suggest-httpd-learn.log (+16 -0) 100644 =================================================================== --- /dev/null +++ doc/source/example/reference/executables/groonga-suggest-httpd-learn.log 2016-04-14 19:01:03 +0900 (3b90fa1) @@ -0,0 +1,16 @@ +Execution example:: + + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=92619&t=complete&q=G' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=93850&t=complete&q=Gr' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=94293&t=complete&q=Gro' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=94734&t=complete&q=Groo' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95147&t=complete&q=Grooon' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95553&t=complete&q=Groonga' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95959&t=submit&q=Groonga' + {} Added: doc/source/example/reference/executables/groonga-suggest-httpd-request.log (+16 -0) 100644 =================================================================== --- /dev/null +++ doc/source/example/reference/executables/groonga-suggest-httpd-request.log 2016-04-14 19:01:03 +0900 (3b90fa1) @@ -0,0 +1,16 @@ +Execution example:: + + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=92619&t=complete&q=G' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=93850&t=complete&q=Gr' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=94293&t=complete&q=Gro' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=94734&t=complete&q=Groo' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95147&t=complete&q=Grooon' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95553&t=complete&q=Groonga' + {} + % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95959&t=submit&q=Groonga' + {} Modified: doc/source/example/reference/executables/groonga-suggest-httpd-setup.log (+64 -1) =================================================================== --- doc/source/example/reference/executables/groonga-suggest-httpd-setup.log 2016-04-14 16:30:01 +0900 (88bc7f2) +++ doc/source/example/reference/executables/groonga-suggest-httpd-setup.log 2016-04-14 19:01:03 +0900 (c44505f) @@ -1,3 +1,66 @@ Execution example:: - % groonga-suggest-create-dataset /tmp/groonga-databases/groonga-suggest-httpd query + % groonga-suggest-create-dataset ${DB_PATH} query + > register suggest/suggest + true + > table_create event_type TABLE_HASH_KEY ShortText + true + > table_create bigram TABLE_PAT_KEY ShortText --default_tokenizer TokenBigram --normalizer NormalizerAuto + true + > table_create kana TABLE_PAT_KEY ShortText --normalizer NormalizerAuto + true + > table_create item_query TABLE_PAT_KEY ShortText --default_tokenizer TokenDelimit --normalizer NormalizerAuto + true + > column_create bigram item_query_key COLUMN_INDEX|WITH_POSITION item_query _key + true + > column_create item_query kana COLUMN_VECTOR kana + true + > column_create kana item_query_kana COLUMN_INDEX item_query kana + true + > column_create item_query freq COLUMN_SCALAR Int32 + true + > column_create item_query last COLUMN_SCALAR Time + true + > column_create item_query boost COLUMN_SCALAR Int32 + true + > column_create item_query freq2 COLUMN_SCALAR Int32 + true + > column_create item_query buzz COLUMN_SCALAR Int32 + true + > table_create pair_query TABLE_HASH_KEY UInt64 + true + > column_create pair_query pre COLUMN_SCALAR item_query + true + > column_create pair_query post COLUMN_SCALAR item_query + true + > column_create pair_query freq0 COLUMN_SCALAR Int32 + true + > column_create pair_query freq1 COLUMN_SCALAR Int32 + true + > column_create pair_query freq2 COLUMN_SCALAR Int32 + true + > column_create item_query co COLUMN_INDEX pair_query pre + true + > table_create sequence_query TABLE_HASH_KEY ShortText + true + > table_create event_query TABLE_NO_KEY + true + > column_create sequence_query events COLUMN_VECTOR|RING_BUFFER event_query + true + > column_create event_query type COLUMN_SCALAR event_type + true + > column_create event_query time COLUMN_SCALAR Time + true + > column_create event_query item COLUMN_SCALAR item_query + true + > column_create event_query sequence COLUMN_SCALAR sequence_query + true + > table_create configuration TABLE_HASH_KEY ShortText + true + > column_create configuration weight COLUMN_SCALAR UInt32 + true + > load --table configuration + > [ + > {"_key": "query", "weight": 1} + > ] + 1 Modified: doc/source/reference/executables/groonga-suggest-httpd.rst (+279 -94) =================================================================== --- doc/source/reference/executables/groonga-suggest-httpd.rst 2016-04-14 16:30:01 +0900 (29b5823) +++ doc/source/reference/executables/groonga-suggest-httpd.rst 2016-04-14 19:01:03 +0900 (07bd2a5) @@ -5,173 +5,334 @@ .. groonga-command .. database: groonga-suggest-httpd -groonga-suggest-httpd -===================== +``groonga-suggest-httpd`` +========================= Summary ------- -groonga-suggest-httpd is a program to provide interface which accepts HTTP request and returns suggestion dataset, then saves logs for learning. -groonga-suggest-httpd behaves similar in point of view of suggestion functionality, but the name of parameter is different. +``groonga-suggest-httpd`` is a program that provides HTTP interface +for the following features: + + * Returning :doc:`/reference/suggest` execution result + * Saving logs for learning + +``groonga-suggest-httpd`` provides suggest feature like +:doc:`/reference/commands/suggest` command. Note that some parameter +names are different of them. Syntax ------ ``groonga-suggest-httpd`` requires database path:: - groonga-suggest-httpd [options] database_path + groonga-suggest-httpd [options] DATABASE_PATH Usage ----- -Set up -^^^^^^ +You need to create one or more datasets to use +``groonga-suggest-httpd``. A dataset consists of tables and +columns. You can define them by :doc:`groonga-suggest-create-dataset`. + +You need to use :doc:`groonga-suggest-learner` to learn suggestion +data from user inputs. You doesn't need to use +:doc:`groonga-suggest-learner` when you create suggestion data by +hand. See :doc:`/reference/suggest` and sub documents about creating +suggestion data by hand. + +You can use ``groonga-suggest-httpd`` via HTTP after you create one or +more datasets. + +The following sections describes the followings: -First you need to set up database for suggestion. + * How to set up a dataset + * How to use ``groonga-suggest-httpd`` with :doc:`groonga-suggest-learner` + * How to use ``groonga-suggest-httpd`` for retrieving suggestions. + +Setup +^^^^^ + +You need to create a dataset by :doc:`groonga-suggest-create-dataset`. + +Here is an example that creates ``query`` dataset: .. groonga-command .. include:: ../../example/reference/executables/groonga-suggest-httpd-setup.log -.. .. % groonga-suggest-create-dataset /tmp/groonga-databases/groonga-suggest-httpd query +.. % groonga-suggest-create-dataset ${DB_PATH} query -Launch groonga-suggest-httpd -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +:doc:`groonga-suggest-create-dataset` outputs executed commands. You +can confirm that what tables and columns are created for the new +dataset. -Execute groonga-suggest-httpd command: +Launch ``groonga-suggest-learner`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. groonga-command -.. include:: ../../example/reference/executables/groonga-suggest-httpd-launch.log -.. .. % groonga-suggest-httpd /tmp/groonga-databases/groonga-suggest-httpd +You can choose whether you use learned suggestion data immediately or +not. -After executing above command, groonga-suggest-httpd accepts HTTP request on 8080 port. +There are two ways to use learned suggestion data immediately: -If you just want to save requests into log file, use ``-l`` option. + * Both of ``groonga-suggest-httpd`` and + :doc:`groonga-suggest-learner` use the same database + * ``groonga-suggest-httpd`` receives learned suggestion data from + :doc:`groonga-suggest-learner` -Here is the example to save log files under ``logs`` directory with ``log`` prefix for each file.:: +In the former case, you must run both ``groonga-suggest-httpd`` and +:doc:`groonga-suggest-learner` on the same host. - % groonga-suggest-httpd -l logs/log /tmp/groonga-databases/groonga-suggest-httpd +In the latter case, you can run ``groonga-suggest-httpd`` and +:doc:`groonga-suggest-learner` on different hosts. -Under ``logs`` directory, log files such as ``logYYYYmmddHHMMSS-00`` are created. +If you don't need to use learned suggestion data immediately, you need +to apply learned suggestion data from database that is used by +:doc:`groonga-suggest-learner` to database that is used by +``groonga-suggest-httpd`` by hand. Normally, this usage is +recommended. Because learned suggestion data may have garbage data by +inputs from evil users. -Request to groonga-suggest-httpd -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +In this document, learned suggestion data are used immediately by +receiving learned suggestion data from +:doc:`groonga-suggest-learner`. Both ``groonga-suggest-httpd`` and +:doc:`groonga-suggest-learner` are running on the same host. Because +it's easy to explain. -Here is the sample requests to learn ``groonga`` for ``query`` dataset:: +Here is an example that launches :doc:`groonga-suggest-learner`. You +need to specify database that has ``query`` dataset. This document +omits the instruction for creating ``query`` dataset: - % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=92619&t=complete&q=g' - % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=93850&t=complete&q=gr' - % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=94293&t=complete&q=gro' - % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=94734&t=complete&q=groo' - % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95147&t=complete&q=grooon' - % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95553&t=complete&q=groonga' - % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95959&t=submit&q=groonga +.. groonga-command +.. database: groonga-suggest-httpd-learner +.. % groonga-suggest-create-dataset ${DB_PATH} query -Options -------- +.. groonga-command +.. include:: ../../example/reference/executables/groonga-suggest-httpd-launch-lerner.log +.. % groonga-suggest-learner --daemon ${DB_PATH} + +The ``groonga-suggest-learner`` process opens two endpoints at +``1234`` port and ``1235`` port: + + * ``1234`` port: Endpoint that accepts user input data from + ``groogna-suggest-httpd`` + * ``1235`` port: Endpoint that sends learned suggestion data to + ``groogna-suggest-httpd`` + +Launch ``groonga-suggest-httpd`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. cmdoption:: -p, --port +You need to launch ``groonga-suggest-httpd`` for the following proposes: - Specify HTTP server port number. The default value is 8080. + * Learning suggestion data from user inputs + * Providing suggestion result to clients -.. cmdoption:: -t, --n-threads +Here is an example that launches ``groonga-suggest-httpd`` that +communicates :doc:`groonga-suggest-learner`: - Specify number of threads. The default value is the number of CPU cores. - This option accepts 128 as the max value, but use the number of CPU cores for - performance. +.. groonga-command +.. database: groonga-suggest-httpd -.. cmdoption:: -s, --send-endpoint +.. groonga-command +.. include:: ../../example/reference/executables/groonga-suggest-httpd-launch.log +.. % groonga-suggest-httpd --send-endpoint 'tcp://127.0.0.1:1234' --receive-endpoint 'tcp://127.0.0.1:1235' --daemon ${DB_PATH} - Specify endpoint for sender. +TODO: httpd and learner communication doesn't work... -.. cmdoption:: -r, --receive-endpoint +The ``groonga-suggest-httpd`` process accepts HTTP requests on +``8080`` port. - Specify endpoint for receiver. +If you want to save requests into log file, use +:option:`--log-base-path` option. -.. cmdoption:: -l, --log-base-path +Here is an example to save log files under ``logs`` directory with +``log`` prefix for each file:: - Specify path prefix of log. + % groonga-suggest-httpd --log-base-path logs/log ${DB_PATH} -.. cmdoption:: --n-lines-per-log-file +``groonga-suggest-httpd`` create log files such as +``logYYYYmmddHHMMSS-00`` under ``logs`` directory. - Specify the max number of lines in a log file. The default value is 1,000,000. +Learn from user inputs +^^^^^^^^^^^^^^^^^^^^^^ -.. cmdoption:: -d, --daemon +Here are example requests to learn user input "Groonga" in ``query`` +dataset:: - Specify this option to daemonize. +.. groonga-command +.. include:: ../../example/reference/executables/groonga-suggest-httpd-learn.log +.. % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=92619&t=complete&q=G' +.. % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=93850&t=complete&q=Gr' +.. % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=94293&t=complete&q=Gro' +.. % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=94734&t=complete&q=Groo' +.. % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95147&t=complete&q=Grooon' +.. % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95553&t=complete&q=Groonga' +.. % curl 'http://localhost:8080/?i=127.0.0.1&l=query&s=95959&t=submit&q=Groonga' + +Inputting data must use ``t=complete`` parameter. -.. cmdoption:: --disable-max-fd-check +Submitted data must use ``t=submit`` parameter. - Specify this option to disable checking max fd on start. +Use suggested response +^^^^^^^^^^^^^^^^^^^^^^ + +TODO Command line parameters ----------------------- -There is one required parameter - ``database_path``. +Required parameters +^^^^^^^^^^^^^^^^^^^ + +There is one required parameter. + +``DATABASE_PATH`` +""""""""""""""""" + +Specifies the path to a Groonga database. This database must have one +or more datasets. Each dataset must be created by +:doc:`groonga-suggest-create-dataset`. + +Optional parameters +^^^^^^^^^^^^^^^^^^^ + +.. option:: -p, --port + + Specify HTTP server port number. + + The default port number is ``8080``. -``database_path`` -^^^^^^^^^^^^^^^^^ +.. option:: -t, --n-threads -Specifies the path to a Groonga database. This database must be -created by :doc:`groonga-suggest-create-dataset` command because it -executes required initialization for suggestion. + Specify number of threads. + This option accepts ``128`` as the max value, but use the number of + CPU cores for performance. + + The default value is the number of CPU cores. + +.. option:: -s, --send-endpoint + + Specify endpoint URI of learner for sending learning data. + + The format is ``tcp://${HOST}:${PORT}`` such as + ``tcp://192.168.0.1:2929``. + + The default value is none. + +.. option:: -r, --receive-endpoint + + Specify endpoint URI of learner for receiving learning data. + + The format is ``tcp://${HOST}:${PORT}`` such as + ``tcp://192.168.0.1:2929``. + + The default value is none. + +.. option:: -l, --log-base-path + + Specify path prefix of log. + + The default value is none. + +.. option:: --n-lines-per-log-file + + Specify the max number of lines in a log file. + + The default value is ``1000000``. + +.. option:: -d, --daemon + + Specify this option to daemonize. + + Don't daemonize by default. + +.. option:: --disable-max-fd-check + + Specify this option to disable checking the max number of file + descriptors on start. + + Check by default. GET parameters -------------- -groonga-suggest-httpd accepts following GET parameters. +``groonga-suggest-httpd`` accepts some GET parameters. + +There are required parameters which depend on query type. -There are required parameters which depends on type of query. +In ``complete``, ``correct`` or ``suggest`` query type, unhandled +parameters are passed through :doc:`/reference/commands/suggest`. It +means that you can use parameters of +:doc:`/reference/commands/suggest`. Required parameters ^^^^^^^^^^^^^^^^^^^ +You must specify the following parameters. + .. list-table:: :header-rows: 1 * - Key - Description - Note - * - q - - UTF-8 encoded string which user fills in form - - - * - t - - The type of query. The value of type must be complete, correct, suggest or submit. It also accepts multiple type of query which is concatinated by ``|``. Note that ``submit`` is invalid value when you specify multiple type of query. - - + * - ``q`` + - Input by user. It must be UTF-8 encoded string. + - + * - ``t`` + - The query type. + + Available values are ``complete``, ``correct``, ``suggest`` and + ``submit``. + - You can specify multiple types except ``submit``. You need to + use ``|`` as separator such as ``complete|correct``. ``submit`` + is a valid value but ``complete|submit`` is invalid value. Required parameters for learning ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +You must specify the following parameters when you specify +:option:`--send-endpoint`. + .. list-table:: :header-rows: 1 * - Key - Description - Note - * - s - - Elapsed time from 0:00 January 1, 1970 - - Note that you need specify the value of ``s`` in milliseconds - * - i - - Unique ID to distinct user - - Use session ID or IP address for example - * - l - - Specify the name of dataset for learning. It also accepts multiple dataset name which is concatinated by ``|`` - - Note that dataset name must be matched to following regular expression ``[A-Za-z ][A-Za-z0-9 ]{0,15}`` + * - ``s`` + - Elapsed time since ``1970-01-01T00:00:00Z``. + - The unit is millisecond. + * - ``i`` + - Unique ID to distinct each user + - Session ID, IP address and so on will be usable + for this value. + * - ``l`` + - One or more learn target dataset names. You need to use ``|`` + as separator such as ``dataset1|dataset2|dataset3``. + - Dataset name is the name that you specify to + :doc:`groonga-suggest-create-dataset`. Required parameters for suggestion ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +You must specify the following parameters when you specify one of +``complete``, ``correct`` and ``suggest`` to ``t`` parameter. + .. list-table:: :header-rows: 1 * - Key - Description - Note - * - n - - Specify the name of dataset for suggestion - - This dataset name is used to calculate suggestion results + * - ``n`` + - The dataset name to use computing suggestion result + - Dataset name is the name that you specify to + :doc:`groonga-suggest-create-dataset`. + +Optional parameters +^^^^^^^^^^^^^^^^^^^ -Optional parameter -^^^^^^^^^^^^^^^^^^ +Here are optional parameters. .. list-table:: :header-rows: 1 @@ -179,33 +340,57 @@ Optional parameter * - Key - Description - Note - * - callback - - Specify the name of function if you prefer JSONP as response format - - The name of function must be matched to reqular expression ``[A-Za-z ][A-Za-z0-9 ]{0,15}`` + * - ``callback`` + - Function name for JSONP + - Return value ------------ -``groonga-suggest-httpd`` command returns following response in JSON or JSONP format. - -In JSON format:: - - {TYPE: [[CANDIDATE_1, SCORE_1], [CANDIDATE_2, SCORE_2], ... [CANDIDATE_N, SCORE_N]]} - -In JSONP format:: - - FUNCTION({TYPE: [[CANDIDATE_1, SCORE_1], [CANDIDATE_2, SCORE_2], ... [CANDIDATE_N, SCORE_N]]}) - +``groonga-suggest-httpd`` returns the following format response. It's +the same format as body of :doc:`/reference/commands/suggest`:: + + { + TYPE1: [ + [CANDIDATE_1, SCORE_1], + [CANDIDATE_2, SCORE_2], + ..., + [CANDIDATE_N, SCORE_N] + ], + TYPE2: [ + [CANDIDATE_1, SCORE_1], + [CANDIDATE_2, SCORE_2], + ..., + [CANDIDATE_N, SCORE_N] + ], + ... + } + +Here is the response when ``t`` is ``submit``:: + + {} ``TYPE`` +^^^^^^^^ - One of ``complete``, ``correct`` and ``suggest``. +One of ``complete``, ``correct`` and ``suggest``. ``CANDIDATE_N`` +^^^^^^^^^^^^^^^ - The string of candidate (UTF-8). +The string of candidate in UTF-8. ``SCORE_N`` +^^^^^^^^^^^ - The score. +The score of the candidate. +Candidates are sorted by score descendant. + +See also +-------- + + * :doc:`/reference/commands/suggest` + +.. groonga-command +.. % killall lt-groonga-suggest-learner lt-groonga-suggest-httpd -------------- next part -------------- HTML����������������������������...다운로드
