Path: csiph.com!newsfeed.xs4all.nl!newsfeed8.news.xs4all.nl!bofh.it!news.nic.it!robomod From: Edward Betts Newsgroups: linux.debian.bugs.dist,linux.debian.devel,linux.debian.maint.python Subject: Bug#993386: ITP: sqlite-fts4 -- Document scoring Python library for SQLite FTS4 Date: Tue, 31 Aug 2021 18:10:01 +0200 Message-ID: X-Mailbox-Line: From debian-bugs-dist-request@lists.debian.org Tue Aug 31 16:03:08 2021 Old-Return-Path: X-Spam-Flag: NO X-Spam-Score: -3.95 Reply-To: Edward Betts , 993386@bugs.debian.org Resent-To: debian-bugs-dist@lists.debian.org Resent-Cc: debian-devel@lists.debian.org, debian-python@lists.debian.org, wnpp@debian.org X-Debian-Pr-Message: report 993386 X-Debian-Pr-Package: wnpp Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Mailer: reportbug 11.0.0 X-Greylist: delayed 1184 seconds by postgrey-1.36 at buxtehude; Tue, 31 Aug 2021 15:58:45 UTC X-Debian-Message: from BTS X-Mailing-List: archive/latest/1676091 List-ID: List-URL: Approved: robomod@news.nic.it Lines: 47 Organization: linux.* mail to news gateway Sender: robomod@news.nic.it X-Original-Date: Tue, 31 Aug 2021 16:38:53 +0100 X-Original-Message-ID: <163042433322.3870366.535276390980582658.reportbug@4angle.com> Xref: csiph.com linux.debian.bugs.dist:1069178 linux.debian.devel:101474 linux.debian.maint.python:13498 Package: wnpp Severity: wishlist Owner: Edward Betts X-Debbugs-Cc: debian-devel@lists.debian.org, debian-python@lists.debian.org * Package name : sqlite-fts4 Version : 1.0.1 Upstream Author : Simon Willison * URL : https://github.com/simonw/sqlite-fts4 * License : Apache 2.0 Programming Lang: Python Description : Document scoring Python library for SQLite FTS4 Custom SQLite functions written in Python for ranking documents indexed using the FTS4 extension. ## rank_score() This is an extremely simple ranking function, based on an example in the SQLite documentation. It generates a score for each document using the sum of the score for each column. The score for each column is calculated as the number of search matches in that column divided by the number of search matches for every column in the index - a classic TF-IDF calculation. ## rank_bm25() An implementation of the Okapi BM25 scoring algorithm. ## decode_matchinfo() SQLite's built-in matchinfo() function returns results as a binary string. This binary represents a list of 32 bit unsigned integers, but reading the binary results is not particularly human-friendly. ## annotate_matchinfo() This function decodes the matchinfo document into a verbose JSON structure that describes exactly what each of the returned integers actually means. Blog post about the creation of this library: https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/ This is a dependency of the sqlite-utils tool by the same author. I plan to maintain this package as part of the python modules team.