An error occurred fetching the project authors.
  1. 14 Sep, 2024 5 commits
    • Sergei Golubchik's avatar
      rename VEC_Distance to VEC_Distance_Euclidean · 919e024b
      Sergei Golubchik authored
      and create a parent Item_func_vec_distance_common class
      919e024b
    • Vicențiu Ciorbaru's avatar
      MDEV-32886 Vec_FromText and Vec_ToText · 097a7ea1
      Vicențiu Ciorbaru authored
      This commit introduces two utility functions meant to make working with
      vectors simpler.
      
      Vec_ToText converts a binary vector into a json array of numbers
      (floats).
      Vec_FromText takes in a json array of numbers and converts it into a
      little-endian IEEE float sequence of bytes (4 bytes per float).
      097a7ea1
    • Sergei Golubchik's avatar
      misc changes · 3407e355
      Sergei Golubchik authored
      * sysvars should be REQUIRED_ARG
      * fix a mix of US and UK spelling (use US)
      * use consistent naming
      * work if VEC_DISTANCE arguments are in the swapped order (const, col)
      * work if VEC_DISTANCE argument is NULL/invalid or wrong length
      * abort INSERT if the value is invalid or wrong length
      * store the "number of neighbors" in a blob in endianness-independent way
      * use field->store(longlong, bool) not field->store(double)
      * a lot more error checking everywhere
      * cleanup after errors
      * simplify calling conventions, remove reinterpret_cast's
      * todo/XXX comments
      * whitespaces
      * use float consistently
      
      memory management is still totally PoC quality
      
      Initial HNSW implementation
      3407e355
    • Vicențiu Ciorbaru's avatar
      Initial HNSW implementation · 16b55b46
      Vicențiu Ciorbaru authored
      This commit includes the work done in collaboration with Hugo Wen from
      Amazon:
      
          MDEV-33408 Alter HNSW graph storage and fix memory leak
      
          This commit changes the way HNSW graph information is stored in the
          second table. Instead of storing connections as separate records, it now
          stores neighbors for each node, leading to significant performance
          improvements and storage savings.
      
          Comparing with the previous approach, the insert speed is 5 times faster,
          search speed improves by 23%, and storage usage is reduced by 73%, based
          on ann-benchmark tests with random-xs-20-euclidean and
          random-s-100-euclidean datasets.
      
          Additionally, in previous code, vector objects were not released after
          use, resulting in excessive memory consumption (over 20GB for building
          the index with 90,000 records), preventing tests with large datasets.
          Now ensure that vectors are released appropriately during the insert and
          search functions. Note there are still some vectors that need to be
          cleaned up after search query completion. Needs to be addressed in a
          future commit.
      
          All new code of the whole pull request, including one or several files
          that are either new files or modified ones, are contributed under the
          BSD-new license. I am contributing on behalf of my employer Amazon Web
          Services, Inc.
      
      As well as the commit:
      
          Introduce session variables to manage HNSW index parameters
      
          Three variables:
      
          hnsw_max_connection_per_layer
          hnsw_ef_constructor
          hnsw_ef_search
      
          ann-benchmark tool is also updated to support these variables in commit
          https://github.com/HugoWenTD/ann-benchmarks/commit/e09784e for branch
          https://github.com/HugoWenTD/ann-benchmarks/tree/mariadb-configurable
      
          All new code of the whole pull request, including one or several files
          that are either new files or modified ones, are contributed under the
          BSD-new license. I am contributing on behalf of my employer Amazon Web
          Services, Inc.
      Co-authored-by: default avatarHugo Wen <wenhug@amazon.com>
      16b55b46
    • Sergei Golubchik's avatar
      initial support for vector indexes · 18ed16b5
      Sergei Golubchik authored
      MDEV-33407 Parser support for vector indexes
      
      The syntax is
      
        create table t1 (... vector index (v) ...);
      
      limitation:
      * v is a binary string and NOT NULL
      * only one vector index per table
      * temporary tables are not supported
      
      MDEV-33404 Engine-independent indexes: subtable method
      
      added support for so-called "high level indexes", they are not visible
      to the storage engine, implemented on the sql level. For every such
      an index in a table, say, t1, the server implicitly creates a second
      table named, like, t1#i#05 (where "05" is the index number in t1).
      This table has a fixed structure, no frm, not accessible directly,
      doesn't go into the table cache, needs no MDLs.
      
      MDEV-33406 basic optimizer support for k-NN searches
      
      for a query like SELECT ... ORDER BY func() optimizer will use
      item_func->part_of_sortkey() to decide what keys can be used
      to resolve ORDER BY.
      18ed16b5
  2. 11 Sep, 2024 1 commit