1. 04 Aug, 2024 1 commit
  2. 26 Jul, 2024 39 commits
    • Vicențiu Ciorbaru's avatar
      Fix windows · bcbb8cbc
      Vicențiu Ciorbaru authored
      bcbb8cbc
    • Sergei Golubchik's avatar
      windows ugh · dc92e9e2
      Sergei Golubchik authored
      dc92e9e2
    • Sergei Golubchik's avatar
      an attempt to fix arm · cc7e5782
      Sergei Golubchik authored
      cc7e5782
    • Sergei Golubchik's avatar
      buildbot · 974620fc
      Sergei Golubchik authored
      failures in --view, --embedded, an old/non-gcc compilers
      974620fc
    • Sergei Golubchik's avatar
      set EXTRA_VERSION=-vector · 62e035fb
      Sergei Golubchik authored
      62e035fb
    • Sergei Golubchik's avatar
      d6e8d536
    • Sergei Golubchik's avatar
      fix for rename · 852b9ee2
      Sergei Golubchik authored
      852b9ee2
    • Sergey Vojtovich's avatar
      VECTOR indexes support for RENAME TABLE · aef2e8f1
      Sergey Vojtovich authored
      Rename high-level indexes along with a table.
      aef2e8f1
    • Sergei Golubchik's avatar
    • Sergey Vojtovich's avatar
      Fixed TRUNCATE TABLE against VECTOR indexes · 0c45567e
      Sergey Vojtovich authored
      This patch fixes only TRUNCATE by recreate variant, there seem to be no
      reasonable engine that uses TRUNCATE by handler method for testing.
      
      Reset index_cinfo so that mi_create is not confused by garbage passed via
      index_file_name and sets MY_DELETE_OLD flag.
      
      Review question: can we add a test case to make sure VECTOR index is empty
      indeed?
      0c45567e
    • Sergey Vojtovich's avatar
      d1849742
    • Sergey Vojtovich's avatar
      a45e742f
    • Sergey Vojtovich's avatar
      f205a7f1
    • Vicențiu Ciorbaru's avatar
      MDEV-32886 Vec_FromText and Vec_ToText · 1dce1222
      Vicențiu Ciorbaru authored
      This commit introduces two utility functions meant to make working with
      vectors simpler.
      
      Vec_ToText converts a binary vector into a json array of numbers
      (floats).
      Vec_FromText takes in a json array of numbers and converts it into a
      little-endian IEEE float sequence of bytes (4 bytes per float).
      1dce1222
    • Vicențiu Ciorbaru's avatar
      Introduce String::append_float · e55f8288
      Vicențiu Ciorbaru authored
      This method will write out a float to a String object, keeping the
      charset of the original string.
      
      Also have Float::to_string make use of String::append_float
      e55f8288
    • Sergei Golubchik's avatar
      mhnsw: make the search less greedy · b4e999b5
      Sergei Golubchik authored
      introduced a generosity factor that makes the search less greedy.
      it dramatically improves the recall by making the search a bit slower
      (for the same recall one can use half the M and smaller ef).
      
      had to add Queue::safe_push() method that removes one of the
      furthest elements (not necessarily the furthest) in the queue
      to keep it from overflowing.
      b4e999b5
    • Sergei Golubchik's avatar
      mhnsw: auto-tune efConstruction · d93f6633
      Sergei Golubchik authored
      * remove hard-coded ef_construction_multiplier
      * instead, let ef_construction go up and down automatically as needed
      * as needed means that expanding the queue changes the result much
      * much is defined by the queue stiffness, as in Hooke's law
      * also search_layer() now returns only as many elements as needed, the
        caller no longer needs to overallocate result arrays for throwaway nodes
      * change _downheap() to return the position where the element ended up
      d93f6633
    • Sergei Golubchik's avatar
      cleanup: FVector · a6c88428
      Sergei Golubchik authored
      make FVector great again
      a6c88428
    • Sergei Golubchik's avatar
      mhnsw: store coordinates in 16 bits, not 32 · 1bca1fc5
      Sergei Golubchik authored
      use int16_t instead of floats, they're faster and smaller.
      but perform intermediate SIMD calculations with floats to avoid overflows.
      recall drop with such scheme is below 0.002, often none.
      
      int8_t would've been better but the precision loss is too big
      and recall degrades too much.
      1bca1fc5
    • Sergei Golubchik's avatar
      UPDATE/DELETE post-fixes · d029a2e0
      Sergei Golubchik authored
      d029a2e0
    • Sergei Golubchik's avatar
      cleanup: prepare_for_insert() -> prepare_for_modify() · 95a59fee
      Sergei Golubchik authored
      make handler::prepare_for_insert() to be called to prepare
      the handler for writes, INSERT/UPDATE/DELETE.
      95a59fee
    • Hugo Wen's avatar
      MDEV-33408 Initial support for vector DELETE and UPDATE · 9de28556
      Hugo Wen authored
      When the source row is deleted, mark the corresponding node in HNSW
      index by setting `tref` to null. An index is added for the `tref` in
      secondary table for faster searching of the to-be-marked nodes.
      
      The nodes marked as deleted will still be used for search, but will not
      be included in the final query results.
      
      As skipping deleted nodes and not adding deleted nodes for new-inserted
      nodes' neighbor list could impact the performance, we now only skip
      these nodes in search results.
      
      - for some reason the bitmap is not set for hlindex during the delete so
        I had to temporarily comment out one line
      
      All new code of the whole pull request, including one or several files
      that are either new files or modified ones, are contributed under the
      BSD-new license. I am contributing on behalf of my employer Amazon Web
      Services, Inc.
      9de28556
    • Sergei Golubchik's avatar
      non-SIMD fallback · 11a37311
      Sergei Golubchik authored
      11a37311
    • Sergei Golubchik's avatar
      mhnsw: inter-statement shared cache · 41334e56
      Sergei Golubchik authored
      * preserve the graph in memory between statements
      * keep it in a TABLE_SHARE, available for concurrent searches
      * nodes are generally read-only, walking the graph doesn't change them
      * distance to target is cached, calculated only once
      * SIMD-optimized bloom filter detects visited nodes
      * nodes are stored in an array, not List, to better utilize bloom filter
      * auto-adjusting heuristic to estimate the number of visited nodes
        (to configure the bloom filter)
      * many threads can concurrently walk the graph. MEM_ROOT and Hash_set
        are protected with a mutex, but walking doesn't need them
      * up to 8 threads can concurrently load nodes into the cache,
        nodes are partitioned into 8 mutexes (8 is chosen arbitrarily, might
        need tuning)
      * concurrent editing is not supported though
      * this is fine for MyISAM, TL_WRITE protects the TABLE_SHARE and the
        graph (note that TL_WRITE_CONCURRENT_INSERT is not allowed, because an
        INSERT into the main table means multiple UPDATEs in the graph)
      * InnoDB uses secondary transaction-level caches linked in a list in
        in thd->ha_data via a fake handlerton
      * on rollback the secondary cache is discarded, on commit nodes
        from the secondary cache are invalidated in the shared cache
        while it is exclusively locked
      * on savepoint rollback both caches are flushed. this can be improved
        in the future with a row visibility callback
      * graph size is controlled by @@mhnsw_cache_size, the cache is flushed
        when it reaches the threshold
      41334e56
    • Sergei Golubchik's avatar
      mhnsw: change storage format · 656845ef
      Sergei Golubchik authored
      instead of one row per node per layer, have one row per node.
      store all neighbors for all layers in that row, and the vector itself too
      
      it completely avoids searches in the graph table and
      will allow to implement deletions in the future
      656845ef
    • Sergei Golubchik's avatar
      mhnsw: closest neighbor precalc heuristic · 5147ca6f
      Sergei Golubchik authored
      This is based on the heuristic  that if a candidate neighbor
      has a very close neighbor of its own, than this close neighbor
      is also likely a candidate neighbor itself.
      
      Meaning, we might replace the loop that compares a candidate
      with all neighbors if we know the distance between the candidate
      and its closest neighbor. Which can be precalculated.
      
      This gives the most speedup when the number of neighbors
      and the number of dimensions are large. In the tests it was
      2.5-3x speedup, with the recall being worse by 0.1%-1%
      
      Incidentally, in the opposite case it gives both litle speedup
      and notably worse recall. Tests have shown 1.13x speedup
      with recall going down by ~20% in the worst - smallest - case.
      
      Thus, this heuristic is only enabled above the certain threshold.
      5147ca6f
    • Sergei Golubchik's avatar
      1165e6a6
    • Sergei Golubchik's avatar
      mhnsw: SIMD for euclidean distance · 86a3ac12
      Sergei Golubchik authored
      86a3ac12
    • Sergei Golubchik's avatar
      mhnsw: configurable parameters · 1249210b
      Sergei Golubchik authored
      1. introduce alpha. the value of 1.1 is optimal, so hard-code it.
      
      2. define ef and efConstruction in terms of limit and M, that is,
         ef = ef_limit_multiplier * limit
         efConstruction = ef_construction_multiplier * M
      
      3. ef_construction_multiplier=4 is almost always optimal, so
         hard-code it too
      
      4. rename hnsw_max_connection_per_layer to mhnsw_max_edges_per_node
         (max_connection is rather ambiguous in MariaDB) and add a help text
      
      5. rename hnsw_ef_search to mhnsw_limit_multiplier and add a help text
      1249210b
    • Sergei Golubchik's avatar
      InnoDB support for hlindexes and mhnsw · a91bc7b7
      Sergei Golubchik authored
      * mhnsw:
        * use primary key, innodb loves and (and the index cannot have dupes anyway)
          * MyISAM is ok with that, performance-wise
        * must be ha_rnd_init(0) because we aren't going to scan
          * MyISAM resets the position on ha_rnd_init(0) so query it before
          * oh, and use the correct handler, just in case
        * HA_ERR_RECORD_IS_THE_SAME is no error
      * innodb:
        * return ref_length on create
        * don't assume table->pos_in_table_list is set
        * ok, assume away, but only for system versioned tables
      * set alter_info on create (InnoDB needs to check for FKs)
      * pair external_lock/external_unlock correctly
      a91bc7b7
    • Sergei Golubchik's avatar
    • Sergei Golubchik's avatar
      bugfix: properly reset db_plugin when hlindex discovery fails · fd80cb58
      Sergei Golubchik authored
      otherwise it'll be free'd twice
      fd80cb58
    • Sergei Golubchik's avatar
    • Sergei Golubchik's avatar
      cleanups · f43f9204
      Sergei Golubchik authored
      f43f9204
    • Sergei Golubchik's avatar
    • Sergei Golubchik's avatar
      mhnsw: search intermediate layers with ef=1 · c532a0c1
      Sergei Golubchik authored
      also add missing candidates.empty();
      c532a0c1
    • Sergei Golubchik's avatar
      03239fd1
    • Sergei Golubchik's avatar
      5dab4964
    • Sergei Golubchik's avatar
      mhnsw: don't create many empty layers · 696d2705
      Sergei Golubchik authored
      696d2705