• Yorick Peterse's avatar
    Rewrite SnippetsFinder to improve performance · d171ff60
    Yorick Peterse authored
    This completely rewrites the SnippetsFinder class from the ground up in
    order to improve its performance. The old code was beyond salvaging. It
    was complex, included various Rails 5 workarounds, comments that
    shouldn't be necessary, and most important of all: it produced a really
    poorly performing database query.
    
    As a result, I opted for rewriting the finder from scratch, instead of
    trying to patch the existing code. Instead of trying to reuse as many
    existing methods as possible, I opted for defining new methods
    specifically meant for the SnippetsFinder. This requires some extra code
    here and there, but allows us to have much more control over the
    resulting SQL queries. It is these changes that then allow us to produce
    a _much_ more efficient query.
    
    To illustrate how bad the old query was, we will use my own snippets as
    an example. Currently I have 52 snippets, most of which are global ones.
    To retrieve these, you would run the following Ruby code:
    
        user = User.find_by(username: 'yorickpeterse')
    
        SnippetsFinder.new(user, author: user).execute
    
    On GitLab.com the resulting query will take between 10 and 15 seconds to
    run, producing the query plan found at
    https://explain.depesz.com/s/Y5IX. Apart from the long execution time,
    the total number of buffers (the sum of all shared hits) is around 185
    GB, though the real number is probably (hopefully) much lower as I doubt
    simply summing these numbers produces the true total number of buffers
    used.
    
    The new query's plan can be found at https://explain.depesz.com/s/wHdN,
    and this query takes between 10 and 100-ish milliseconds to run. The
    total number of buffers used is only about 30 MB.
    
    Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/52639
    d171ff60
snippets_finder_spec.rb 6.65 KB