1. 17 Mar, 2021 1 commit
    • Dylan Griffith's avatar
      Improve performance of Elasticsearch notes permissions migration · 9abb6fb5
      Dylan Griffith authored
      We are currently running this migration in production to reindex all the
      notes in GitLab.com. There happen to be over 30M of them so it will take
      quite a while to run. We have found in
      https://gitlab.com/gitlab-org/gitlab/-/issues/324745 that the migration
      is not performing as well as we'd hoped. Given that we want to keep each
      job taking less than 1 minute to run we want to reduce the batch size to
      make that easier.
      
      We have found that 9000 per 3 minutes is too much and we are regularly
      overlapping with the previous batch and thus we end up often processing
      the same batch twice.
      
      In addition it will be more efficient to push the work into the "initial
      indexing queue" instead of our main "incremental indexing queue" as the
      initial indexing queue has far less traffic (it only covers newly
      created projects while incremental is any record update in any project).
      9abb6fb5
  2. 16 Mar, 2021 39 commits