An error occurred fetching the project authors.
  1. 02 Aug, 2016 6 commits
  2. 19 Jul, 2016 1 commit
    • Jérome Perrin's avatar
      gitlab: enable parameters-extra options when creating wrapper · 6082d6e9
      Jérome Perrin authored
      @jerome says at nexedi/slapos@5f5d5102 (comment 17119):
      
      before f4e51f77, we had:
      `~/srv/runner/instance/slappart0/bin/gitlab-rake` containing:
          ```python
          ...
          if __name__ == '__main__':
              sys.exit(slapos.recipe.librecipe.execute.generic_exec((['/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/bin/bundle', 'exec', 'sh', '-c', 'cd /srv/slapgrid/slappart16/srv/runner/instance/slappart0/gitlab-work && rake "$@"', 'rake'], None, {'BUNDLE_GEMFILE': '/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/parts/gitlab/Gemfile', 'HOME': '/srv/slapgrid/slappart16/srv/runner/instance/slappart0', 'SIDEKIQ_MEMORY_KILLER_MAX_RSS': '1000000', 'RAILS_ENV': 'production'})))
          ```
      
      after, `~/srv/runner/instance/slappart0/bin/gitlab-rake` contains:
          ```shell
          #!/bin/bash
          COMMAND=/srv/slapgrid/slappart16/srv/runner/instance/slappart0/bin/gitlab-rake.py
      
          # If the wrapped command uses a shebang, execute the referenced
          # executable passing the script path as first argument.
          # This is to workaround the limitation of 127 characters in #!
          if [[ -f $COMMAND && x$(head -c2 "$COMMAND") = x"#!" ]]; then
            SHEBANG=$(head -1 "$COMMAND")
            INTERPRETER=( ${SHEBANG#\#!} )
            COMMAND="${INTERPRETER[@]} $COMMAND"
          fi
      
          exec $COMMAND
          ```
      
      which is a wrapper around `gitlab-rake.py` containing:
          ```python
          ...
          if __name__ == '__main__':
              sys.exit(slapos.recipe.librecipe.execute.generic_exec((['/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/bin/bundle', 'exec', 'sh', '-c', 'cd /srv/slapgrid/slappart16/srv/runner/instance/slappart0/gitlab-work && rake "$@"', 'rake'], None, {'BUNDLE_GEMFILE': '/srv/slapgrid/slappart16/srv/runner/software/fffb3c99781923d3adb8bc53eb6c027a/parts/gitlab/Gemfile', 'HOME': '/srv/slapgrid/slappart16/srv/runner/instance/slappart0', 'SIDEKIQ_MEMORY_KILLER_MAX_RSS': '1000000', 'RAILS_ENV': 'production'})))
          ```
      
      `gitlab-rake.py` after is same as `gitlab-rake` before.
      
      This [slapos.cookbook:wrapper](https://lab.nexedi.com/nexedi/slapos/blob/cd9faac0/slapos/recipe/wrapper.py#L39) has an argument *parameters-extra* which if set to true, propagate command line arguments to the wrapped script. The default value for this parameter is false.
      
      Before f4e51f77, the generated wrapper was also propagating arguments even when *parameters-extra* was not set, but since this commit, this *parameters-extra* option is now handled as expected.
      
      This is the reason for this regression. In our case, when we see `/srv/slapgrid/slappart16/srv/runner/instance/slappart0/bin/gitlab-rake assets:clean`, it just calls `rake` without arguments.
      
      So a simple patch that fix the problem would be jerome/slapos@d3d05f02 . This way, the generated wrapper becomes:
      
      ```shell
      ...
      exec $COMMAND $@
      ```
      
      and arguments are correctly propagated.
      
      Feel free to cherry-pick that patch for now, but it may be nice to rethink this *parameters-extra* option, after this debugging session, I believe it should be true by default.
      
      /cc @seb for introducing the parameter in 80bb4305 and @vpelletier for touching this code in e7083872
      
      /reviewed-by @kirr
      6082d6e9
  3. 22 Apr, 2016 2 commits
  4. 14 Mar, 2016 1 commit
  5. 06 Mar, 2016 1 commit
  6. 03 Mar, 2016 1 commit
  7. 02 Mar, 2016 1 commit
  8. 29 Feb, 2016 1 commit
  9. 28 Feb, 2016 5 commits
  10. 17 Feb, 2016 2 commits
  11. 16 Feb, 2016 1 commit
  12. 13 Feb, 2016 1 commit
    • Kirill Smelkov's avatar
      gitlab: Wait a bit for PostgreSQL to be ready in Unicorn startup · 949e55e2
      Kirill Smelkov authored
      As it was outlined in 5a744de7 (gitlab: Compile assets on instantiation
      and make sure DB is properly setup/migrated before unicorn runs) we are
      performing DB initialization and migration in pre-action as part of
      unicorn startup script. But that currently has one drawback:
      
          if all services start at the same time - e.g. both PostgreSQL and
          Unicorn - and that is a common scenario when SR is compiled and
          instantiated / started, PostgreSQL is usually not yet ready to
          process queries from Unicorn startup script, and first-time Unicorn
          startup fails.
      
      Until now this problem was workarounded by manually starting unicorn
      second time - after some time postgresql is started and ready. But why
      do it manually, if we can do the same logic automatically. So fix it:
      
          in Unicorn startup script wait a bit (up to 5 seconds) for
          PostgreSQL to become ready.
      
      /cc @kazuhiko, @jerome
      949e55e2
  13. 11 Feb, 2016 9 commits
  14. 31 Jan, 2016 1 commit
  15. 17 Jan, 2016 7 commits
    • Kirill Smelkov's avatar
      gitlab: First SR version works - freeze md5 sums · 729be3b8
      Kirill Smelkov authored
      We've reached a state where first gitlab SR version should work. So as
      promised let's freeze the md5 checksums.
      
      All later patches should update corresponding md5 info when they change
      a file.
      
      /cc @kazuhiko, @jerome
      729be3b8
    • Kirill Smelkov's avatar
      gitlab: Optimize raw blob downloading · a913c2e4
      Kirill Smelkov authored
      In slapos we do a lot of automated software rebuild constantly, and thus
      there is constant flow of requests to get raw blobs from git service,
      e.g. like this
      
          https://lab.nexedi.com/nexedi/slapos/raw/master/software/wendelin/software.cfg
      
      A lot of requests comes to slapos.git repository and currently gitlab,
      out of the box, cannot keep up with that load.
      
      I've prepared patches to offload raw blobs download requests handling
      from unicorn (ruby) to gitlab-workhorse (go), and that resulted in ~ 17x
      speedup - e.g. previously our std shuttle can handle ~ 70 raw-blob
      requests/s and with my changes it is now ~ 1200 requests/s.
      
      The patches were sent upstream
      
          https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17
      
      and we discussed with GitLab people and made a plan how to proceed
      incrementally. It will probably take some time for gitlab team to fully
      accept the approach though.
      
      For now we can use our gitlab-workhorse fork. The patches itself are:
      
          kirr/gitlab-workhorse@1b274d0d
          kirr/gitlab-workhorse@2beb8c95
      
      /cc @kazuhiko, @jerome, @jm
      a913c2e4
    • Kirill Smelkov's avatar
      gitlab: Switch to "GitLab Nexedi Edition" · 74d4ea62
      Kirill Smelkov authored
      GitLab Nexedi Edition is currently upstream 8.2.X + the following
      patches:
      
          - HTTP(S) is made to be default clone protocol
      
              kirr/gitlab-ce@5c1f2fb3
      
            and SSH info is completely removed from UI
      
              kirr/gitlab-ce@dfe9fb16
              kirr/gitlab-ce@f3f84743
      
            so essentially the only way to access a repository is via HTTP(S).
      
          - Rake check tasks are adjusted to exit with non-zero code if there
            is a failure
      
              kirr/gitlab-ce@a93ae418
      
            We need this for promises to work correctly with failures being
            detected, not silently skipped. The patch was sent upstream:
      
              https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1885
      
          - GitLab supports setting up site's ICP License in gitlab.yml and
            shows it in appropriate places together with info about GitLab
            itself:
      
              kirr/gitlab-ce@e7e0fd88
              kirr/gitlab-ce@79c127e6
      
          + other cosmetic/minor changes.
      
      More patches will probably come (e.g. apply a single patch from a
      merge-request with `git am` without creating merge commit for just 1
      patch, etc) but for now that's all.
      
      NOTE ICP is non-ascii text with hieroglyphs. slapos.core was taught to
          be able to pass parameters with non-ascii values to instance:
      
              nexedi/slapos.core@347d33d6
      
          That patch is included in slapos.core 1.3.15, but as we currently
          have a lot of older slapos.core deployed (e.g. 1.3.5 on my
          development webrunner) a workaround is (hopefully temporarily) used
          to pass non-ascii values as URL-encoded strings.
      
      /cc @kazuhiko, @jerome, @rafael
      74d4ea62
    • Kirill Smelkov's avatar
      gitlab: Setup sidekiq service · 4c127fdd
      Kirill Smelkov authored
      Sidekiq[1] is used in GitLab as background jobs manager - i.e. if a
      request handler needs to spawn some non-light job - it adds it to
      sidekiq queue (in Redis) and relies on sidekiq service to later pick
      this job up and execute it.
      
      The service is setup with just to run bin/gitlab-sidekiq with
      appropriate queues (extracted from omnibus-gitlab) and appropriate
      settings to controlling GitLab's sidekiq Out-Of-Memory killer[2].
      
      NOTE Unlike unicorn OOM killer, Sidekiq memory killer just makes sidekiq
          processes to be SIGKILL terminated and relies on managing service to
          restart it. In slapos we don't have mechanism to set autorestart=true,
          nor bang/watchdog currently work with slapproxy, so we setup to do
          such monitoring ourselves manually with here-introduced
          watcher-sigkill program.
      
      NOTE2 sidekiq promise, because it is rake/gitlab based, is slow to
          load/run and thus is put into etc/promise.slow/
      
      [1] http://sidekiq.org/
      [2] https://gitlab.com/gitlab-org/gitlab-ce/blob/1322bd78/doc/operations/sidekiq_memory_killer.md
      
      /cc @kazuhiko, @jerome
      4c127fdd
    • Kirill Smelkov's avatar
      gitlab/nginx: Slapos'ify config and turn nginx into a service · 85f7d7e3
      Kirill Smelkov authored
      Go through nginx configuration templates and convert them to jinja2 with
      slapos parameters (reminder: names and default values are imported from
      omnibus-gitlab 8.2.3+ce.0-0-g8eda093), except commenting out features we
      do not want to support (yet ?).
      
      As nginx is a reverse-proxy, i.e. it integrates all internal services
      and works as frontend to them, our gitlab service is now ready to listen
      and talk to the world over (standard to slapos services backend) IPv6.
      
      Nginx also acts as SSL termination point - for it to work by default we
      setup self-signed certificate for the backend, which can be manually
      changed to proper certificate if needed. Backend certificate is used
      if gitlab is configured to work in HTTPS mode (and frontend certificate
      is another story).
      
      NOTE ssl certificate is generated with just `openssl req ...` - yes, there
          is slapos.cookbook:certificate_authority.request but it requires
          to start whole service and has up to 60 seconds latency to generate
          certificate. And we only need to run 1 command to do that...
      
      The features disabled are:
      
          - http -> https redirection
      
            not needed for us at nginx level - the frontend can do the
            redirection and also gitlab speaks HSTS on https port so when we access
            https port via http protocol, it gets redirected to https.
      
          - kerberos
          - ssl_dhparam
          - providing custom nginx configuration via instance parameter
      
      /cc @kazuhiko, @jerome
      85f7d7e3
    • Kirill Smelkov's avatar
      gitlab: Upgrade gitlab-shell & gitlab-workhorse to versions which propagate $HOME · 76e371cd
      Kirill Smelkov authored
      As was described in the previous patch, we need $HOME to be propagated
      by this programs so that git can find partition's .gitconfig.
      
      Specifically we need the following patches to be present in our build:
      
          https://gitlab.com/gitlab-org/gitlab-shell/commit/9e087f64
          https://gitlab.com/gitlab-org/gitlab-workhorse/commit/b5f1b803
      
      They both have been applied upstream very close to revisions we
      previously had in software.cfg, so we only need to update the revisions
      to get them.
      
      /cc @kazuhiko, @jerome
      76e371cd
    • Kirill Smelkov's avatar
      gitlab: Hook nginx configuration files into SR system · 45127f6d
      Kirill Smelkov authored
      Like with Rails configuration files, hook nginx configuration files into
      SR / instance build process; rename *.erb -> *.in and add our header.
      
      The templates are still not valid - a lot of erb code is left there -
      we'll slapos'ify it incrementally in the following patches.
      
      /cc @kazuhiko, @jerome
      45127f6d