An error occurred fetching the project authors.
  1. 17 Jan, 2016 13 commits
    • Kirill Smelkov's avatar
      gitlab: First SR version works - freeze md5 sums · 729be3b8
      Kirill Smelkov authored
      We've reached a state where first gitlab SR version should work. So as
      promised let's freeze the md5 checksums.
      
      All later patches should update corresponding md5 info when they change
      a file.
      
      /cc @kazuhiko, @jerome
      729be3b8
    • Kirill Smelkov's avatar
      gitlab: Optimize raw blob downloading · a913c2e4
      Kirill Smelkov authored
      In slapos we do a lot of automated software rebuild constantly, and thus
      there is constant flow of requests to get raw blobs from git service,
      e.g. like this
      
          https://lab.nexedi.com/nexedi/slapos/raw/master/software/wendelin/software.cfg
      
      A lot of requests comes to slapos.git repository and currently gitlab,
      out of the box, cannot keep up with that load.
      
      I've prepared patches to offload raw blobs download requests handling
      from unicorn (ruby) to gitlab-workhorse (go), and that resulted in ~ 17x
      speedup - e.g. previously our std shuttle can handle ~ 70 raw-blob
      requests/s and with my changes it is now ~ 1200 requests/s.
      
      The patches were sent upstream
      
          https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17
      
      and we discussed with GitLab people and made a plan how to proceed
      incrementally. It will probably take some time for gitlab team to fully
      accept the approach though.
      
      For now we can use our gitlab-workhorse fork. The patches itself are:
      
          kirr/gitlab-workhorse@1b274d0d
          kirr/gitlab-workhorse@2beb8c95
      
      /cc @kazuhiko, @jerome, @jm
      a913c2e4
    • Kirill Smelkov's avatar
      gitlab: Switch to "GitLab Nexedi Edition" · 74d4ea62
      Kirill Smelkov authored
      GitLab Nexedi Edition is currently upstream 8.2.X + the following
      patches:
      
          - HTTP(S) is made to be default clone protocol
      
              kirr/gitlab-ce@5c1f2fb3
      
            and SSH info is completely removed from UI
      
              kirr/gitlab-ce@dfe9fb16
              kirr/gitlab-ce@f3f84743
      
            so essentially the only way to access a repository is via HTTP(S).
      
          - Rake check tasks are adjusted to exit with non-zero code if there
            is a failure
      
              kirr/gitlab-ce@a93ae418
      
            We need this for promises to work correctly with failures being
            detected, not silently skipped. The patch was sent upstream:
      
              https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1885
      
          - GitLab supports setting up site's ICP License in gitlab.yml and
            shows it in appropriate places together with info about GitLab
            itself:
      
              kirr/gitlab-ce@e7e0fd88
              kirr/gitlab-ce@79c127e6
      
          + other cosmetic/minor changes.
      
      More patches will probably come (e.g. apply a single patch from a
      merge-request with `git am` without creating merge commit for just 1
      patch, etc) but for now that's all.
      
      NOTE ICP is non-ascii text with hieroglyphs. slapos.core was taught to
          be able to pass parameters with non-ascii values to instance:
      
              nexedi/slapos.core@347d33d6
      
          That patch is included in slapos.core 1.3.15, but as we currently
          have a lot of older slapos.core deployed (e.g. 1.3.5 on my
          development webrunner) a workaround is (hopefully temporarily) used
          to pass non-ascii values as URL-encoded strings.
      
      /cc @kazuhiko, @jerome, @rafael
      74d4ea62
    • Kirill Smelkov's avatar
      gitlab/nginx: Slapos'ify config and turn nginx into a service · 85f7d7e3
      Kirill Smelkov authored
      Go through nginx configuration templates and convert them to jinja2 with
      slapos parameters (reminder: names and default values are imported from
      omnibus-gitlab 8.2.3+ce.0-0-g8eda093), except commenting out features we
      do not want to support (yet ?).
      
      As nginx is a reverse-proxy, i.e. it integrates all internal services
      and works as frontend to them, our gitlab service is now ready to listen
      and talk to the world over (standard to slapos services backend) IPv6.
      
      Nginx also acts as SSL termination point - for it to work by default we
      setup self-signed certificate for the backend, which can be manually
      changed to proper certificate if needed. Backend certificate is used
      if gitlab is configured to work in HTTPS mode (and frontend certificate
      is another story).
      
      NOTE ssl certificate is generated with just `openssl req ...` - yes, there
          is slapos.cookbook:certificate_authority.request but it requires
          to start whole service and has up to 60 seconds latency to generate
          certificate. And we only need to run 1 command to do that...
      
      The features disabled are:
      
          - http -> https redirection
      
            not needed for us at nginx level - the frontend can do the
            redirection and also gitlab speaks HSTS on https port so when we access
            https port via http protocol, it gets redirected to https.
      
          - kerberos
          - ssl_dhparam
          - providing custom nginx configuration via instance parameter
      
      /cc @kazuhiko, @jerome
      85f7d7e3
    • Kirill Smelkov's avatar
      gitlab: Hook nginx configuration files into SR system · 45127f6d
      Kirill Smelkov authored
      Like with Rails configuration files, hook nginx configuration files into
      SR / instance build process; rename *.erb -> *.in and add our header.
      
      The templates are still not valid - a lot of erb code is left there -
      we'll slapos'ify it incrementally in the following patches.
      
      /cc @kazuhiko, @jerome
      45127f6d
    • Kirill Smelkov's avatar
      gitlab: Setup sidekiq service · 4c127fdd
      Kirill Smelkov authored
      Sidekiq[1] is used in GitLab as background jobs manager - i.e. if a
      request handler needs to spawn some non-light job - it adds it to
      sidekiq queue (in Redis) and relies on sidekiq service to later pick
      this job up and execute it.
      
      The service is setup with just to run bin/gitlab-sidekiq with
      appropriate queues (extracted from omnibus-gitlab) and appropriate
      settings to controlling GitLab's sidekiq Out-Of-Memory killer[2].
      
      NOTE Unlike unicorn OOM killer, Sidekiq memory killer just makes sidekiq
          processes to be SIGKILL terminated and relies on managing service to
          restart it. In slapos we don't have mechanism to set autorestart=true,
          nor bang/watchdog currently work with slapproxy, so we setup to do
          such monitoring ourselves manually with here-introduced
          watcher-sigkill program.
      
      NOTE2 sidekiq promise, because it is rake/gitlab based, is slow to
          load/run and thus is put into etc/promise.slow/
      
      [1] http://sidekiq.org/
      [2] https://gitlab.com/gitlab-org/gitlab-ce/blob/1322bd78/doc/operations/sidekiq_memory_killer.md
      
      /cc @kazuhiko, @jerome
      4c127fdd
    • Kirill Smelkov's avatar
      gitlab: Upgrade gitlab-shell & gitlab-workhorse to versions which propagate $HOME · 76e371cd
      Kirill Smelkov authored
      As was described in the previous patch, we need $HOME to be propagated
      by this programs so that git can find partition's .gitconfig.
      
      Specifically we need the following patches to be present in our build:
      
          https://gitlab.com/gitlab-org/gitlab-shell/commit/9e087f64
          https://gitlab.com/gitlab-org/gitlab-workhorse/commit/b5f1b803
      
      They both have been applied upstream very close to revisions we
      previously had in software.cfg, so we only need to update the revisions
      to get them.
      
      /cc @kazuhiko, @jerome
      76e371cd
    • Kirill Smelkov's avatar
      gitlab: Slapos'ify / tweak gitconfig and hook it into the system · 264d785a
      Kirill Smelkov authored
      Convert gitconfig template to jinja2 (reusing already-there
      `email_display_name` and `email_from` parameters for commits generated by
      gitlab).
      
      System-level git config from gitlab-omnibus is also imported to this
      file (on slapos we cannot tweak system-level git config -
      software/.../parts/git/... is read-only for programs in instance
      partitions - so we move all gitlab's system-wide git settings to this
      "user-level" gitconfig.
      
      System gitconfig in omnibus is defined here:
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda093/files/gitlab-cookbooks/gitlab/attributes/default.rb#L23
      
      so it is
      
          pack.threads = 1    and
          receive.fsckObjects = true
      
      which makes sense to not waste a lot of memory when packing and not to
      allow corrupt objects to enter to system by evil users intentionally.
      
      To make the file foundable by git - we put it into partition root
      directory and set $HOME to point to partition root when running
      appropriate programs / services.
      
      NOTE we'll need to upgrade gitlab-shell and gitlab-workhorse to
          propagate $HOME for this setting to actually have effect.
          See the next patch.
      
      /cc @kazuhiko, @jerome
      264d785a
    • Kirill Smelkov's avatar
      gitlab: Compile assets on instantiation and make sure DB is properly... · 5a744de7
      Kirill Smelkov authored
      gitlab: Compile assets on instantiation and make sure DB is properly setup/migrated before unicorn runs
      
      There are several actions that needs to be done on gitlab instance
      upgrade:
      
          - we have to (re-)compile assets
          - we have to migrate DB
      
      and also before the first run
      
          - we have to initialize DB
      
      We can compile assets as part of instantiation process, but regarding
      DB migration / setup - it is not currently possible to do that as part
      of instantiation - for that operations we need PG & Redis to be already
      running, but the first time slapos instantiates an SR it first prepares
      all services, and only after instantiation is done, starts them all. There is
      currently no way to hook into starting process, and run some scripts
      after one service is started but before another service startup...
      
      So the solution is: to perform such actions in delayed mode as part of
      application - unicorn service - startup: it makes sure PG is running and
      initializes it and does other actions which needs to be done to migrate
      the DB. Only if/after they succeed the main application is started.
      
      NOTE the comment about unicorn/gitlab startup slowness from the previous
          patch still holds true - so in order to get "all ok" after
          instantiation, it is required to perform the instantiation several
          times, because unicorn promise initially fails.
      
      /cc @kazuhiko, @jerome
      5a744de7
    • Kirill Smelkov's avatar
      gitlab: Add helper to set up promise to check something via url · 2772191c
      Kirill Smelkov authored
      Like with [promise-wrapper] a recipe could do
      
          [promise-<service>]
          <= promise-byurl
          url     = ...
      
      and a script to check such ur will be generated and automatically put
      into etc/promise/<service>.
      
      /cc @kazuhiko, @jerome
      2772191c
    • Kirill Smelkov's avatar
      gitlab: Introduce macro library · 2e81276b
      Kirill Smelkov authored
      Introduce a library of Jinja2 macros that will be handy to use in
      templates. For now we add only 2 macros:
      
          cfg(name)   - to get instance configuration parameter `name`,   and
          cfg_bool    - to get truth value of ----//----
      
      The reason we introduce cfg() is that we will need to use a lot of
      parameters in many places and it is much more handy to write, e.g.
      
          cfg('email_enabled')
      
      compared to
      
          instance_parameter:configuration.email_enabled
      
      /cc @kazuhiko, @jerome
      2e81276b
    • Kirill Smelkov's avatar
      gitlab: Organize place to keep parameters & their default imported from gitlab-omnibus · de860ba5
      Kirill Smelkov authored
      We will be using a several dozens of parameters to control gitlab
      instance. It makes sense not to deviate in such parameters namings and
      defaults from omnibus version.
      
      Thus for such parameters - for clarity - we organize a separate file
      where we will be keeping them - gitlab-parameters.cfg.
      
      In this patch series all used parameters will be "imported" from
      omnibus-gitlab 8.2.3+ce.0-0-g8eda093.
      
      NOTE it is maybe better to try to autogenerate that file from upstream
          omnibus parameters definitions. If time will tell it becomes hard to
          maintain our copy - we'll consider going that way.
      
      /cc @kazuhiko, @jerome
      de860ba5
    • Kirill Smelkov's avatar
      gitlab: Hook gitlab- and gitlab-shell- configuration files into the system · 13169cab
      Kirill Smelkov authored
      - Download them on SR build and pass info to instance
      - Instance prepares to process them as jinja2 templates
      - Instance hooks the files into configuration location as appropriate
      
      Every file so far is renamed *.erb -> *.in and a header added showing
      that this file is autogenerated with links about what was the base
      gitlab and/or omnibus version and omnibus reference revision this
      template was last updated for.
      
      So far all result configuration files are invalid - because ERB syntax
      is there. We will convert the configuration files to proper jinja2
      syntax and to using slapos parameters incrementally in the upcoming
      patches.
      
      NOTE (again): md5 sums are not yet fixed - we will fix them in the end
          of gitlab patches series after applying all tweaking changes.
      
      /cc @kazuhiko, @jerome
      13169cab
  2. 06 Jan, 2016 2 commits
    • Kirill Smelkov's avatar
      gitlab: Add empty instance · ab6d2f28
      Kirill Smelkov authored
      Add stub instance configuration which just establishes a way to have
      several software types(*), pass all needed info from software to
      instance, organizes base directory and establishes log rotation base for
      upcoming services.
      
      Log rotation is done with the help of cron periodicallly calling
      logrotate. The rotation is done in "copytruncate" mode - i.e. log file
      is not moved away and signal sent for service to reopen it, but instead
      log content is just copied to outside and there is no need for a service
      to reopen it's log file.
      
      The reason it is done this way, is that there is a chance of not
      handling such "reopen-log-file" callbacks correctly on a service side,
      and so the net is full of crashing reports, e.g. like this:
      
          http://serverfault.com/questions/627521/why-is-logrotate-causing-apache-to-seg-fault-each-time
      
      That's why we take a safer approach instead, even if "copytruncate" mode
      is risking to loose several log entries(**) on rotation.
      
      NOTE services will organize log rotation with just
      
          [logrotate-entry-<service>]
          <= logrotate-entry
          log     = path/to/log/files/*.log
      
      For this to work some "!py!" magic (our way to serialize object into
      executable python and process it in buildout recipes) is used to process
      section names.
      
      The approach trick is also used for cron, e.g. logrotate registers to
      cron this way:
      
          [cron-entry-logrotate]
          <= cron-entry
          time    = daily
          command = ${logrotate:wrapper}
      
      NOTE2 instance md5 are not fixed yet - we'll fix them after applying all
          patches in gitlab series.
      
      (*) for now there is only 1 - "gitlab", but we'll need to have "-export"
          and "-import" for resiliency in the future.
      
      (**) ideally such things should be done with logfs - a filesystem
          specializeing in logging - for client services it will look like as
          they just continue to write to log file, and on log service side, the
          rotation can happen, all transparent to client service.
      
      /cc @kazuhiko, @jerome
      ab6d2f28
    • Kirill Smelkov's avatar
      Start of GitLab Software Release · 5e971c58
      Kirill Smelkov authored
      First step - build all needed software. We build:
      
      - Git
      - PostgreSQL 9.2
      - Redis 2.8
      - Nginx
      
      - gitlab-shell
      - gitlab-workhorse
      - gitlab-ce 8.2 itself
      
      and everything which is needed to build the above programs.
      
      Git is needed because GitLab is a git-hosting service and uses git
      underneath. PostgreSQL is used as DB by gitlab and Redis as a cache.
      
      GitLab-shell is a small project to manage ssh access to the service
      (we'll disable ssh though) and to perform all "change a repository"
      operations.
      
      GitLab-workhorse is a service which offloads long-running or slow
      request from main GitLab service.
      
      GitLab-ce is the main Ruby-on-Rails-based web application.
      
      Ruby- and Go- based programs are built in a way similar to:
      
          - 31a45a94    (helloworld & helloweb: Ruby version), and
          - 24e82414    (helloworld & helloweb: Go version)
      
      Version of all components, except Git, were picked the same, as used by
      gitlab omnibus v8.2 .
      
      /cc @kazuhiko, @jerome
      5e971c58