tutorial.txt 42 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12
.. include:: <s5defs.txt>

===========================
Introduction to zc.buildout
===========================

.. raw:: html

   <br /><br /><br />

Jim Fulton, Zope Corporation

Jim Fulton's avatar
Jim Fulton committed
13
DZUG 2007
14 15 16 17

What is zc.buildout?
====================

18
- Coarse-grained python-based configuration-driven build tool
19 20 21 22 23 24 25 26 27 28 29 30 31

- Tool for working with eggs

- Repeatable

  .. class:: handout

     It should be possible to check-in a buildout specification and
     reproduce the same software later by checking out the
     specification and rebuilding.

- Developer oriented

32
Coarse-grained building
33 34 35 36 37 38 39 40 41 42 43 44 45 46
=======================

- make and scons (and distutils) are fine grained

  - Focus on individual files

  - Good when one file is computed from another

    .c -> .o -> .so

  - rule-driven

  - dependency and change driven

47
- zc.buildout is coarse-grained
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127

  - Build large components of a system

    - applications

    - configurations files

    - databases

  - configuration driven

Python-based
============

- make is an awful scripting language

  - uses shell 

  - non-portable

- Python is a good scripting language

  .. class:: handout

     Fortunately, distutils addresses most of my building needs.  If I
     had to write my own fine-grained build definition, I'd use scons.

Working with eggs
=================

- Eggs rock!

- easy_install

  - Easy!

  - Installs into system Python

  - Not much control

- workingenv makes easy_install much more usable

  - Avoids installing into system Python

  - Avoids conflicts with packages installed in site_packages

  - Really nice for experimentation

  - Easy!

  - Not much control

``zc.buildout`` and eggs
========================

- Control

  - Configuration driven

    - easier to control versions used

    - always look for most recent versions by default

      .. class:: handout
      
         When upgrading a distribution, ``easy_install`` doesn't upgrade
         dependencies, 

    - support for custom build options

- Greater emphasis on develop eggs

  - Automates install/uninstall

  - preference to develop eggs

  .. class:: handout

     I often switch between develop and non-develop eggs.  I may be
     using a regular egg and realize I need to fix it.  I checkout the
Jim Fulton's avatar
Jim Fulton committed
128
     egg's project into my buildout and tell buildout to treat it as a
129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178
     develop egg.  It creates the egg link in develop eggs and will
     load the develop egg in preference to the non-develop egg.

     (``easy_install`` gives preference to released eggs of the same
     version.) 

     When I'm done making my change, I make a new egg release and tell
     buildout to stop using a develop egg.


.. class:: handout

   ``zc.buildout`` is built on setuptools and ``easy_install``.


zc.buildout current status
==========================

- Actively used for development and deployment

- Third-generation of ZC buildout tools

  .. class:: handout

     Our earliest buildouts used make.  These were difficult to
     maintain and reuse.

     Two years ago, we created a prototype Python-based buildout
     system. 

     ``zc.buildout`` is a non-prototype system that reflects
     experience using the prototype.

- A number of "recipes" available

A Python Egg Primer
===================

Eggs are simple!

- directories to be added to path

  - may be zipped

  - "zero" installation

- Meta data

  - dependencies

Jim Fulton's avatar
Jim Fulton committed
179
  - entry points
180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234

- May be distributed as source distributions

  .. class:: handout

     ``easy_install`` and ``zc.buildout`` can install source
     distributions as easily as installing eggs.  I've found that
     source distributions are more convenient to distribute in a lot
     of ways.

- Automatic discovery through PyPI

Egg jargon
==========

- Distribution

  .. class:: handout

     "distribution" is the name distutils uses for something that can
     be distributed.  There are several kinds of distributions that
     can be created by distutils, including source distributions,
     binary distributions, eggs, etc.

- source and binary distributions

  .. class:: handout

     A source distribution contains the source for a project.

     A binary distributions contains a compiled version of a project,
     including .pyc files and built extension modules.

     Eggs are a type of binary distribution.

- Platform independent and platform dependent eggs

  .. class:: handout

     Platform dependent eggs contain built extension modules and are
     thus tied to a specific operating system.  In addition, they may
     depend on build options that aren't reflected in the egg name.
  
- develop egg links

  .. class:: handout

     Develop egg links (aka develop eggs) are special files that allow
     a source directory to be treated as an egg.  An egg links is a
     file containing the path of a source directory.
  
- requirements

  .. class:: handout

235
     Requirements are strings that name distributions.  They consist
236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330
     of a project name, optional version specifiers, and optional
     extras specifiers.  Extras are names of features of a package
     that may have special dependencies.

- index and link servers

  ``easy_install`` and ``zc.buildout`` will automatically download
  distributions from the Internet.  When looking for distributions,
  they will look on zero or more links servers for links to
  distributions.

  They will also look on a single index server, typically (always)
  http://www.python.org/pypi.   Index servers are required to provide
  a specific web interface.

Entry points
============

- Very similar to utilities

  - Named entry point groups define entry point types

  - Named entry points within groups provide named components of a
    given type.

- Allow automated script generation

  Wrapper script:

  - Sets up path

    .. class:: handout

       ``easy_install`` and ``zc.buildout`` take very different
       approaches to this.

       ``easy_install`` generates scripts that call an API that loads
       eggs dynamically at run time.

       ``zc.buildout`` determines the needed eggs at build time and
       generates code in scripts to explicitly add the eggs to
       ``sys.path``.

       The approach taken by ``zc,buildout`` is intended to make
       script execution deterministic and less susceptible to
       accidental upgrades.

  - Imports entry point

  - Calls entry point without arguments

    .. class:: handout

       Buildout allows more control over script generation.
       Initialization code and entry point arguments can be
       specified. 


Buildout overview
=================

- Configuration driven

  - ConfigParser +

    .. class:: handout

       Buildout uses the raw ConfigParser format extended with
       a variable-substitution syntax that allows reference to
       variables by section and option::

          ${sectionname:optionname}

  - Allows full system to be defined with a single file

    .. class:: handout

       Although it is possible and common to factor into multiple
       files. 

- Specify a set of "parts"

  - recipe
 
  - configuration data

  .. class:: handout
  
     Each part is defined by a recipe, which is Python software for
     installing or uninstalling the part, and data used by the recipe.

- Install and uninstall

  .. class:: handout

Jim Fulton's avatar
Jim Fulton committed
331
     If a part is removed from a specification, it is uninstalled.
332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467

     If a part's recipe or configuration changes, it is uninstalled
     and reinstalled.
     

Buildout overview (continued)
=============================

- Recipes

  - Written in python

  - Distributed as eggs

- Egg support

  - Develop eggs

  - Egg-support recipes

Quick intro
===========

- Most common case

  - Working on a package

  - Want to run tests

  - Want to generate distributions

- buildout is source project

- Example: ``zope.event``

``zope.event`` project files
=============================

- source in ``src`` directory

  .. class:: handout

     Placing source in a separate ``src`` directory is a common
     convention.  It violates "shallow is better than nested". Smaller
     projects may benefit from putting sources in the root directory,

- ``setup.py`` for defining egg

  .. class:: handout

    Assuming that the project will eventually produce an egg, we have a
    setup file for the project.  As we'll see later, this can be very
    minimal to start.

- ``README.txt``

  .. class:: handout

     It is conventional to put a README.txt in the root of the
     project. distutils used to complain if this wasn't available.
  
- ``bootstrap.py`` for bootstrapping buildout

  .. class:: handout

     The bootstrap script makes it easy to install the buildout
     software.  We'll see another way to do this later.

- ``buildout.cfg`` defines the buildout

zope.event buildout.cfg
=======================

::
  
  [buildout]
  parts = test
  develop = .

  [test]
  recipe = zc.recipe.testrunner
  eggs = zope.event

.. class:: handout

   Let's go through this line by line.

   ::

     [buildout]

   defines the buildout section. It is the only required section in
   the configuration file.  It is options in this section that may
   cause other sections to be used.

   ::

     parts = test

   Every buildout is required to specify a list of parts, although the
   parts list is allowed to be empty.  The parts list specifies what
   to build.  If any of the parts listed depend on other parts, then
   the other parts will be built too.

   ::

     develop = .

   The develop option is used to specify one or more directories from
   which to create develop eggs. Here we specify the current
   directory. Each of these directories must have a setup file.

   ::

     [test]

   The ``test`` section is used to define our test part.

   ::

     recipe = zc.recipe.testrunner

   Every part definition is required to specify a recipe.  The recipe
   contains the Python code with the logic to install the part.  A
   recipe specification is a distribution requirement. The requirement
   may be followed by an colon and a recipe name.  Recipe eggs can
   contain multiple recipes and can also define an default recipe.
   
   The ``zc.recipe.testrunner`` egg defines a default recipe that
   creates a test runner using the ``zope.testing.testrunner``
   framework.

   ::

     eggs = zope.event

Jim Fulton's avatar
Jim Fulton committed
468 469 470
   The zc.recipe.testrunnner recipe has an eggs option for specifying
   which eggs should be tested.  The generated test script will load
   these eggs along with their dependencies.
471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505

   For more information on the ``zc.recipe.testrunner`` recipe, see
   http://www.python.org/pypi/zc.recipe.testrunner.

Buildout steps
==============

- Bootstrap the buildout::

    python bootstrap.py

  .. class:: handout

       This installs setuptools and zc.buildout locally in your
       buildout. This avoids changing your system Python.

- Run the buildout::

    bin/buildout

  .. class:: handout

       This generates the test script, ``bin/test``.

- Run the tests::

    bin/test

- Generate a distribution::

    bin/buildout setup . sdist register upload
    bin/buildout setup . bdist_egg register upload

  ::

Jim Fulton's avatar
Jim Fulton committed
506
    bin/buildout setup . egg_info -rbdev sdist register upload
507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531

  .. class:: handout

     Buildout accepts a number of commands, one of which is
     ``setup``.  The ``setup`` command takes a directory name and runs
     the setup script found there.  It arranges for setuptools to be
     imported before the script runs. This causes setuptools defined
     commands to work even for distributions that don't use
     setuptools.

     The sdist, register, upload, bdist_egg, and egg_info commands are
     setuptools and distutils defined commands.

     The sdist command causes a source distribution to be created.

     The register command causes a release to be registered with PyPI
     and the upload command uploads the generated distribution.
     You'll need to have an account on PyPI for this to work, but
     these commands will actually help you set an account up.

     The bdist_egg command generates an egg.

     The egg_info command allows control of egg meta-data.  The -r
     option to the egg_info command causes the distribution to have a
     version number that includes the subversion revision number of
Jim Fulton's avatar
Jim Fulton committed
532 533
     the project.  The -b option specified a revision tag. Here we
     specified a revision tag of "dev", which marks the release as a
534
     development release. These are useful when making development
Jim Fulton's avatar
Jim Fulton committed
535
     releases.
536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638

Exercise 1
===========

.. class:: handout

   We won't have time to stop the lecture while you do the
   exercises. If you can play and listen at the same time, then feel
   free to work on them while I speak. Otherwise, I recommend doing
   them later in the week. Feel free to ask me questions if you run
   into problems.

Try building out ``zope.event``.

- Check out: svn://svn.zope.org/repos/main/zope.event/trunk

- Bootstrap

- Run the buildout

- Run the tests

- Look around the buildout to see how things are laid out.

- Look at the scripts in the bin directory.

buildout layout
===============

- ``bin`` directory for generated scripts

- ``parts`` directory for generated part data

  Many parts don't use this.

- ``eggs`` directory for (most) installed eggs

  - May be shared across buildouts.

- ``develop-eggs`` directory

  - develop egg links

  - custom eggs

- ``.installed.cfg`` records what has been installed

.. class:: handout

   Some people find the buildout layout surprising, as it isn't
   similar to a Unix directory layout.  The buildout layout was guided
   by "shallow is better than nested".  

   If you prefer a different layout, you can specify a different
   layout using buildout options.  You can set these options globally
   so that all of your buildouts have the same layout.

Common buildout use cases
=========================

- Working on a single package

  .. class:: handout

     zope.event is an example of this use case.

- System assembly

- Try out new packages

  - workingenv usually better

  - buildout better when custom 
    build options needed

- Installing egg-based scripts for personal use

  ``~/bin`` directory is a buildout

Creating eggs
=============

Three levels of egg development

- Develop eggs, a minimal starting point

- Adding data needed for distribution

- Polished distributions

A Minimal/Develop ``setup.py``
==============================

.. code-block:: Python

   from setuptools import setup
   setup(
       name='foo',
       package_dir = {'':'src'},
       )

.. class:: handout

639
   If we're only going to use a package as a develop egg, we just need
640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723
   to specify the project name, and, if there is a separate source
   directory, then we need to specify that location.

   We'd also need to specify entry points if we had any.  We'll see an
   example of that later.

   See the setuptools and distutils documentation for more information.

Distributable ``setup.py``
==========================

.. code-block:: Python

  from setuptools import setup, find_packages
  name='zope.event'
  setup(
      name=name,
      version='3.3.0',
      url='http://www.python.org/pypi/'+name,
      author='Zope Corporation and Contributors',
      author_email='zope3-dev@zope.org',
      package_dir = {'': 'src'},
      packages=find_packages('src'),
      namespace_packages=['zope',],
      include_package_data = True,
      install_requires=['setuptools'],
      zip_safe = False,
      )

.. class:: handout

   If we want to be able to create a distribution, then we need to
   specify a lot more information.

   The options used are documented in either the distutils or
   setuptools documentation. Most of the options are fairly obvious.

   We have to specify the Python packages used. The ``find_packages``
   function can figure this out for us, although it would often be
   easy to specify it ourselves.  For example, we could have
   specified::

      packages=['zope', 'zope.event'],

   The zope package is a namespace package.  This means that it exists
   solely as a container for other packages. It doesn't have any files
   or modules of it's own.  It only contains an `__init__` module
   with::

     pkg_resources.declare_namespace(__name__)

   or, perhaps::

     # this is a namespace package
     try:
         import pkg_resources
         pkg_resources.declare_namespace(__name__)
     except ImportError:
         import pkgutil
         __path__ = pkgutil.extend_path(__path__, __name__)

   Namespace packages have to be declared, as we've done here.
  
   We always want to include package data.

   Because the `__init__` module uses setuptools, we declare it as a
   dependency, using ``install_requires``.

   We always want to specify whether a package is zip safe.  A zip
   safe package doesn't try to access the package as a directory.  If
   in doubt, specify False.  If you don't specify anything, setuptools
   will guess.

Polished ``setup.py`` (1/3)
===========================

.. code-block:: Python

  import os
  from setuptools import setup, find_packages

  def read(*rnames):
      return open(os.path.join(os.path.dirname(__file__), *rnames)).read()

Jim Fulton's avatar
Jim Fulton committed
724
  long_description=(
725 726 727 728 729 730 731 732 733
          read('README.txt')
          + '\n' +
          'Detailed Documentation\n'
          '**********************\n'
          + '\n' +
          read('src', 'zope', 'event', 'README.txt')
          + '\n' +
          'Download\n'
          '**********************\n'
Jim Fulton's avatar
Jim Fulton committed
734 735 736
          )

  open('documentation.txt', 'w').write(long_description)
737 738 739

.. class:: handout

Jim Fulton's avatar
Jim Fulton committed
740
In the polished version we flesh out the meta data a bit more.
741

Jim Fulton's avatar
Jim Fulton committed
742 743 744 745 746
When I create distributions that I consider ready for broader use and
upload to PyPI, I like to include the full documentation in the long
description so PyPI serves it for me.

Polished ``setup.py`` (2/3)
747 748
===========================

Jim Fulton's avatar
Jim Fulton committed
749
.. code-block:: Python
750

Jim Fulton's avatar
Jim Fulton committed
751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768
  name='zope.event'
  setup(
      name=name,
      version='3.3.0',
      url='http://www.python.org/pypi/'+name,
      license='ZPL 2.1',
      description='Zope Event Publication',
      author='Zope Corporation and Contributors',
      author_email='zope3-dev@zope.org',
      long_description=long_description,

      packages=find_packages('src'),
      package_dir = {'': 'src'},
      namespace_packages=['zope',],
      include_package_data = True,
      install_requires=['setuptools'],
      zip_safe = False,
      )
769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789

Extras
======

.. code-block:: Python

  name = 'zope.component'
  setup(name=name,
        ...
        namespace_packages=['zope',],
        install_requires=['zope.deprecation', 'zope.interface',
                          'zope.deferredimport', 'zope.event',
                          'setuptools', ],
        extras_require = dict(
            service = ['zope.exceptions'],
            zcml = ['zope.configuration', 'zope.security', 'zope.proxy',
                    'zope.i18nmessageid',
                    ],
            test = ['zope.testing', 'ZODB3',
                    'zope.configuration', 'zope.security', 'zope.proxy',
                    'zope.i18nmessageid',
790
                    'zope.location', # should be dependency of zope.security
791 792 793 794 795 796 797 798 799 800 801 802
                    ],
            hook = ['zope.hookable'],
            persistentregistry = ['ZODB3'],
            ),
        )


.. class:: handout

   Extras provide a way to help manage dependencies.

   A common use of extras is to separate test dependencies from normal
803
   dependencies.  A package may provide other optional features that
804
   cause other dependencies.  For example, the zcml module in
805
   zope.component adds lots of dependencies that we don't want to
Jim Fulton's avatar
Jim Fulton committed
806
   impose on people that don't use it.
807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860

``zc.recipe.egg``
=================

Set of recipes for:

- installing eggs

- generating scripts

- custom egg compilation

- custom interpreters

See: http://www.python.org/pypi/zc.recipe.egg.

Installing eggs
===============

::

  [buildout]
  parts = some-eggs

  [some-eggs]
  recipe = zc.recipe.egg:eggs
  eggs = docutils
         ZODB3 <=3.8
         zope.event

.. class:: handout

   The eggs option accepts one or more distribution requirements.
   Because requirements may contain spaces, each requirement must be
   on a separate line.  We used the eggs option to specify the eggs we
   want.

   Any dependencies of the named eggs will also be installed.


Installing scripts
==================

::

  [buildout]
  parts = rst2

  [rst2]
  recipe = zc.recipe.egg:scripts
  eggs = zc.rst2

.. class:: handout

861
   If any of the named eggs have ``console_script`` entry
862 863
   points, then scripts will be generated for the entry points.

864 865
   If a distribution doesn't use setuptools, it may not declare it's entry
   points. In that case, you can specify entry points in the recipe data.
866 867
   Buildout *does* detect distutils-style scripts without an entry point and
   will generate a script for them when found.
868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953
  
Script initialization
=====================

::

  [buildout]
  develop = codeblock
  parts = rst2
  find-links = http://sourceforge.net/project/showfiles.php?group_id=45693

  [rst2]
  recipe = zc.recipe.egg
  eggs = zc.rst2 
         codeblock
  initialization = 
      sys.argv[1:1] = (
        's5 '
        '--stylesheet ${buildout:directory}/zope/docutils.css '
        '--theme-url file://${buildout:directory}/zope' 
        ).split()
  scripts = rst2=s5

.. class:: handout

   In this example, we omitted the recipe entry point entry name
   because the scripts recipe is the default recipe for the
   zc.recipe.egg egg.

   The initialization option lets us specify some Python code to be included.

   We can control which scripts get installed and what their names are
   with the scripts option. In this example, we've used the scripts
   option to request a script named ``s5`` from the ``rst2`` entry point.

Custom interpreters
===================

The script recipe allows an interpreter script to be created.

::

  [buildout]
  parts = mypy

  [mypy]
  recipe = zc.recipe.egg:script
  eggs = zope.component
  interpreter = py

This will cause a ``bin/py`` script to created.

.. class:: handout

   Custom interpreters can be used to get an interactive Python prompt
   with the specified eggs and and their dependencies on ``sys.path``.
   
   You can also use custom interpreters to run scripts, just like you
   would with the usual Python interpreter.  Just call the interpreter
   with the script path and arguments, if any.

Exercise 2
==========

- Add a part to the ``zope.event`` project to create a custom interpreter.

- Run the interpreter and verify that you can import zope.event.

Custom egg building
===================

::

  [buildout]
  parts = spreadmodule

  [spreadtoolkit]
  recipe = zc.recipe.cmmi
  url = http://yum.zope.com/buildout/spread-src-3.17.1.tar.gz

  [spreadmodule]
  recipe = zc.recipe.egg:custom
  egg = SpreadModule ==1.4
  find-links = http://www.python.org/other/spread/
  include-dirs = ${spreadtoolkit:location}/include
  library-dirs = ${spreadtoolkit:location}/lib
Jim Fulton's avatar
Jim Fulton committed
954
  rpath = ${spreadtoolkit:location}/lib
955 956 957 958 959 960 961 962 963 964

.. class:: handout

   Sometimes a distribution has extension modules that need to be
   compiled with special options, such as the location of include
   files and libraries,  The custom recipe supports this.  The
   resulting eggs are placed in the develop-eggs directory because the
   eggs are buildout specific.

   This example illustrates use of the zc.recipe.cmmi recipe with
965
   supports installation of software that uses configure, make, make install.
966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985
   Here, we used the recipe to install the spread toolkit, which is
   installed in the parts directory.

Part dependencies
=================

- Parts can read configuration from other parts

- The parts read become dependencies of the reading parts

  - Dependencies are added to parts list, if necessary

  - Dependencies are installed first


.. class:: handout

   In the previous example, we used the spread toolkit location in the
   spreadmodule part definition. This reference was sufficient to make
   the spreadtoolkit part a dependency of the spreadmodule part and
986
   cause it to be installed first.
987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054

Custom develop eggs
===================

::

  [buildout]
  parts = zodb

  [zodb]
  recipe = zc.recipe.egg:develop
  setup = zodb
  define = ZODB_64BIT_INTS

.. class:: handout

   We can also specify custom build options for develop eggs.  Here we
   used a develop egg just to make sure our custom build of ZODB took
   precedence over normal ZODB eggs in our shared eggs directory.

Writing recipes
===============

- The recipe API

  - install

    - __init__

      .. class:: handout

         The initializer is responsible for computing a part's
         options.  After the initializer call, the options directory
         must reflect the full configuration of the part. In
         particular, if a recipe reads any data from other sections,
         it must be reflected in the options.  The options data after
         the initializer is called is used to determine if a
         configuration has changed when deciding if a part has to
         be reinstalled.  When a part is reinstalled, it is
         uninstalled and then installed.
      
    - install

      .. class:: handout

         The install method installs the part.  It is used when a part
         is added to a buildout, or when a part is reinstalled.

         The install recipe must return a sequence of paths that that
         should be removed when the part is uninstalled.  Most recipes
         just create files or directories and removing these is
         sufficient for uninstalling the part.

    - update

      .. class:: handout

         The update method is used when a part is already installed
         and it's configuration hasn't changed from previous
         buildouts.  It can return None or a sequence of paths. If
         paths are returned, they are added to the set of installed
         paths. 

  - uninstall

    .. class:: handout

       Most recipes simply create files or directories and the
1055
       built-in buildout uninstall support is sufficient.  If a recipe
1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091
       does more than simply create files, then an uninstall recipe
       will likely be needed.

Install Recipes
===============

``mkdirrecipe``.py:

.. code-block:: Python

  import logging, os, zc.buildout

  class Mkdir:

      def __init__(self, buildout, name, options):
          self.name, self.options = name, options
          options['path'] = os.path.join(
                                buildout['buildout']['directory'],
                                options['path'],
                                )
          if not os.path.isdir(os.path.dirname(options['path'])):
              logging.getLogger(self.name).error(
                  'Cannot create %s. %s is not a directory.',
                  options['path'], os.path.dirname(options['path']))
              raise zc.buildout.UserError('Invalid Path')

.. class:: handout

   - The path option in our recipe is interpreted relative to the
     buildout. We reflect this by saving the adjusted path in the
     options.

   - If there is a user error, we:

     - Log error details using the Python logger module.

1092
     - Raise a zc.buildout.UserError exception.
1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217

``mkdirrecipe``.py continued
============================

  .. code-block:: Python

        def install(self):
            path = self.options['path']
            logging.getLogger(self.name).info(
                'Creating directory %s', os.path.basename(path))
            os.mkdir(path)
            return path

        def update(self):
            pass

.. class:: handout

   A well-written recipe will log what it's doing.

   Often the update method is empty, as in this case.

Uninstall recipes
=================

``servicerecipe.py``:

.. code-block:: Python

   import os

   class Service:
   
       def __init__(self, buildout, name, options):
           self.options = options
   
       def install(self):
           os.system("chkconfig --add %s" % self.options['script'])         
           return ()
   
       def update(self):
           pass
   
   def uninstall_service(name, options):
       os.system("chkconfig --del %s" % options['script'])

.. class:: handout

   Uninstall recipes are callables that are passed the part name and
   the **original options**.

Buildout entry points
=====================

``setup.py``:

.. code-block:: Python

   from setuptools import setup

   entry_points = """
   [zc.buildout]
   mkdir = mkdirrecipe:Mkdir
   service = servicerecipe:Service
   default = mkdirrecipe:Mkdir

   [zc.buildout.uninstall]
   service = servicerecipe:uninstall_service
   """

   setup(name='recipes', entry_points=entry_points)
   
Exercise 3
==========

- Write recipe that creates a file from source given in a
  configuration option.

- Try this out in a buildout, either by creating a new buildout, or by
  extending the ``zope.event`` buildout.

Command-line options
====================

Buildout command-line:

- command-line options and option setting

- command and arguments

::

  bin/buildout -U -c rpm.cfg install zrs

.. class:: handout

   Option settings are of the form::

     section:option=value

   Any option you can set in the configuration file, you can set on
   the command-line. Option settings specified on the command line
   override settings read from configuration files.

   There are a few command-line options, like -c to specify a
   configuration file, or -U to disable reading user defaults.

   See the buildout documentation, or use the -h option to get a list
   of available options.

Buildout modes
==============

- newest

  - default mode always tries to get newest versions

  - Turn off with -N or buildout newest option set to false.

- offline

  - If enabled, then don't try to do network access

  - Disabled by default

1218
  - If enabled, turn off with -o or buildout offline option set to false.
1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384

.. class:: handout

   By default, buildout always tries to find the newest distributions
   that match requirements. Looking for new distributions can be very
   time consuming.  Many people will want to specify the -N option to
   disable this.  We'll see later how we can change this default
   behavior.

   If you aren't connected to a network, you'll want to use the
   offline mode, -o.

``~/.buildout/default.cfg``
===========================

Provides default buildout settings (unless -U option is used):

::

    [buildout]
    # Shared eggs directory:
    eggs-directory = /home/jim/.eggs
    # Newest mode off, reenable with -n
    newst = false

    [python24]
    executabe = /usr/local/python/2.4/bin/python

    [python25]
    executabe = /usr/local/python/2.5/bin/python

.. class:: handout

   Unless the -U command-line option is used, user default settings
   are read before reading regular configuration files.  The user
   defaults are read from the default.cfg file in the .buildout
   subdirectory of the directory specified in the HOME environment
   variable, if any.

   In this example:

   - I set up a shared eggs directory.

   - I changed the default mode to non-newest so that buildout doesn't
     look for new distributions if the distributions it has meet it's
     requirements.  To get the newest distributions, I'll have to use
     the -n option.

   - I've specified Python 2.4 and 2.5 sections that specify locations
     of Python interpreters.  Sometimes, a buildout uses multiple
     versions of Python. Many recipes accept a python option that
     specifies the name of a section with an executable option
     specifying the location of a Python interpreter.

Extending configurations
========================

The ``extends`` option allows one configuration file to extend
another. 

For example:

- ``base.cfg`` has common definitions and settings

- ``dev.cfg`` adds development-time options::

   [buildout]
   extends = base.cfg

   ...

- ``rpm.cfg`` has options for generating an RPM packages from a
  buildout.

Bootstrapping from existing buildout
====================================

- The buildout script has a ``bootstrap`` command

- Can use it to bootstrap any directory.

- Much faster than running ``bootstrap.py`` because it can use an already
  installed ``setuptools`` egg.

Example: ~/bin directory
========================

::

  [buildout]
  parts = rst2 buildout24 buildout25
  bin-directory = .

  [rst2]
  recipe = zc.recipe.egg
  eggs = zc.rst2

  [buildout24]
  recipe = zc.recipe.egg
  eggs = zc.buildout
  scripts = buildout=buildout24
  python = python24

  [buildout25]
  recipe = zc.recipe.egg
  eggs = zc.buildout
  scripts = buildout=buildout25
  python = python25


.. class:: handout

   Many people have a personal scripts directory.

   I've converted mine to a buildout using a buildout configuration
   like the one above.

   I've overridden the bin-directory location so that scripts are
   installed directly into the buildout directory.

   I've specified that I want the zc.rst2 distribution installed.  The
   rst2 distribution has a generalized version of the restructured
   text processing scripts in a form that can be installed by buildout
   (or easy_install).

   I've specified that I want buildout scripts for Python 2.4 and
   2.5. (In my buildout, I also create one for Python 2.3.)  These
   buildout scripts allow me to quickly bootstrap buildouts or to run
   setup files for a given version of python.  For example, to
   bootstrap a buildout with Python 2.4, I'll run::

     buildout24 bootstrap

   in the directory containing the buildout.  This can also be used to
   convert a directory to a buildout, creating a buildout.cfg file is
   it doesn't exist.

Example: zc.sharing (1/2)
=========================

::

  [buildout]
  develop = . zc.security 
  parts = instance test
  find-links = http://download.zope.org/distribution/

  [instance]
  recipe = zc.recipe.zope3instance
  database = data
  user = jim:123
  eggs = zc.sharing
  zcml = 
    zc.resourcelibrary zc.resourcelibrary-meta
    zc.sharing-overrides:configure.zcml zc.sharing-meta
    zc.sharing:privs.zcml zc.sharing:zope.manager-admin.zcml
    zc.security zc.table zope.app.securitypolicy-meta zope.app.twisted
    zope.app.authentication

.. class:: handout

   This is a small example of the "system assembly" use case.  In this
   case, we define a Zope 3 instance, and a test script.

   You can largely ignore the details of the Zope 3 instance  recipe.
   If you aren't a Zope user, you don't care.  If you are a Zope user,
1385
   you should be aware that much better recipes have been developped. 
1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430

   This project uses multiple source directories, the current
   directory and the zc.security directory, which is a subversion
   external to a project without its own distribution. We've listed
   both in the develop option.

   We've requested the instance and test parts.  We'll get other parts
   installed due to dependencies of the instance part.  In particular,
   we'll get a Zope 3 checkout because the instance recipe refers to
   the zope3 part.  We'll get a database part because of the reference
   in the database option of the instance recipe.

   The buildout will look for distributions at
   http://download.zope.org/distribution/. 

Example: zc.sharing (2/2)
=========================

::

  [zope3]
  recipe = zc.recipe.zope3checkout
  url = svn://svn.zope.org/repos/main/Zope3/branches/3.3

  [data]
  recipe = zc.recipe.filestorage

  [test]
  recipe = zc.recipe.testrunner
  defaults = ['--tests-pattern', 'f?tests$']
  eggs = zc.sharing 
         zc.security
  extra-paths = ${zope3:location}/src

.. class:: handout

   Here we see the definition of the remaining parts.

   The test part has some options we haven't seen before.

   - We've customized the way the testrunner finds tests by providing
     some testrunner default arguments.

   - We've used the extra-paths option to tell the test runner to
     include the Zope 3 checkout source directory in sys.path.  This
1431
     is not necessary as Zope 3 is now available entirely as eggs.
1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461

Source vs Binary
================

- Binary distributions are Python version and often platform specific

- Platform-dependent distribution can reflect build-time setting not
  reflected in egg specification.

  - Unicode size

  - Library names and locations

- Source distributions are more flexible

- Binary eggs can go rotten when system libraries are upgraded

  .. class:: handout

     Recently, I had to manually remove eggs from my shared eggs
     directory.  I had installed an operating system upgrade that
     caused the names of open-ssl library files to change.  Eggs build
     against the old libraries no-longer functioned. 


RPM experiments
===============

Initial work creating RPMs for deployment in our hosting environment:

Jim Fulton's avatar
Jim Fulton committed
1462 1463
- Separation of software and configuration

1464 1465 1466 1467
- Buildout used to create rpm containing software 

- Later, the installed buildout is used to set up specific processes

Jim Fulton's avatar
Jim Fulton committed
1468 1469 1470 1471
  - Run as root in offline mode

  - Uses network configuration server

1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611
.. class:: handout

   Our philosophy is to separate software and configuration.  We
   install software using RPMs. Later, we configure the use of the
   software using a centralized configuration database.

   I'll briefly present the RPM building process below.  This is
   interesting, in part, because it illustrates some interesting issues.


ZRS spec file (1/3)
===================

::

  %define python zpython
  %define svn_url svn+ssh://svn.zope.com/repos/main/ZRS-buildout/trunk
  requires: zpython
  Name: zrs15
  Version: 1.5.1
  Release: 1
  Summary: Zope Replication Service
  URL: http://www.zope.com/products/zope_replication_services.html

  Copyright: ZVSL
  Vendor: Zope Corporation
  Packager: Zope Corporation <sales@zope.com>
  Buildroot: /tmp/buildroot
  Prefix: /opt
  Group: Applications/Database
  AutoReqProv: no

.. class:: handout

   Most of the options above are pretty run of the mill.

   We specify the Python that we're going to use as a dependency.  We
   build our Python RPMs so we can control what's in them.  System
   packagers tend to be too creative for us.

   Normally, RPM installs files in their run-time locations at build
   time. This is undesirable in a number of ways.  I used the rpm
   build-root mechanism to allow files to be build in a temporary
   tree.

   Because the build location is different than the final install
   location, paths written by the buildout, such as egg paths in
   scripts are wrong.  There are a couple of ways to deal with this:

   - I could try to adjust the paths at build time, 

   - I could try to adjust the paths at install time.

   Adjusting the paths at build time means that the install locations
   can;'t be controlled at install time.  It would also add complexity
   to all recipes that deal with paths.  Adjusting the paths at
   install time simply requires rerunning some of the recipes to
   generate the paths.

   To reinforce the decision to allow paths to be specified at install
   time, we've made the RPM relocatable using the prefix option.

ZRS spec file (2/3)
===================

::

  %description
  %{summary}

  %build
  rm -rf $RPM_BUILD_ROOT
  mkdir $RPM_BUILD_ROOT
  mkdir $RPM_BUILD_ROOT/opt
  mkdir $RPM_BUILD_ROOT/etc
  mkdir $RPM_BUILD_ROOT/etc/init.d
  touch $RPM_BUILD_ROOT/etc/init.d/%{name}
  svn export %{svn_url} $RPM_BUILD_ROOT/opt/%{name}
  cd $RPM_BUILD_ROOT/opt/%{name}
  %{python} bootstrap.py -Uc rpm.cfg
  bin/buildout -Uc rpm.cfg buildout:installed= \
     bootstrap:recipe=zc.rebootstrap

.. class:: handout

   I'm not an RPM expert and RPM experts would probably cringe to see
   my spec file.  RPM specifies a number of build steps that I've
   collapsed into one.

   - The first few lines set up build root.

   - We export the buildout into the build root.

   - We run the buildout

     - The -U option is used mainly to avoid using a shared eggs
       directory

     - The -c option is used to specify an RPM-specific buildout file
       that installs just software, including recipe eggs that will be
       needed after installation for configuration.

     - We suppress creation of an .installed.cfg file

     - We specify a recipe for a special bootstrap part. The bootstrap
       part is a script that will adjust the paths in the buildout
       script after installation of the rpm.

ZRS spec file (3/3)
===================

::

  %post
  cd $RPM_INSTALL_PREFIX/%{name}
  %{python} bin/bootstrap -Uc rpmpost.cfg
  bin/buildout -Uc rpmpost.cfg \
     buildout:offline=true buildout:find-links= buildout:installed= \
     mercury:name=%{name} mercury:recipe=buildoutmercury
  chmod -R -w . 

  %preun
  cd $RPM_INSTALL_PREFIX/%{name}
  chmod -R +w . 
  find . -name \*.pyc | xargs rm -f

  %files
  %attr(-, root, root) /opt/%{name}
  %attr(744, root, root) /etc/init.d/%{name}

.. class:: handout

   We specify a post-installation script that:

   - Re-bootstraps the buildout using the special bootstrap script
     installed in the RPM.

   - Reruns the buildout:

     - Using a post-installation configuration that specified the
1612
       parts whose paths need to be adjusted.
1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629

     - In offline mode because we don't want any network access or new
       software installed that isn't in the RPM.

     - Removing any find links.  This is largely due to a specific
       detail of our configurations.

     - Suppressing the creation of .installed.cfg

     - Specifying information for installing a special script that
       reads our centralized configuration database to configure the
       application after the RPM is installed.

   We have a pre-uninstall script that cleans up .pyc files.

   We specify the files to be installed. This is just the buildout
   directory and a configuration script.
Jim Fulton's avatar
Jim Fulton committed
1630 1631 1632 1633

Repeatability
=============

1634
We want to be able to check certain configuration into svn that can
Jim Fulton's avatar
Jim Fulton committed
1635 1636
be checked out and reproduced.

1637
- We let buildout tell what versions it picked for distributions
Jim Fulton's avatar
Jim Fulton committed
1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658

  - Run with -v
  
  - Look for outout lines of form:
  
    ::

      Picked: foo = 1.2

- Include a versions section:

  ::

    [buildout]
    ...
    versions = myversions

    [myversions]
    foo = 1.2
    ...
 
1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682
Deployment issues
=================

- Need a way to record the versions of eggs used.

- Need a way to generate distributable buildouts that contain all of the source
  distributions needed to build on a target machine (e.g. source
  RPMs).

- Need to be able to generate source distributions.  We need a way of
  gathering the sources used by a buildout so they can be distributed
  with it.

PyPI availability
=================

A fairly significant issue is the availability of PyPI. PyPI is
sometimes not available for minutes or hours at a time. This can cause
buildout to become unusable.

For more information
====================

See http://www.python.org/pypi/zc.buildout