Skip to content

S
slapos.buildout
Commit 7052070c authored by Jim Fulton's avatar Jim Fulton
Browse files
Options

Provide a way to express option values such that relative indentation 

is preserved.

Tighten config file syntax to allow fewer variations.
parent 64803254
Show whitespace changes
Inline Side-by-side
Showing with 245 additions and 116 deletions
src/zc/buildout/buildout.txt
View file @ 7052070c
......@@ -690,9 +690,84 @@ Now, we'll fix the typo again and we'll get the directories we expect:
Configuration file syntax
-------------------------
As mentioned earlier, buildout configuration files use the format
defined by the Python ConfigParser module with extensions. The
extensions are:
A buildout configuration file consists of a sequence of sections. A
section has a section header followed by 0 or more section options.
(Buildout configuration files may be viewed as a variation on INI
files.)
A section header consists of a section name enclosed in square braces.
A section name consists of one or more non-whitespace characters
other than square braces ('[', ']'), curly braces ('{', '}'), colons
(':') or equal signs ('='). Whitespace surrounding section names is ignored.
Options consist of option names, followed by optional space or tab
characters, an optional plus or minus sign and an equal signs and
values. An option value may be spread over multiple lines as long as
the lines after the first start with a whitespace character. An
option name consists of one or more non-whitespace characters other
than equal signs, square braces ("[", "]"), curly braces ("{", "}"),
plus signs or colons (":"). The option name '<' is reserved. An
option's data consists of the characters following the equal sign on
the start line, plus the continuation lines.
Option values have extra whitespace stripped. How this is done
depends on whether the value has non-whitespace characterts on the
first line. If an option value has non-whitespace characters on the
first line, then each line is stripped and blank lines are removed.
For exampe, in::
[foo]
bar = 1
baz = a
b
c
.. -> text
>>> import pprint, StringIO, zc.buildout.configparser
>>> pprint.pprint(zc.buildout.configparser.parse(StringIO.StringIO(
... text), 'test'))
{'foo': {'bar': '1', 'baz': 'a\nb\nc'}}
The value of of ``bar`` is ``'1'`` and the value of ``baz`` is
``'a\nb\nc'``.
If the first line of an option doesn't contain whitespace, then the
value is dedented (with ``textwrap.dedent``), trailing spaces in lines
are removed, and leading and trailing blank lines are removed. For
example, in::
[foo]
bar =
baz =
a
b
c
.. -> text
>>> pprint.pprint(zc.buildout.configparser.parse(StringIO.StringIO(
... text), 'test'))
{'foo': {'bar': '', 'baz': 'a\n b\n\nc'}}
The value of bar is ``''``, and the value of baz is ``'a\n b\n\nc'``.
Lines starting with '#' or ';' characters are comments. Comments can
also be placed after the closing square bracket (']') in a section header.
Buildout configuration data are Python strings, which are bytes in
Python 2 and unicode in Python 3.
Sections and options within sections may be repeated. Multiple
occurrences of of a section are treated as if they were concantinated.
The last option value for a given name in a section overrides previous
values.
In addition top the syntactic details above:
- option names are case sensitive
......@@ -702,18 +777,6 @@ extensions are:
- option values can be appended or removed using the - and +
operators.
The ConfigParser syntax is very flexible. Section names can contain
any characters other than newlines and right square braces ("]").
Option names can contain any characters other than newlines, colons,
and equal signs, can not start with a space, and don't include
trailing spaces.
It is likely that, in the future, some characters will be given
special buildout-defined meanings. This is already true of the
characters ":", "$", "%", "(", and ")". For now, it is a good idea to
keep section and option names simple, sticking to alphanumeric
characters, hyphens, and periods.
Annotated sections
------------------
......@@ -839,15 +902,15 @@ examples:
...
... [debug]
... recipe = recipes:debug
... File 1 = ${data-dir:path}/file
... File 2 = ${debug:File 1}/log
... File-1 = ${data-dir:path}/file
... File-2 = ${debug:File-1}/log
...
... [data-dir]
... recipe = recipes:mkdir
... path = mydata
... """)
We used a string-template substitution for File 1 and File 2. This
We used a string-template substitution for File-1 and File-2. This
type of substitution uses the string.Template syntax. Names
substituted are qualified option names, consisting of a section name
and option name joined by a colon.
......@@ -861,8 +924,8 @@ substituted.
Installing data-dir.
data-dir: Creating directory mydata
Installing debug.
File 1 /sample-buildout/mydata/file
File 2 /sample-buildout/mydata/file/log
File-1 /sample-buildout/mydata/file
File-2 /sample-buildout/mydata/file/log
recipe recipes:debug
Note that the substitution of the data-dir path option reflects the
......@@ -878,8 +941,8 @@ the buildout:
Develop: '/sample-buildout/recipes'
Updating data-dir.
Updating debug.
File 1 /sample-buildout/mydata/file
File 2 /sample-buildout/mydata/file/log
File-1 /sample-buildout/mydata/file
File-2 /sample-buildout/mydata/file/log
recipe recipes:debug
We can see that mydata was not recreated.
......@@ -904,8 +967,8 @@ _buildout_section_name_ to get the current section name.
...
... [debug]
... recipe = recipes:debug
... File 1 = ${data-dir:path}/file
... File 2 = ${:File 1}/log
... File-1 = ${data-dir:path}/file
... File-2 = ${:File-1}/log
... my_name = ${:_buildout_section_name_}
...
... [data-dir]
......@@ -918,8 +981,8 @@ _buildout_section_name_ to get the current section name.
Uninstalling debug.
Updating data-dir.
Installing debug.
File 1 /sample-buildout/mydata/file
File 2 /sample-buildout/mydata/file/log
File-1 /sample-buildout/mydata/file
File-2 /sample-buildout/mydata/file/log
my_name debug
recipe recipes:debug
......@@ -940,8 +1003,8 @@ example, we can leave data-dir out of the parts list:
...
... [debug]
... recipe = recipes:debug
... File 1 = ${data-dir:path}/file
... File 2 = ${debug:File 1}/log
... File-1 = ${data-dir:path}/file
... File-2 = ${debug:File-1}/log
...
... [data-dir]
... recipe = recipes:mkdir
......@@ -956,8 +1019,8 @@ It will still be treated as a part:
Uninstalling debug.
Updating data-dir.
Installing debug.
File 1 /sample-buildout/mydata/file
File 2 /sample-buildout/mydata/file/log
File-1 /sample-buildout/mydata/file
File-2 /sample-buildout/mydata/file/log
recipe recipes:debug
>>> cat('.installed.cfg') # doctest: +ELLIPSIS
......@@ -979,8 +1042,8 @@ the data-dir part after the debug part, it will be included before:
...
... [debug]
... recipe = recipes:debug
... File 1 = ${data-dir:path}/file
... File 2 = ${debug:File 1}/log
... File-1 = ${data-dir:path}/file
... File-2 = ${debug:File-1}/log
...
... [data-dir]
... recipe = recipes:mkdir
......@@ -994,8 +1057,8 @@ It will still be treated as a part:
Develop: '/sample-buildout/recipes'
Updating data-dir.
Updating debug.
File 1 /sample-buildout/mydata/file
File 2 /sample-buildout/mydata/file/log
File-1 /sample-buildout/mydata/file
File-2 /sample-buildout/mydata/file/log
recipe recipes:debug
>>> cat('.installed.cfg') # doctest: +ELLIPSIS
......
src/zc/buildout/configparser.py
View file @ 7052070c
......@@ -18,6 +18,7 @@
# - dict of dicts is a much simpler api
import re
import textwrap
class Error(Exception):
"""Base class for ConfigParser exceptions."""
......@@ -44,6 +45,8 @@ class Error(Exception):
def __repr__(self):
return self.message
__str__ = __repr__
class ParsingError(Error):
"""Raised when a configuration file does not follow legal syntax."""
......@@ -68,19 +71,13 @@ class MissingSectionHeaderError(ParsingError):
self.lineno = lineno
self.line = line
SECTCRE = re.compile(
r'\[' # [
r'(?P<header>[^]]+)' # very permissive!
r'\]' # ]
)
OPTCRE = re.compile(
r'(?P<option>[^:=\s][^:=]*)' # very permissive!
r'\s*(?P<vi>[:=])\s*' # any number of space/tab,
# followed by separator
# (either : or =), followed
# by any # space/tab
r'(?P<value>.*)$' # everything up to eol
)
section_header = re.compile(
r'\[\s*(?P<header>[^\s[\]:{}]+)\s*]\s*([#;].*)?$').match
option_start = re.compile(
r'(?P<name>[^\s{}[\]=:]+\s*[-+]?)'
r'='
r'(?P<value>.*)$').match
leading_blank_lines = re.compile(r"^(\s*\n)+")
def parse(fp, fpname):
"""Parse a sectioned setup file.
......@@ -92,63 +89,56 @@ def parse(fp, fpname):
leading whitespace. Blank lines, lines beginning with a '#',
and just about everything else are ignored.
"""
_sections = {}
sections = {}
cursect = None # None, or a dictionary
blockmode = None
optname = None
lineno = 0
e = None # None, or an exception
while True:
line = fp.readline()
if not line:
break
break # EOF
lineno = lineno + 1
# comment or blank line?
if line.strip() == '' or line[0] in '#;':
continue
if line.split(None, 1)[0].lower() == 'rem' and line[0] in "rR":
# no leading whitespace
continue
# continuation line?
if line[0] in '#;':
continue # comment
if line[0].isspace() and cursect is not None and optname:
value = line.strip()
if value:
cursect[optname] = "%s\n%s" % (cursect[optname], value)
# a section header or option header?
# continuation line
if blockmode:
line = line.rstrip()
else:
line = line.strip()
if not line:
continue
cursect[optname] = "%s\n%s" % (cursect[optname], line)
else:
# is it a section header?
mo = SECTCRE.match(line)
mo = section_header(line)
if mo:
# section header
sectname = mo.group('header')
if sectname in _sections:
cursect = _sections[sectname]
if sectname in sections:
cursect = sections[sectname]
else:
cursect = {}
_sections[sectname] = cursect
sections[sectname] = cursect = {}
# So sections can't start with a continuation line
optname = None
# no section header in the file?
elif cursect is None:
if not line.strip():
continue
# no section header in the file?
raise MissingSectionHeaderError(fpname, lineno, line)
# an option line?
else:
mo = OPTCRE.match(line)
mo = option_start(line)
if mo:
optname, vi, optval = mo.group('option', 'vi', 'value')
# This check is fine because the OPTCRE cannot
# match if it would set optval to None
if optval is not None:
if vi in ('=', ':') and ';' in optval:
# ';' is a comment delimiter only if it follows
# a spacing character
pos = optval.find(';')
if pos != -1 and optval[pos-1].isspace():
optval = optval[:pos]
optval = optval.strip()
# allow empty values
if optval == '""':
optval = ''
# option start line
optname, optval = mo.group('name', 'value')
optname = optname.rstrip()
optval = optval.strip()
cursect[optname] = optval
blockmode = not optval
else:
# a non-fatal parsing error occurred. set up the
# exception but keep going. the exception will be
......@@ -157,8 +147,17 @@ def parse(fp, fpname):
if not e:
e = ParsingError(fpname)
e.append(lineno, repr(line))
# if any parsing errors occurred, raise an exception
if e:
raise e
return _sections
for sectname in sections:
section = sections[sectname]
for name in section:
value = section[name]
if value[:1].isspace():
section[name] = leading_blank_lines.sub(
'', textwrap.dedent(value.rstrip()))
return sections
src/zc/buildout/configparser.test 0 → 100644
View file @ 7052070c
Some tests of the basic config-file parser:
First, an example that illustrates a well-formed configuration::
[s1]
a = 1
[ s2 ] # a comment
long = a
b
c
l2 =
a
# not a comment
# comment
; also a coment
b
c
empty =
c=1
b += 1
[s3]; comment
x = a b
.. -> text
>>> import pprint, StringIO, zc.buildout.configparser
>>> pprint.pprint(zc.buildout.configparser.parse(StringIO.StringIO(
... text), 'test'))
{'s1': {'a': '1'},
's2': {'b +': '1',
'c': '1',
'empty': '',
'l2': 'a\n\n\n# not a comment\n\n\nb\n\n c',
'long': 'a\nb\nc'},
's3': {'x': 'a b'}}
Here's an example with leading blank lines:
>>> text = '\n\n[buildout]\nz=1\n\n'
>>> pprint.pprint(zc.buildout.configparser.parse(StringIO.StringIO(
... text), 'test'))
{'buildout': {'z': '1'}}
Some examples that should error:
src/zc/buildout/tests.py
View file @ 7052070c
......@@ -15,6 +15,9 @@ from zc.buildout.buildout import print_
from zope.testing import renormalizing
import doctest
import manuel.capture
import manuel.doctest
import manuel.testing
import os
import pkg_resources
import re
......@@ -3015,10 +3018,11 @@ normalize_S = (
def test_suite():
test_suite = [
doctest.DocFileSuite(
'buildout.txt',
setUp=buildout_txt_setup,
tearDown=zc.buildout.testing.buildoutTearDown,
manuel.testing.TestSuite(
manuel.doctest.Manuel() + manuel.capture.Manuel(),
'configparser.test'),
manuel.testing.TestSuite(
manuel.doctest.Manuel(
checker=renormalizing.RENormalizing([
zc.buildout.testing.normalize_path,
zc.buildout.testing.normalize_endings,
......@@ -3050,6 +3054,10 @@ def test_suite():
(re.compile('distribute'), 'setuptools'),
(re.compile('Got zc.recipe.egg \S+'), 'Got zc.recipe.egg'),
])
) + manuel.capture.Manuel(),
'buildout.txt',
setUp=buildout_txt_setup,
tearDown=zc.buildout.testing.buildoutTearDown,
),
doctest.DocFileSuite(
'runsetup.txt', 'repeatable.txt', 'setup.txt',
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7