Most of the ccan Makefiles use $(RM) to remove files.  However, 'rm' is
traditionally considered one of the few shell tools which can be used in
Makefiles without indirecting via a variable.

rm is also typically invoked with -f in Makefiles, so that it doesn't cause
errors if the files don't exist (because they haven't been built).  A
number of instances in ccan were missing this.

This corrects these warts.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

At present, "make clean" will not remove the module-Makefile files for
non-top-level modules.  It also won't remove the generated config.h.
Correct those errors.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

Seems to want more stack.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Turns out that patterns with / cause % to match /.  OK...
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Useful if you have more than one object with same key.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

If we are to use should_grp_score_cos(x,y) as a filter the the following
relationship must hold (from least to most expensive):

        should_grp_score_len(x,y)
                >= should_grp_score_cos(x,y)
                >= grp_score(x)

should_grp_score_cos(x,y) wasn't holding up its part of the bargain, so
real data was used to generate a fudge curve to bring
should_grp_score_cos(x,y) results into the same space. Really this is a
terrible hack and the problem needs more thought. Evaluation of
should_grp_score_cos(x,y)'s performance benefit (given the relaxation of
the filter under the fudge curve) is sorely needed.

Distance metrics allow us to compare similarity results, however
applying the change leads to test suite breakage as we no longer satisfy
the requirement that each filter's score is at most as large as that of
the previous filter^. As such, also stop ccanlint from executing the
tests that are known to fail until we work around the problem.

^ This is a problem that has existed since the introduction of the
cosine similarity filter, it just wasn't detected by the test suite.

Ensure comparing filter results is sensible by using a consistent
calculation. Note that the cosine similarity measurement doesn't yet
conform and this can give spurious results that are not detected by the
test suite.

Likely this was optimised away, but the code now represents the intent.

Olaoluwa Osuntokun came up with an alternative which used binary trees;
that's a much better way to explain it, so do that in design.txt and
update the implementation to work the same way.

Anthony Towns pointed out that the numbering is the reverse of the normal
hash chaining descriptions, so fix that too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Using some serious macro magic, this patch extends generators to allow
them to take arbitrary arguments.  The arguments are marshalled into a
structure placed at the far end of the generator's stack when it is
created.  Then, they're unmarshalled back into C parameters when we first
context switch into the generator.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

Generators are a limited form of co-routine, which people may be familiar
with from Python.  This module adds an implementation of generators for C.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

This adds a new HAVE_UCONTEXT define, which indicates that ucontext.h
is present, and more-or-less works.

It also adds HAVE_POINTER_SAFE_MAKECONTEXT, which indicates whether
pointer valued arguments can be passed through the varargs parameters
to makecontext().
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

The chkfail() and chkok() macros in altstack's test program are pretty
difficult to read.  More importantly, though, they do all their tests with
one big ok1().  That means if the test fails, you get no indication which
of the checks was actually wrong, making debugging harder.

This reworks the macros into a more verbose form that's easier to read,
and splits them into multiple ok1() tests to make failures more explicit.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

altstack includes a couple of inline asm blocks with x86 push and pop
instructions.  These instructions will access memory (the stack), but
that's not declared in inline asm statement.  We seem to be getting away
with it, but in theory that could allow the compiler to re-order accesses
to local variables across the asm block.  Since those blocks change the
location of the stack, that could be very bad.

Adding a "memory" clobber should prevent this (effectively making the asm
blocks a compiler memory barrier).
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

ccan programs should always include config.h before anything else to make
sure everything is set up correctly.  Doing so in altstack's run.c means
it no longer needs an explicit _XOPEN_SOURCE 700, since _GNU_SOURCE is set
in config.h (for GNU libc, anyway).
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

At present the altstack module uses a stack of size MINSIGSTKSZ for its
SIGSEGV handler.  Although MINSIGSTKSZ is defined to be large enough to
execute a signal handler, it doesn't guarantee that you can do anything
very much within it.

With certain libc versions, MINSIGSTKSZ is not enough to execute the
longjmp() used in altstack.  Specfically, with Ubuntu 12.04 (the default
install for Travis containers), the first time longjmp() is executed the
symbol must be resolved by the dynamic linker in a process which overruns
the MINSIGSTKSZ sized stack.  That then corrupts local variables in
altstack() itself causing a number of subsequent failures.

This patch addresses the problem by changing from MINSIGSTKSZ to SIGSTKSZ
which is supposed to cover "the usual requirements for an alternate signal
stack".
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

* add altstack_remn, returns amount of stack remaining
* increase mapping by 1 page to handle abutment case
* capture rsp earlier
* align stack to 16 bytes
Signed-off-by: Dan Good <dan@dancancode.com>

Otherwise it describes what we need.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

This implements macros which iterate across their arguments.  This is
implemented in terms of (kinda sorta) recursion.  In fact, they will stop
working with enough arguments, but the limit is large and can be easily
increased by changing the depth of the CPPMAGIC_EVAL() macro.

There are 3 iterators (for now):
  CPPMAGIC_MAP
    applies another macro to each of its remaining arguments - the results
    are comma separated, so they can be passed into another CPPMAGIC_MAP
    invocation.
  CPPMAGIC_2MAP
    does the same thing, but takes the arguments a pair at a time, using
    a supplied two-argument macro.
  CPPMAGIC_JOIN
    combines the arguments with a chosen delimiter (effectively replacing
the commas between the arguments with the delimiter)
same thing, but takes the arguments a pair at a time.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

Recursion (and therefore iteration) in cpp is difficult, since the
preprocessor explicitly looks for and inhibits recursion.

But, it's possible to trick it, up to a point.  CPPMAGIC_DEFER1() and
CPPMAGIC_DEFER2() can "hide" a macro, preventing it from being expanded
and being noticed as recursion.

Along with that we need to cause extra expansion passes to be executed.
There has to be a finite limit here - true recursion is impossible - but
that number can be made very large pretty easily.  CPPMAGIC_EVAL() multiply
expands its argument(s) - up to 1024 times.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

Implement CPPMAGIC_IFELSE which operates similar to the C ? : operator, but
is evaluated at preprocessing time.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

In order to implement fancier things, we need to represent truth values in
cpp.  We use '0' and '1' strings, like in C, but we need ways to get these
values from other conditions.

CPPMAGIC_ISZERO() and CPPMAGIC_NONZERO() test if the argument is '0' or
anything else (ISZERO doubles as a logical not).

CPPMAGIC_ISEMPTY() and CPPMAGIC_NON_EMPTY() expand to 0 or 1 depending on
whether they have any arguments at all or not.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

A module for some of the awesome / horrifying techniques described at:
    http://jhnet.co.uk/articles/cpp_magic
    https://github.com/pfultz2/Cloak/wiki/C-Preprocessor-tricks,-tips,-and-idioms

Start off with just some simple things.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

At the moment when Travis runs make check it will stop on the first
failure.  That's not particularly useful, so add a -k so that all ccanlint
failures can be seen.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

When run in --summary mode ccanlint doesn't actually report whether it
passed or failed in the message .  In particular this means that when make
check is run with -j, it can be hard to tell which modules failed.

This adds a more obvious failure message.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

It returns EINVAL instead of ENOMEM for test/run.c line 96, then
complains on line 137:

==29368== Invalid read of size 4
==29368==    at 0x4033BC: main (run.c:137)
==29368==  Address 0x400000000018 is not stack'd, malloc'd or (recently) free'd
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Useful if you're going to reuse a buffer later.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Since we use -Wundef by default, ccanlint gets upset if __X86_64__ isn't set.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

If _info handles the arg "ported" it should print out 1 or 0; 0 means
it can't be compiled/run/tested on this platform.  This lets ccanlint
easily skip such modules.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

~0 will be signed and negative on any 2s complement system, and
left shifting a negative value has undefined behaviour in C.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

idtree.c:146 triggers a "comparison is always false" warning on some
compiler configurations, since the 'id' variable is unsigned.

Elsewhere in the module ids seem to be represented by (signed) ints, so
use the same convention here, suppressing the warning and also maybe being
more correct in other ways.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

The HTABLE_DEFINE_TYPE macro builds a type-specific hash table by
constructing a bunch of simple wrapper functions.  The user of the hash
table may not end up using all of these.  With gcc the fact that the
functions are inline stops an warnings about unused functions, but that's
not the case with clang.

Suppress these warnings by marking all the constructed functions except
for name##_add() as UNNEEDED (using the macro from ccan/compiler).  _add
is left alone on the grounds that a hash table you never add anything to
isn't much use, so this will help you to spot an entirely redundant
HTABLE_DEFINE_TYPE invocation.  *_init() would be a more obvious choice,
except that there is both *_init() and *_init_sized() and you only need
to use one of these.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

The usual way of construction strmap objects is to use the STRMAP_MEMBERS()
macro which expands to both a raw strmap structure and a tcon type canary.
However, the tcon type canary involves a flexible array member which means
that in standard C99 STRMAP_MEMBERS() must appear only at the end of a
structure definition.  But worse, that structure can then only appear at
the end of any other structure it is included in, which is pretty
inconvenient for the intended purpose of creating type specific strmaps.

gcc extensions allow this to work (somehow), but clang complains loudly
about it.

The tcon module already includes the TCON_WRAP() mechanism, which already
provides this same sort of type-specific definitions in a more general way.
So convert strmap (and its users) to that approach.

This removes STRMAP_MEMBERS() entirely, breaking compatibility.  I'm hoping
strmap is used in few enough places that we can get away with that.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>