Merge branch 'master' of https://git.erp5.org/repos/vifibnet

Conflicts:
	vifibnet.py

README

@@ -2,8 +2,8 @@ Vifibnet is a daemon setting up a resilient virtual private network over the
 internet
 
 HOW TO:
-    Vifibnet ( sic ) has three separate components : a setup, a server and
-    a client.
+    Vifibnet ( sic ) has three separate components : a setup (setup.py), a 
+    server (registry.py) and a client (vifibnet.py.
     Lambda users only have to launch the setup and then their client.
     The server is meant to be started once on a node which also will be running
     a client instance.

TODO

-G : We should focus on clearing the TODO List, then go to testing phase,
-    since the end is nearing... ( I finish on august, 3 )
-
 To be done :
-    Catch a more precise exception thant Exception at line 108 in vifibnet.py
-    ( UPnP forwarding )
-
     Upgrade the logging function in order to be able to log message like
     "Refreshing peers DB ... done", or add log messages to specify that an
     action advertised by a previous log message has been completed
@@ -14,6 +8,8 @@ To be done :
 
     Use an algorithm to choose which connections to keep and/or establish
     instead of pure randomness
+        |-> number of routes / tunnel : doesn't work
+        |-> favorise most used roads ?
 
     Replace comments at the beginning of functions with docstrings & give all
     fn docstrings
@@ -24,27 +20,12 @@ To be done :
     Use a timeout for the server peersDB so we can flag unreachable peers and
     remove the peers whose certificate is no longer valid
 
-    Specify a lease duration in ForwardViaUPnP
+    Specify a lease duration in ForwardViaUPnP and also forward a tcp port
 
     Handle LAN internally in order not to have catastrophic results ....
     ( avahi could be used )
 
 To be discussed:
-    Ideas for the name :
-          CON ( cloud overlay network ), cloudnet. From J.P -> already taken
-
-    G, J : To get traffic stats ( bytes in/out ), you can use
-          /sys/class/net/interface/statistics/rx_bytes, etc...
-        or /proc/net/dev/snmp6/interface ( all in one file ). This can be enough
-        if used as follows: get traffic diff from last time we checked in order
-        to choose which connection is significantly unused compared to others,
-        and close it. Of course, too recent connections (i.e. those for which we
-        have no previous stat) would be always kept.
-        This should be combined with routing table (i.e. how many nodes are
-        served by each tunnel), which is possibly redundant.
-        ip6tables should be avoided if possible.
-    U : Great !!!
-
     U : Babel seems to be very long to establish the routes : maybe we should
         tell him thant we are not on a wired network but on a mobile network ?
     G : babel establish routes quickly enough i'd say. There are two new
@@ -69,28 +50,13 @@ To be discussed:
         on nexedi's server downtime ? it could be useful for the internship
         rapport
 
-    U : The peer DB size should depend on the number of connection and the
-        refresh time
-    G : ?! I don't agree, the db size should be proportional ( with a factor
-        like 0.01 or less ) to the total number of peers in the entire network,
-        with maybe a max size.
-    U : what we need to do  is to keep the randomness. For this, we need a big
-        enought DB to ensure we can still choose a peer as if it was choosen
-        directly from the server. The requiered db size can be calculated from
-        the number of connections and the refresh time.
-    G : ok, you can erase this talk
-
-    U : Why are --ip and internal-port mutually exclusive ?
-        Currently upnp only forward via UDP. Should he also forward via TCP ?
-        Why dont we only use UDP ?
-        No error should be raised when no upnp is detected : we should allow
-        machines having public IP to do an automatic configuration using the
-        discovery by an other peer
-    G : Actually, i was wrong, --ip and internal-port are no longer exclusive
-        Julien said udp might not be used by some people because of
-        restrictions imposed by the ISP ( FAI in french ), so we should
-        allow both, and act according to the options specifying which servers
-        to start (upd, tcp-server)
-
     G : I think the number of route going through an interface should be a
         Connection attribute, not a dict in tunnelManager
+    U : Yes, it was planned, just wait for me to finish implementing it
+
+    U :  '--up', 'ovpn-server %s/%u' % (server_ip, len(network)) in plib.py
+        if you use len(network), this means that all our network is on the 
+        same LAN and that the interface of the server is connected to it
+        wich means that any packet should be routed to this interface
+        an interface should only advertise the /64 (or less) which has been 
+        attributed to it

plib.py

@@ -22,7 +22,7 @@ def server(server_ip, ip_length, max_clients, dh_path, pipe_fd, port, proto, hel
     return openvpn(hello_interval,
         '--tls-server',
         '--mode', 'server',
-        '--up', 'ovpn-server %s/%u' % (server_ip, ip_length),
+        '--up', 'ovpn-server %s/%u' % (server_ip, 64), 
         '--client-connect', 'ovpn-server ' + str(pipe_fd),
         '--client-disconnect', 'ovpn-server ' + str(pipe_fd),
         '--dh', dh_path,

simulation/graph.cpp

@@ -2,31 +2,64 @@
 #include <cmath>
 #include <map>
 #include <queue>
+#include <stack>
+
+void erase(vector<int>& v, int value)
+{
+    for(int i=0;;i++)
+        if(v[i] == value)
+        {
+            v[i] = v.back();
+            v.pop_back();
+            break;
+        }
+}
 
 Graph::Graph(int size, int k, int maxPeers, mt19937& rng) :
     distrib(uniform_int_distribution<int>(0, size-1)),
-    size(size), generator(rng)
+    size(size), generator(rng), k(k), maxPeers(maxPeers)
 {
     adjacency = new vector<int>[size];
+    generated = new vector<int>[size];
+
     for(int i=0; i<size; i++)
     {
+        int otherNode;
         unordered_set<int> alreadyConnected;
         alreadyConnected.insert(i);
 
         for(int j=0; j<k; j++)
         {
-            int otherNode;
-
-            while(alreadyConnected.count(otherNode = distrib(rng)) == 1
-                || otherNode > i && adjacency[otherNode].size() > maxPeers-10
+            while(alreadyConnected.count(otherNode = distrib(generator)) >= 1
+                || otherNode > i && adjacency[otherNode].size() > maxPeers-k
                 || adjacency[otherNode].size() > maxPeers)
             { }
             adjacency[i].push_back(otherNode);
+            generated[i].push_back(otherNode);
             adjacency[otherNode].push_back(i);
         }
     }
 }
 
+int Graph::AddEdge(int from, unordered_set<int>& alreadyConnected)
+{
+    int otherNode;
+    while(alreadyConnected.count(otherNode = distrib(generator)) >= 1
+        || (adjacency[otherNode].size() > maxPeers ))
+    { }
+    adjacency[from].push_back(otherNode);
+    generated[from].push_back(otherNode);
+    adjacency[otherNode].push_back(from);
+    return otherNode;
+}
+
+int Graph::RemoveEdge(int from, int to)
+{
+    erase(generated[from], to);
+    erase(adjacency[from], to);
+    erase(adjacency[to], from);
+}
+
 void Graph::GetDistancesFrom(int node, int* distance)
 {
     for(int j=0; j<size; j++)
@@ -50,6 +83,54 @@ void Graph::GetDistancesFrom(int node, int* distance)
     }
 }
 
+void Graph::GetRoutesFrom(int node, int* distance, float* routesCount)
+{
+    
+    unordered_set<int> prevs[size];
+
+    for(int i=0; i<size; i++)
+    {
+        distance[i] = -1;
+        routesCount[i] = 1;
+    }
+    distance[node] = 0;
+
+    queue<int> remainingNodes;
+    remainingNodes.push(node);
+
+    stack<int> order;
+
+    // Get the order
+    while(!remainingNodes.empty())
+    {
+        int node = remainingNodes.front();
+        int d = distance[node];
+        remainingNodes.pop();
+        order.push(node);
+
+        for(int neighbor : adjacency[node])
+            if(distance[neighbor] == -1)
+            {
+                distance[neighbor] = d+1;
+                prevs[neighbor].insert(node);
+                remainingNodes.push(neighbor);
+            }
+            else if(distance[neighbor] == d+1)
+                prevs[neighbor].insert(node);
+    }
+
+    // get the BC
+    while(!order.empty())
+    {
+        int node = order.top();
+        order.pop();
+        float w = routesCount[node];
+        for(int i : prevs[node])
+            routesCount[i] += w;
+    }
+
+}
+
 int Graph::CountUnreachableFrom(int node)
 {
     bool accessibility[size];

simulation/main.cpp

-// To compile : g++ -std=c++0x results.cpp graph.cpp main.cpp -lpthread
+// To compile : g++ -std=c++0x results.cpp graph.cpp main.cpp -lpthread 
 #include "main.h"
 #include <future>
 #include <sstream>
@@ -14,8 +14,8 @@ Results Simulate(int seed,  int n, int k, int maxPeer, int maxDistanceFrom, floa
     for(int r=0; r<runs; r++)
     {
         Graph graph(n, k, maxPeer, rng);
-        graph.KillMachines(alivePercent);
-        results.AddAccessibilitySample(((double)graph.CountUnreachableFrom(0))/((double)n));
+        //graph.KillMachines(alivePercent);
+        //results.AddAccessibilitySample(((double)graph.CountUnreachableFrom(0))/((double)n));
         //int minCut = graph.GetMinCut();
         //if(results.minKConnexity == -1 || results.minKConnexity > minCut)
         //results.minKConnexity = minCut;
@@ -28,6 +28,56 @@ Results Simulate(int seed,  int n, int k, int maxPeer, int maxDistanceFrom, floa
             graph.GetDistancesFrom(i, distance);
             results.UpdateDistance(distance, graph.size);
         }*/
+
+        int distance[graph.size];
+        float routesCount[graph.size];
+        int nRefresh = 1;
+
+        graph.GetDistancesFrom(0, distance);
+        double moy = 0;
+        for(int i=0; i<graph.size; i++)
+            moy += distance[i];
+        moy /= graph.size;
+        cout << "Avg distance : " << moy << endl; cout.flush();
+
+        for(int i = 0; i<100; i++)
+        {
+            for(int j=0; j<graph.size; j++)
+            {
+                graph.GetRoutesFrom(j, distance, routesCount);
+                unordered_set<int> alreadyConnected;
+
+                // erase some edge
+                for(int k=0; k<nRefresh; k++)
+                {
+                    int minNode = -1;
+                    int minimum = -1;
+                    for(int index = 0; index < graph.generated[j].size(); index++)
+                        if(minNode == -1 || routesCount[graph.generated[j][index]] < minimum)
+                        {
+                            minNode = graph.generated[j][index];
+                            minimum = routesCount[minNode];
+                        }
+
+                    graph.RemoveEdge(j, minNode);
+                }
+
+                // Add new edges
+                alreadyConnected.insert(j);
+                for(int k : graph.adjacency[j])
+                    alreadyConnected.insert(k);
+
+                for(int k=0; k<nRefresh; k++)
+                    alreadyConnected.insert(graph.AddEdge(j, alreadyConnected));
+            }
+
+            graph.GetDistancesFrom(0, distance);
+            moy = 0;
+            for(int i=0; i<graph.size; i++)
+                moy += distance[i];
+            moy /= graph.size;
+            cout << "Avg distance : " << moy << endl;
+        }
     }
 
     results.Finalise();
@@ -46,12 +96,12 @@ int main(int argc, char** argv)
     vector<future<string>> outputStrings;
     for(int n=2000; n<=2000; n*=2)
         for(int k=10; k<=10; k+=5)
-            for(float a=0.01; a<=1; a+=0.05)
+            for(float a=1; a<=1; a+=0.05)
             {
                 int seed = rng();
                 outputStrings.push_back(async(launch::async, [seed, n, k, a]()
                     {
-                        Results results = Simulate(seed, n, k, 3*k, 10000, a, 1);
+                        Results results = Simulate(seed, n, k, 2.5*k, 10000, a, 1);
                         ostringstream out;
                         out << n << "," << k << "," << a << "," << 3*k << ","
                             << results.avgDistance << ","

simulation/main.h

@@ -35,22 +35,28 @@ class Graph
 {
 public:
     Graph(int size, int k, int maxPeers, mt19937& rng);
-    ~Graph() { delete[] adjacency; };
+    ~Graph() { delete[] adjacency; delete[] generated; };
 
     void GetDistancesFrom(int node, int* distance);
+    int AddEdge(int from, unordered_set<int>& alreadyConnected);
+    int RemoveEdge(int from, int to);
     int GetMinCut();
     int CountUnreachableFrom(int node);
+    void GetRoutesFrom(int node, int* distance, float* routesCount);
 
     void KillMachines(float proportion);
     //void SplitAS(float proportionAS1, float proportionAS2);
 
     vector<int>* adjacency;
+    vector<int>* generated;
     int size;
 private:
     int GetMinCut(MinCutGraph& graph);
 
     uniform_int_distribution<int> distrib;
     mt19937& generator;
+    int maxPeers;
+    int k;
 };
 
 class Results

tunnel.py

@@ -14,6 +14,7 @@ class Connection:
                 os.O_WRONLY|os.O_CREAT|os.O_TRUNC))
 
         self.iface = iface
+        self.routes = 0
         self._prefix = prefix
         self._creation_date = time.time()
         self._bandwidth = None
@@ -64,7 +65,7 @@ class TunnelManager:
         self._write_pipe = write_pipe
         self._peer_db = peer_db
         self._connection_dict = {}
-        self._route_count = {}
+        self._iface_to_prefix = {}
         self._ovpn_args = openvpn_args
         self._hello = hello_interval
         self._refresh_time = refresh
@@ -87,6 +88,7 @@ class TunnelManager:
 
     def _cleanDeads(self):
         for prefix in self._connection_dict.keys():
+
             if not self._connection_dict[prefix].refresh():
                 self._kill(prefix)
 
@@ -106,7 +108,7 @@ class TunnelManager:
             pass
         self.free_interface_set.add(connection.iface)
         self._peer_db.unusePeer(prefix)
-        del self._route_count[connection.iface]
+        del self._iface_to_prefix[connection.iface]
 
     def _makeNewTunnels(self):
         utils.log('Trying to make %i new tunnels' %
@@ -119,7 +121,7 @@ class TunnelManager:
                 self._connection_dict[prefix] = Connection(address,
                         self._write_pipe, self._hello, iface,
                         prefix, self._ovpn_args)
-                self._route_count[iface] = 0
+                self._iface_to_prefix[iface] = prefix
                 self._peer_db.usePeer(prefix)
         except KeyError:
             utils.log("""Can't establish connection with %s
@@ -129,16 +131,18 @@ class TunnelManager:
 
     def _countRoutes(self):
         utils.log('Starting to count the routes on each interface', 3)
-        for iface in self._route_count.keys():
-            self._route_count[iface] = 0
+        for iface in self._iface_to_prefix.keys():
+            self._connection_dict[self._iface_to_prefix[iface]].routes = 0
         f = open('/proc/net/ipv6_route', 'r')
         for line in f:
-            ip, subnet_size, iface = struct.unpack("""32s x 2s x 32x x 2x x 
-                    32x x 8x x 8x x 8x x 8x x %ss x""" % (len(line)-142), line)
+            ip, subnet_size, iface = struct.unpack("""32s x 2s 106x 
+                %ss x""" % (len(line)-142), line)
             iface = iface.replace(' ', '')
-            if iface in self._route_count.keys():
-                self._route_count[iface] += 1
-        for iface in self._route_count.keys():
-            utils.log('Routes on iface %s : %s' % (iface,self._route_count[iface] ), 5)
+            if iface in self._iface_to_prefix.keys():
+                self._connection_dict[self._iface_to_prefix[iface]].routes += 1
+        for p in self._connection_dict.keys():
+            utils.log('Routes on iface %s : %s' % (
+                self._connection_dict[p].iface,
+                self._connection_dict[p].routes ), 5)
 
 

upnpigd.py

@@ -2,17 +2,32 @@ import miniupnpc
 import socket
 
 # return (address, port)
-def ForwardViaUPnP(local_port):
+def ForwardViaUPnP(local_port, protos):
     u = miniupnpc.UPnP()
     u.discoverdelay = 200
     u.discover()
     u.selectigd()
-    external_port = 1194
+    external_port = 1000
+
     while True:
-        while u.getspecificportmapping(external_port, 'UDP') != None:
-            external_port = max(externalPort + 1, 49152)
-            if external_port == 65536:
-                raise Exception
-        if u.addportmapping(external_port, 'UDP', u.lanaddr, local_port, 'Vifib openvpn server', ''):
-            return (u.externalipaddress(), external_port)
+        if 'udp' in protos:
+            while u.getspecificportmapping(external_port, 'UDP') != None :
+                external_port += 1
+                if external_port == 65536:
+                    raise Exception
+        if 'tcp-server' in protos:
+            while u.getspecificportmapping(external_port, 'TCP') != None :
+                external_port += 1
+                if external_port == 65536:
+                    raise Exception
+
+        if 'udp' in protos:
+            u.addportmapping(external_port, 'UDP', u.lanaddr, local_port,
+                'Vifib openvpn server', '')
+        if 'tcp-server' in protos:
+            u.addportmapping(external_port, 'TCP', u.lanaddr, local_port, 
+                'Vifib openvpn server', '')
+
+        print (u.externalipaddress(), external_port)
+        return (u.externalipaddress(), external_port)
 

vifibnet.py

@@ -101,8 +101,8 @@ def main():
     else:
         utils.log('Attempting automatic configuration via UPnP', 4)
         try:
-            ext_ip, ext_port = upnpigd.ForwardViaUPnP(config.internal_port)
-            config.address = list([ext_ip, ext_port, proto]
+            ext_ip, ext_port = upnpigd.ForwardViaUPnP(config.internal_port, config.proto)
+            config.address = list([ext_ip, str(ext_port), proto]
                                   for proto in config.proto)
         except Exception:
             utils.log('An atempt to forward a port via UPnP failed', 4)