Commit 81845140 authored by Tom Niget's avatar Tom Niget

Add runtime submodule, move stdlib, add README

parent 968bedc0
[submodule "runtime"]
path = runtime
url = https://lab.nexedi.com/typon/typon-concurrency
# (*Working Title*) Typon # Typon compiler
Typon is a three-part project to bring practical GIL-free concurrency to Python: This repository hosts the source code of the Typon compiler and standard library.
1. (*in progress*) [`typon/rt`](#typonrt-a-concurrency-runtime), a C++ concurrency runtime
2. (*not started*) [`typon/compiler`](#typoncompiler-a-python-to-c-compiler), a compiler from Python syntax into C++
3. (*not started*) [`typon/bindings`](#typonbindings-pythonc-bindings-for-typon), C++/Python bindings for interoperability with actual Python
## Directory structure
## `typon/rt`, A Concurrency Runtime - `docs`: documentation
- `include`: standard library and compiler primitives/intrisics
- `runtime`: concurrency runtime (Git submodule)
- `trans`: transpiler
A continuation-stealing concurrency runtime using cutting-edge C++20 coroutines, ## Development setup
featuring both `fork`/`sync` structured concurrency and `future`-based unbounded
concurrency.
Install the dependencies using `pip3 install -r requirements.txt`.
### Status ## Test harness
- [x] structured concurrency with `fork`/`sync`
- [x] systematic exception propagation from `fork`ed tasks
- [x] unbounded concurrency with the `future` primitive
- [x] asynchronous waiting on `Future` and `Promise` objects
- [x] mutexes (with asynchronous blocking)
- [x] asynchronous I/O (with [liburing](https://github.com/axboe/liburing))
- [ ] task and I/O cancellation
- [ ] channels
### `fork`/`sync`, Structured Concurrency
```C++
using namespace typon;
Join<int> fibo(int n) {
if (n < 2) {
co_return n;
}
// Start two potentially concurrent tasks
Forked a = co_await fork(fibo(n - 1));
Forked b = co_await fork(fibo(n - 2));
// Wait until they both complete
co_await Sync();
// Access the results
co_return a.get() + b.get();
}
```
[Structured concurrency](https://en.wikipedia.org/wiki/Structured_concurrency)
is a concurrency paradigm where parallel execution paths are always joined
before exiting the function which created them.
It makes it possible to reason locally about concurrent code, since concurrent
tasks will not outlive the current function. This makes concurrent code more
intuitive and readable, and makes it much easier to reason about memory safety
and thread safety. It also enables systematic exception propagation from
spawned tasks to the parent scope.
The `fork`/`sync` primitives mirror the `spawn`/`sync` paradigm of [Cilk](http://cilk.mit.edu/).
`fork` introduces a potentially parallel nested task. `sync` waits for all
the tasks forked so far in the current scope. `fork`/`sync` can only be used
inside a `Join` coroutine, which guarantees all forked tasks complete before
the parent `Join` coroutine returns, even when there is no explicit `sync`.
### Exception Propagation with `fork`/`sync`
When an exception escapes out of a `fork`, execution of the body of the `Join`
coroutine may have already resumed concurrently and continued past the point
where the `fork` occured.
Therefore propagating an exception from a `fork` is not as easy as in the case
of a simple function call: first the body of the`Join` coroutine must reach a
point where it can stop executing, and then all concurrent forks must complete.
The `Join` coroutine may stop executing at the site of the original `fork` (if
the continuation has not been resumed concurrently), or at any subsequent call
to `fork` after the exception occurs, or at the latest at the next explicit or
implicit `sync`.
Once all concurrent forks have completed, execution jumps directly to the call
site of the `Join` coroutine, where the exception is immediately rethrown as if
the `Join` coroutine had thrown the exception itself.
There is no way to catch that exception directly inside the body of the `Join`
coroutine.
```C++
using namespace typon;
Task<void> throw_exception() {
// ...
throw std::exception();
co_return; // so that this is a coroutine
}
Join<void> parallel() {
co_await fork(throw_exception());
// 1. execution may resume concurrently here, or jump straight to 6.
for (int i = 0; i < 5; i++) {
// 2. execution may stop abruptly here and jump straight to 6.
co_await fork(some_task());
}
// 3. execution might reach this point
co_await Sync(); // 4. after this, execution will jump to 6.
// 5. execution will never reach here
}
Task<void> caller() {
// ...
co_await parallel(); // 6. exception is rethrown here
// ...
}
```
### `future`, A Primitive for Unbounded Concurrency
```
using namespace typon;
Task<int> fibo(int n) {
if (n < 2) {
co_return n;
}
// Start two potentially concurrent tasks
Future a = co_await future(fibo(n - 1));
Future b = co_await future(fibo(n - 2));
// Wait for each future and retrieve the results
co_return co_await a.get() + co_await b.get();
}
```
In cases when structured concurrency is too constraining, the `future` primitive
creates concurrent tasks that may outlive the parent scope, more like the `go`
statement in Go. Unlike `go`, `future` immediately returns a `Future` object
that can be used to wait for the task to complete. If the `Future` object is
not used, the task becomes "detached" and lives independantly.
### Examples
See `rt/examples`.
### Using `rt/typon`
`typon/rt` is a headers-only library, so no preliminary build step is required.
Programs only need to include `rt/include/typon/typon.hpp`.
Compiling requires a compiler supporting C++20, such as `gcc++-11` or
`clang++-14` or more recent.
See `rt/examples` for compilation flags.
### References
##### Structured Concurrency
- Nathaniel J. Smith's [Notes on structured concurrency](https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/)
- Martin Sústrik's blog articles on [structured concurrency](https://250bpm.com/blog:137/index.html)
- Lewis Baker's talk on [structured concurrency with C++20 coroutines](https://www.youtube.com/watch?v=1Wy5sq3s2rg)
##### C++20 Coroutines
- Lewis Baker's [Asymmetric Transfer blog on C++20 coroutines](https://lewissbaker.github.io/)
- [C++20 coroutines at cppreference.com](https://en.cppreference.com/w/cpp/language/coroutines)
##### Papers
- N. S. Arora, R. D. Blumofe, and C. G. Plaxton. 1998.
Thread scheduling for multiprogrammed multiprocessors.
https://doi.org/10.1145/277651.277678
- D. Chase and Y. Lev. 2005.
Dynamic circular work-stealing deque.
https://doi.org/10.1145/1073970.1073974
- N. M. Lê, A. Pop, A. Cohen, and F. Zappa Nardelli. 2013.
Correct and efficient work-stealing for weak memory models.
https://doi.org/10.1145/2517327.2442524
- Kyle Singer, Yifan Xu, and I-Ting Angelina Lee. 2019.
Proactive work stealing for futures.
https://doi.org/10.1145/3293883.3295735
- C. X. Lin, T .W Huang, and M. D. F. Wong. 2020.
An efficient work-stealing scheduler for task dependency graph.
https://tsung-wei-huang.github.io/papers/icpads20.pdf
- F. Schmaus, N. Pfeiffer, W. Schröder-Preikschat, T. Hönig, and J. Nolte.
2021. Nowa: a wait-free continuation-stealing concurrency platform.
https://www4.cs.fau.de/Publications/2021/schmaus2021nowa.pdf
## `typon/compiler`, A Python to C++ compiler
### Status
This part has not been started yet.
### Driving Idea
Several projects show that it is quite feasible to use Python's `ast` module
to write compilers that take Python syntax as input language. The abstract
syntax tree can then be used to generate C++ output code. The compiler may
freely reinterpret the input syntax and give it different semantic meanings
than in the original Python. As such it is a basis to create an independent
language with a Python-compatible syntax.
In particular such a compiler may use Python's `type hints` syntax for type
annotations to introduce static typing in the language and generate required
typing annotations in the C++ output.
### References
##### Language Design
- Abilian's [design note on syntax for a Python-derived language](https://github.com/abilian/cythonplus-sandbox/blob/devel/sandbox/comparison.md)
##### Technical References
- Python's [`ast` module](https://docs.python.org/3/library/ast.html)
- [PEP 484 - Type Hints](https://peps.python.org/pep-0484/)
- [PEP 526 – Syntax for Variable Annotations](https://peps.python.org/pep-0526/)
##### Related Works
- Lukas Martinelli's [`py14` project](https://github.com/lukasmartinelli/py14)
- [The `py2many` project](https://github.com/py2many/py2many)
- [`mypyc`](https://github.com/mypyc/mypyc)
- [`mycpp`, a Python to C++ translator](https://www.oilshell.org/blog/2022/05/mycpp.html)
## `typon/bindings`, Python/C++ bindings for Typon
### Status
This part has not been started yet.
### Driving Idea
The previous part `typon/compiler`, aims to produce a language that takes
Python syntax as input, but is otherwise completely independent of Python.
Using Python/C++ bindings, this language can be made to be interoperable
with true Python, allowing it to be called from standard Python and to call
standard Python code.
Using the first part `typon/rt`, concurrent code may acquire the Python GIL
asynchronously so as not to block the underlying worker thread, in order to
safely call standard Python code.
### References
##### Technical References
- [The Python Global Interpreter Lock](https://wiki.python.org/moin/GlobalInterpreterLock)
##### Related Works
- [`nanobind`, a C++17/Python bindings library](https://github.com/wjakob/nanobind)
- [`pybind11`, a C++11/Python bindings library](https://github.com/pybind/pybind11)
- [Cython](https://cython.org/)
- [The Cython+ project](https://www.cython.plus/)
- [`mypyc`](https://github.com/mypyc/mypyc)
From inside the `trans` directory, run `python3 test_runner.py`.
\ No newline at end of file
// //
// Created by Tom on 24/03/2023. // Created by Tom on 24/03/2023.
// //
#ifndef TYPON_BASEDEF_HPP #ifndef TYPON_BASEDEF_HPP
#define TYPON_BASEDEF_HPP #define TYPON_BASEDEF_HPP
template<typename Self> template<typename Self>
class TyBuiltin { class TyBuiltin {
template <typename... Args> template <typename... Args>
auto sync_wrapper(Args&&... args) auto sync_wrapper(Args&&... args)
{ {
return static_cast<Self*>(this)->sync(std::forward<Args>(args)...); return static_cast<Self*>(this)->sync(std::forward<Args>(args)...);
} }
public: public:
template <typename... Args> template <typename... Args>
auto operator()(Args&&... args) -> decltype(sync_wrapper(std::forward<Args>(args)...)) auto operator()(Args&&... args) -> decltype(sync_wrapper(std::forward<Args>(args)...))
{ {
return sync_wrapper(std::forward<Args>(args)...); return sync_wrapper(std::forward<Args>(args)...);
} }
}; };
struct method {}; struct method {};
template <typename Func, typename Self> template <typename Func, typename Self>
struct boundmethod { struct boundmethod {
[[no_unique_address]] Func func; [[no_unique_address]] Func func;
Self self; Self self;
boundmethod(Func func, Self self) : func(func), self(self) {} boundmethod(Func func, Self self) : func(func), self(self) {}
template <typename... Args> template <typename... Args>
auto operator()(Args &&... args) const { auto operator()(Args &&... args) const {
return func(self, std::forward<Args>(args)...); return func(self, std::forward<Args>(args)...);
} }
}; };
template <typename Obj, std::derived_from<method> Attr> template <typename Obj, std::derived_from<method> Attr>
auto dot_bind(Obj obj, Attr attr) { auto dot_bind(Obj obj, Attr attr) {
return boundmethod(attr, obj); return boundmethod(attr, obj);
} }
template <typename Obj, typename Attr> template <typename Obj, typename Attr>
requires (! std::derived_from<Attr, method>) requires (! std::derived_from<Attr, method>)
auto dot_bind(Obj, Attr attr) { auto dot_bind(Obj, Attr attr) {
return attr; return attr;
} }
#define dot(OBJ, NAME) [](auto && obj) -> auto { return dot_bind(obj, obj.NAME); }(OBJ) #define dot(OBJ, NAME) [](auto && obj) -> auto { return dot_bind(obj, obj.NAME); }(OBJ)
#define dotp(OBJ, NAME) [](auto && obj) -> auto { return dot_bind(obj, obj->NAME); }(OBJ) #define dotp(OBJ, NAME) [](auto && obj) -> auto { return dot_bind(obj, obj->NAME); }(OBJ)
#define dots(OBJ, NAME) [](auto && obj) -> auto { return std::remove_reference<decltype(obj)>::type::py_type::NAME; }(OBJ) #define dots(OBJ, NAME) [](auto && obj) -> auto { return std::remove_reference<decltype(obj)>::type::py_type::NAME; }(OBJ)
#endif // TYPON_BASEDEF_HPP #endif // TYPON_BASEDEF_HPP
// //
// Created by Tom on 08/03/2023. // Created by Tom on 08/03/2023.
// //
#ifndef TYPON_BOOL_HPP #ifndef TYPON_BOOL_HPP
#define TYPON_BOOL_HPP #define TYPON_BOOL_HPP
#include <ostream> #include <ostream>
#include "print.hpp" #include "print.hpp"
template <> void print_to<bool>(const bool &x, std::ostream &s) { template <> void print_to<bool>(const bool &x, std::ostream &s) {
s << (x ? "True" : "False"); s << (x ? "True" : "False");
} }
#endif // TYPON_BOOL_HPP #endif // TYPON_BOOL_HPP
// //
// Created by Tom on 08/03/2023. // Created by Tom on 08/03/2023.
// //
#ifndef TYPON_COMPLEX_HPP #ifndef TYPON_COMPLEX_HPP
#define TYPON_COMPLEX_HPP #define TYPON_COMPLEX_HPP
#include <complex> #include <complex>
#include <ostream> #include <ostream>
#include "print.hpp" #include "print.hpp"
using PyComplex = std::complex<double>; using PyComplex = std::complex<double>;
PyComplex operator+(int a, const PyComplex &b) { return PyComplex(a) + b; } PyComplex operator+(int a, const PyComplex &b) { return PyComplex(a) + b; }
PyComplex operator-(int a, const PyComplex &b) { return PyComplex(a) - b; } PyComplex operator-(int a, const PyComplex &b) { return PyComplex(a) - b; }
template <> void print_to<PyComplex>(const PyComplex &x, std::ostream &s) { template <> void print_to<PyComplex>(const PyComplex &x, std::ostream &s) {
if (x.real() == 0) { if (x.real() == 0) {
s << x.imag() << "j"; s << x.imag() << "j";
} else { } else {
s << '(' << x.real() << "+" << x.imag() << "j)"; s << '(' << x.real() << "+" << x.imag() << "j)";
} }
} }
#endif // TYPON_COMPLEX_HPP #endif // TYPON_COMPLEX_HPP
// //
// Created by Tom on 08/03/2023. // Created by Tom on 08/03/2023.
// //
#ifndef TYPON_DICT_HPP #ifndef TYPON_DICT_HPP
#define TYPON_DICT_HPP #define TYPON_DICT_HPP
#include <unordered_map> #include <unordered_map>
/* /*
template <typename K, typename V> template <typename K, typename V>
class PyDict : public std::unordered_map<K, V> { class PyDict : public std::unordered_map<K, V> {
public: public:
PyDict(std::unordered_map<K, V> &&m) PyDict(std::unordered_map<K, V> &&m)
: std::unordered_map<K, V>(std::move(m)) {} : std::unordered_map<K, V>(std::move(m)) {}
PyDict(std::initializer_list<std::pair<const K, V>> m) PyDict(std::initializer_list<std::pair<const K, V>> m)
: std::unordered_map<K, V>(m) {} : std::unordered_map<K, V>(m) {}
operator std::unordered_map<K, V>() const { operator std::unordered_map<K, V>() const {
return std::unordered_map<K, V>(this->begin(), this->end()); return std::unordered_map<K, V>(this->begin(), this->end());
} }
operator std::unordered_map<K, V> &() { operator std::unordered_map<K, V> &() {
return *reinterpret_cast<std::unordered_map<K, V> *>(this); return *reinterpret_cast<std::unordered_map<K, V> *>(this);
} }
std::size_t py_len() const { return this->size(); } std::size_t py_len() const { return this->size(); }
bool py_contains(const K &k) const { return this->find(k) != this->end(); } bool py_contains(const K &k) const { return this->find(k) != this->end(); }
class iterator { class iterator {
public: public:
using value_type = std::pair<K, V>; using value_type = std::pair<K, V>;
using difference_type = std::ptrdiff_t; using difference_type = std::ptrdiff_t;
using pointer = std::pair<K, V> *; using pointer = std::pair<K, V> *;
using reference = std::pair<K, V> &; using reference = std::pair<K, V> &;
using iterator_category = std::forward_iterator_tag; using iterator_category = std::forward_iterator_tag;
iterator(typename std::unordered_map<K, V>::iterator it) : _it(it) {} iterator(typename std::unordered_map<K, V>::iterator it) : _it(it) {}
iterator &operator++() { iterator &operator++() {
_it++; _it++;
return *this; return *this;
} }
iterator operator++(int) { iterator operator++(int) {
iterator tmp = *this; iterator tmp = *this;
_it++; _it++;
return tmp; return tmp;
} }
bool operator==(const iterator &rhs) const { return _it == rhs._it; } bool operator==(const iterator &rhs) const { return _it == rhs._it; }
bool operator!=(const iterator &rhs) const { return _it != rhs._it; } bool operator!=(const iterator &rhs) const { return _it != rhs._it; }
const std::pair<K, V> &operator*() const { return *_it; } const std::pair<K, V> &operator*() const { return *_it; }
const std::pair<K, V> *operator->() const { return &*_it; } const std::pair<K, V> *operator->() const { return &*_it; }
private: private:
typename std::unordered_map<K, V>::iterator _it; typename std::unordered_map<K, V>::iterator _it;
}; };
iterator py_iter() const { return this->begin(); } iterator py_iter() const { return this->begin(); }
void py_print(std::ostream &s) const { void py_print(std::ostream &s) const {
s << '{'; s << '{';
if (this->size() > 0) { if (this->size() > 0) {
print_to(this->begin()->first, s); print_to(this->begin()->first, s);
s << ": "; s << ": ";
print_to(this->begin()->second, s); print_to(this->begin()->second, s);
for (auto it = ++this->begin(); it != this->end(); it++) { for (auto it = ++this->begin(); it != this->end(); it++) {
s << ", "; s << ", ";
print_to(it->first, s); print_to(it->first, s);
s << ": "; s << ": ";
print_to(it->second, s); print_to(it->second, s);
} }
} }
s << '}'; s << '}';
} }
}; };
template <typename K, typename V> template <typename K, typename V>
PyDict(std::initializer_list<std::pair<K, V>>) -> PyDict<K, V>;*/ PyDict(std::initializer_list<std::pair<K, V>>) -> PyDict<K, V>;*/
namespace typon { namespace typon {
template <typename K, typename V> class PyDict { template <typename K, typename V> class PyDict {
public: public:
PyDict(std::shared_ptr<std::unordered_map<K, V>> &&m) : _m(std::move(m)) {} PyDict(std::shared_ptr<std::unordered_map<K, V>> &&m) : _m(std::move(m)) {}
PyDict(std::unordered_map<K, V> &&m) PyDict(std::unordered_map<K, V> &&m)
: _m(std::move( : _m(std::move(
std::make_shared<std::unordered_map<K, V>>(std::move(m)))) {} std::make_shared<std::unordered_map<K, V>>(std::move(m)))) {}
PyDict(std::initializer_list<std::pair<const K, V>> &&m) PyDict(std::initializer_list<std::pair<const K, V>> &&m)
: _m(std::make_shared<std::unordered_map<K, V>>(std::move(m))) {} : _m(std::make_shared<std::unordered_map<K, V>>(std::move(m))) {}
PyDict() : _m(std::make_shared<std::unordered_map<K, V>>()) {} PyDict() : _m(std::make_shared<std::unordered_map<K, V>>()) {}
template <typename... Args> template <typename... Args>
PyDict(Args &&...args) PyDict(Args &&...args)
: _m(std::make_shared<std::unordered_map<K, V>>( : _m(std::make_shared<std::unordered_map<K, V>>(
std::forward<Args>(args)...)) {} std::forward<Args>(args)...)) {}
auto begin() const { return _m->begin(); } auto begin() const { return _m->begin(); }
auto end() const { return _m->end(); } auto end() const { return _m->end(); }
auto py_contains(const K &k) const { return _m->find(k) != _m->end(); } auto py_contains(const K &k) const { return _m->find(k) != _m->end(); }
constexpr const V &operator[](const K &k) const { return (*_m)[k]; } constexpr const V &operator[](const K &k) const { return (*_m)[k]; }
constexpr V &operator[](const K &k) { return (*_m)[k]; } constexpr V &operator[](const K &k) { return (*_m)[k]; }
size_t py_len() const { return _m->size(); } size_t py_len() const { return _m->size(); }
void py_repr(std::ostream &s) const { void py_repr(std::ostream &s) const {
s << '{'; s << '{';
if (_m->size() > 0) { if (_m->size() > 0) {
repr_to(_m->begin()->first, s); repr_to(_m->begin()->first, s);
s << ": "; s << ": ";
repr_to(_m->begin()->second, s); repr_to(_m->begin()->second, s);
for (auto it = ++_m->begin(); it != _m->end(); it++) { for (auto it = ++_m->begin(); it != _m->end(); it++) {
s << ", "; s << ", ";
repr_to(it->first, s); repr_to(it->first, s);
s << ": "; s << ": ";
repr_to(it->second, s); repr_to(it->second, s);
} }
} }
s << '}'; s << '}';
} }
void py_print(std::ostream &s) const { py_repr(s); } void py_print(std::ostream &s) const { py_repr(s); }
private: private:
std::shared_ptr<std::unordered_map<K, V>> _m; std::shared_ptr<std::unordered_map<K, V>> _m;
}; };
} }
#endif // TYPON_DICT_HPP #endif // TYPON_DICT_HPP
// //
// Created by Tom on 08/03/2023. // Created by Tom on 08/03/2023.
// //
#ifndef TYPON_LIST_HPP #ifndef TYPON_LIST_HPP
#define TYPON_LIST_HPP #define TYPON_LIST_HPP
#include <algorithm> #include <algorithm>
#include <ostream> #include <ostream>
#include <vector> #include <vector>
//#include <nanobind/stl/detail/nb_list.h> //#include <nanobind/stl/detail/nb_list.h>
#include <pybind11/stl.h> #include <pybind11/stl.h>
namespace typon { namespace typon {
template <typename T> class PyList { template <typename T> class PyList {
public: public:
using value_type = T; using value_type = T;
PyList(std::shared_ptr<std::vector<T>> &&v) : _v(std::move(v)) {} PyList(std::shared_ptr<std::vector<T>> &&v) : _v(std::move(v)) {}
PyList(std::vector<T> &&v) PyList(std::vector<T> &&v)
: _v(std::move(std::make_shared<std::vector<T>>(std::move(v)))) {} : _v(std::move(std::make_shared<std::vector<T>>(std::move(v)))) {}
PyList(std::initializer_list<T> &&v) PyList(std::initializer_list<T> &&v)
: _v(std::make_shared<std::vector<T>>(std::move(v))) {} : _v(std::make_shared<std::vector<T>>(std::move(v))) {}
PyList() : _v(std::make_shared<std::vector<T>>()) {} PyList() : _v(std::make_shared<std::vector<T>>()) {}
auto begin() const { return _v->begin(); } auto begin() const { return _v->begin(); }
auto end() const { return _v->end(); } auto end() const { return _v->end(); }
METHOD(auto, append, (Self self, const T &x), { METHOD(auto, append, (Self self, const T &x), {
self._v->push_back(x); self._v->push_back(x);
}) })
METHOD(auto, py_contains, (Self self, const T &x), { METHOD(auto, py_contains, (Self self, const T &x), {
return std::find(self.begin(), self.end(), x) != self.end(); return std::find(self.begin(), self.end(), x) != self.end();
}) })
auto size() const { return _v->size(); } auto size() const { return _v->size(); }
void push_back(const T &value) { _v->push_back(value); } void push_back(const T &value) { _v->push_back(value); }
void clear() { _v->clear(); } void clear() { _v->clear(); }
/*operator std::vector<T>() const { /*operator std::vector<T>() const {
return std::vector<T>(this->begin(), this->end()); return std::vector<T>(this->begin(), this->end());
} }
operator std::vector<T> &() { operator std::vector<T> &() {
return *reinterpret_cast<std::vector<T> *>(this); return *reinterpret_cast<std::vector<T> *>(this);
}*/ }*/
constexpr const T &operator[](size_t i) const { return _v->operator[](i); } constexpr const T &operator[](size_t i) const { return _v->operator[](i); }
constexpr T &operator[](size_t i) { return _v->operator[](i); } constexpr T &operator[](size_t i) { return _v->operator[](i); }
size_t py_len() const { return _v->size(); } size_t py_len() const { return _v->size(); }
void py_repr(std::ostream &s) const { void py_repr(std::ostream &s) const {
s << '['; s << '[';
if (_v->size() > 0) { if (_v->size() > 0) {
repr_to(_v->operator[](0), s); repr_to(_v->operator[](0), s);
for (size_t i = 1; i < _v->size(); i++) { for (size_t i = 1; i < _v->size(); i++) {
s << ", "; s << ", ";
repr_to(_v->operator[](i), s); repr_to(_v->operator[](i), s);
} }
} }
s << ']'; s << ']';
} }
void py_print(std::ostream &s) const { py_repr(s); } void py_print(std::ostream &s) const { py_repr(s); }
PyList<T> operator+(const PyList<T> &other) const { PyList<T> operator+(const PyList<T> &other) const {
std::vector<T> v; std::vector<T> v;
v.reserve(_v->size() + other._v->size()); v.reserve(_v->size() + other._v->size());
v.insert(v.end(), _v->begin(), _v->end()); v.insert(v.end(), _v->begin(), _v->end());
v.insert(v.end(), other._v->begin(), other._v->end()); v.insert(v.end(), other._v->begin(), other._v->end());
return PyList<T>(std::move(v)); return PyList<T>(std::move(v));
} }
PyList<T> operator*(size_t n) const { PyList<T> operator*(size_t n) const {
PyList<T> v{}; PyList<T> v{};
v._v->reserve(this->_v->size() * n); v._v->reserve(this->_v->size() * n);
for (size_t i = 0; i < n; i++) { for (size_t i = 0; i < n; i++) {
v._v->insert(v._v->end(), this->_v->begin(), this->_v->end()); v._v->insert(v._v->end(), this->_v->begin(), this->_v->end());
} }
return v; return v;
} }
private: private:
std::shared_ptr<std::vector<T>> _v; std::shared_ptr<std::vector<T>> _v;
}; };
} }
template <typename T> typon::PyList<T> list(std::initializer_list<T> &&v) { template <typename T> typon::PyList<T> list(std::initializer_list<T> &&v) {
return typon::PyList<T>(std::move(v)); return typon::PyList<T>(std::move(v));
} }
namespace PYBIND11_NAMESPACE { namespace PYBIND11_NAMESPACE {
namespace detail { namespace detail {
template <typename Type> template <typename Type>
struct type_caster<typon::PyList<Type>> : list_caster<typon::PyList<Type>, Type> {}; struct type_caster<typon::PyList<Type>> : list_caster<typon::PyList<Type>, Type> {};
}} }}
/*NAMESPACE_BEGIN(NB_NAMESPACE) /*NAMESPACE_BEGIN(NB_NAMESPACE)
NAMESPACE_BEGIN(detail) NAMESPACE_BEGIN(detail)
template <typename Type> struct type_caster<PyList<Type>> template <typename Type> struct type_caster<PyList<Type>>
: list_caster<PyList<Type>, Type> { }; : list_caster<PyList<Type>, Type> { };
NAMESPACE_END(detail) NAMESPACE_END(detail)
NAMESPACE_END(NB_NAMESPACE) NAMESPACE_END(NB_NAMESPACE)
*/ */
#endif // TYPON_LIST_HPP #endif // TYPON_LIST_HPP
// //
// Created by Tom on 09/03/2023. // Created by Tom on 09/03/2023.
// //
#ifndef TYPON_PRINT_HPP #ifndef TYPON_PRINT_HPP
#define TYPON_PRINT_HPP #define TYPON_PRINT_HPP
#include <iostream> #include <iostream>
#include <ostream> #include <ostream>
#include <functional> #include <functional>
#include "str.hpp" #include "str.hpp"
#include <typon/typon.hpp> #include <typon/typon.hpp>
template <typename T> template <typename T>
concept Streamable = requires(const T &x, std::ostream &s) { concept Streamable = requires(const T &x, std::ostream &s) {
{ s << x } -> std::same_as<std::ostream &>; { s << x } -> std::same_as<std::ostream &>;
}; };
template <Streamable T> void print_to(const T &x, std::ostream &s) { s << x; } template <Streamable T> void print_to(const T &x, std::ostream &s) { s << x; }
template <Streamable T> void repr_to(const T &x, std::ostream &s) { s << x; } template <Streamable T> void repr_to(const T &x, std::ostream &s) { s << x; }
template <typename T> template <typename T>
concept PyPrint = requires(const T &x, std::ostream &s) { concept PyPrint = requires(const T &x, std::ostream &s) {
{ x.py_print(s) } -> std::same_as<void>; { x.py_print(s) } -> std::same_as<void>;
}; };
template <PyPrint T> void print_to(const T &x, std::ostream &s) { template <PyPrint T> void print_to(const T &x, std::ostream &s) {
x.py_print(s); x.py_print(s);
} }
template <typename T> template <typename T>
concept PyRepr = requires(const T &x, std::ostream &s) { concept PyRepr = requires(const T &x, std::ostream &s) {
{ x.py_repr(s) } -> std::same_as<void>; { x.py_repr(s) } -> std::same_as<void>;
}; };
template <PyRepr T> void repr_to(const T &x, std::ostream &s) { x.py_repr(s); } template <PyRepr T> void repr_to(const T &x, std::ostream &s) { x.py_repr(s); }
template <typename T> template <typename T>
concept Printable = requires(const T &x, std::ostream &s) { concept Printable = requires(const T &x, std::ostream &s) {
{ print_to(x, s) } -> std::same_as<void>; { print_to(x, s) } -> std::same_as<void>;
}; };
template <typename T> template <typename T>
concept Reprable = requires(const T &x, std::ostream &s) { concept Reprable = requires(const T &x, std::ostream &s) {
{ repr_to(x, s) } -> std::same_as<void>; { repr_to(x, s) } -> std::same_as<void>;
}; };
template <typename T> template <typename T>
concept FunctionPointer = concept FunctionPointer =
std::is_function_v<T> or std::is_member_function_pointer_v<T> or std::is_function_v<T> or std::is_member_function_pointer_v<T> or
std::is_function_v<std::remove_pointer_t<T>>; std::is_function_v<std::remove_pointer_t<T>>;
template <Streamable T> template <Streamable T>
requires(FunctionPointer<T>) requires(FunctionPointer<T>)
void repr_to(const T &x, std::ostream &s) { void repr_to(const T &x, std::ostream &s) {
s << "<function at 0x" << std::hex << (size_t)x << std::dec << ">"; s << "<function at 0x" << std::hex << (size_t)x << std::dec << ">";
} }
template <typename T> template <typename T>
void repr_to(const std::function<T> &x, std::ostream &s) { void repr_to(const std::function<T> &x, std::ostream &s) {
s << "<function at 0x" << std::hex << (size_t)x.template target<T*>() << std::dec s << "<function at 0x" << std::hex << (size_t)x.template target<T*>() << std::dec
<< ">"; << ">";
} }
template <> void repr_to(const PyStr &x, std::ostream &s) { template <> void repr_to(const PyStr &x, std::ostream &s) {
s << '"' << x << '"'; s << '"' << x << '"';
} }
template <Streamable T> template <Streamable T>
requires(Reprable<T>) requires(Reprable<T>)
void print_to(const T &x, std::ostream &s) { void print_to(const T &x, std::ostream &s) {
repr_to(x, s); repr_to(x, s);
} }
template <Printable T> template <Printable T>
void print_to(const std::shared_ptr<T> &x, std::ostream &s) { void print_to(const std::shared_ptr<T> &x, std::ostream &s) {
print_to(*x, s); print_to(*x, s);
} }
template <Reprable T> template <Reprable T>
void repr_to(const std::shared_ptr<T> &x, std::ostream &s) { void repr_to(const std::shared_ptr<T> &x, std::ostream &s) {
repr_to(*x, s); repr_to(*x, s);
} }
template <> void print_to<PyStr>(const PyStr &x, std::ostream &s) { s << x; } template <> void print_to<PyStr>(const PyStr &x, std::ostream &s) { s << x; }
/* /*
template <Printable T, Printable... Args> template <Printable T, Printable... Args>
typon::Task<void> print(T const &head, Args const &...args) { typon::Task<void> print(T const &head, Args const &...args) {
print_to(head, std::cout); print_to(head, std::cout);
(((std::cout << ' '), print_to(args, std::cout)), ...); (((std::cout << ' '), print_to(args, std::cout)), ...);
std::cout << '\n'; co_return; std::cout << '\n'; co_return;
}*/ }*/
struct { struct {
void operator()() { std::cout << '\n'; } void operator()() { std::cout << '\n'; }
template <Printable T, Printable... Args> template <Printable T, Printable... Args>
void operator()(T const &head, Args const &...args) { void operator()(T const &head, Args const &...args) {
print_to(head, std::cout); print_to(head, std::cout);
(((std::cout << ' '), print_to(args, std::cout)), ...); (((std::cout << ' '), print_to(args, std::cout)), ...);
std::cout << '\n'; std::cout << '\n';
} }
} print; } print;
// typon::Task<void> print() { std::cout << '\n'; co_return; } // typon::Task<void> print() { std::cout << '\n'; co_return; }
struct { struct {
PyStr operator()(const PyStr& s = ""_ps) { PyStr operator()(const PyStr& s = ""_ps) {
std::cout << s; std::cout << s;
PyStr input; PyStr input;
std::getline(std::cin, input); std::getline(std::cin, input);
return input; return input;
} }
} input; } input;
#endif // TYPON_PRINT_HPP #endif // TYPON_PRINT_HPP
// //
// Created by Tom on 13/03/2023. // Created by Tom on 13/03/2023.
// //
#ifndef TYPON_RANGE_HPP #ifndef TYPON_RANGE_HPP
#define TYPON_RANGE_HPP #define TYPON_RANGE_HPP
#include <ranges> #include <ranges>
namespace view = std::views; namespace view = std::views;
#include <python/basedef.hpp> #include <python/basedef.hpp>
auto stride = [](int n) { auto stride = [](int n) {
return [s = -1, n](auto const&) mutable { s = (s + 1) % n; return !s; }; return [s = -1, n](auto const&) mutable { s = (s + 1) % n; return !s; };
}; };
// todo: proper range support // todo: proper range support
struct range_s : TyBuiltin<range_s> struct range_s : TyBuiltin<range_s>
{ {
template <typename T> template <typename T>
auto sync(T stop) { return sync(0, stop); } auto sync(T stop) { return sync(0, stop); }
template <typename T> template <typename T>
auto sync(T start, T stop, T step = 1) { auto sync(T start, T stop, T step = 1) {
// https://www.modernescpp.com/index.php/c-20-pythons-map-function/ // https://www.modernescpp.com/index.php/c-20-pythons-map-function/
if(step == 0) { if(step == 0) {
throw std::invalid_argument("Step cannot be 0"); throw std::invalid_argument("Step cannot be 0");
} }
auto Step = start < stop ? step : -step; auto Step = start < stop ? step : -step;
auto Begin = std::min(start, stop); auto Begin = std::min(start, stop);
auto End = Step < 0 ? Begin : std::max(start, stop); auto End = Step < 0 ? Begin : std::max(start, stop);
return view::iota(Begin, End) return view::iota(Begin, End)
| view::filter(stride(std::abs(Step))) | view::filter(stride(std::abs(Step)))
| view::transform([start, stop](std::size_t i){ | view::transform([start, stop](std::size_t i){
return start < stop ? i : stop - (i - start); return start < stop ? i : stop - (i - start);
}); });
} }
} range; } range;
#endif // TYPON_RANGE_HPP #endif // TYPON_RANGE_HPP
// //
// Created by Tom on 08/03/2023. // Created by Tom on 08/03/2023.
// //
#ifndef TYPON_SET_HPP #ifndef TYPON_SET_HPP
#define TYPON_SET_HPP #define TYPON_SET_HPP
#include <unordered_set> #include <unordered_set>
namespace typon { namespace typon {
template <typename T> class PySet : public std::unordered_set<T> { template <typename T> class PySet : public std::unordered_set<T> {
public: public:
PySet(std::unordered_set<T> &&s) : std::unordered_set<T>(std::move(s)) {} PySet(std::unordered_set<T> &&s) : std::unordered_set<T>(std::move(s)) {}
PySet(std::initializer_list<T> &&s) : std::unordered_set<T>(std::move(s)) {} PySet(std::initializer_list<T> &&s) : std::unordered_set<T>(std::move(s)) {}
operator std::unordered_set<T>() const { operator std::unordered_set<T>() const {
return std::unordered_set<T>(this->begin(), this->end()); return std::unordered_set<T>(this->begin(), this->end());
} }
std::size_t py_len() const { return this->size(); } std::size_t py_len() const { return this->size(); }
bool py_contains(const T &t) const { return this->find(t) != this->end(); } bool py_contains(const T &t) const { return this->find(t) != this->end(); }
class iterator { class iterator {
public: public:
using value_type = T; using value_type = T;
using difference_type = std::ptrdiff_t; using difference_type = std::ptrdiff_t;
using pointer = T *; using pointer = T *;
using reference = T &; using reference = T &;
using iterator_category = std::forward_iterator_tag; using iterator_category = std::forward_iterator_tag;
iterator(typename std::unordered_set<T>::iterator it) : _it(it) {} iterator(typename std::unordered_set<T>::iterator it) : _it(it) {}
iterator &operator++() { iterator &operator++() {
_it++; _it++;
return *this; return *this;
} }
iterator operator++(int) { iterator operator++(int) {
iterator tmp = *this; iterator tmp = *this;
_it++; _it++;
return tmp; return tmp;
} }
bool operator==(const iterator &rhs) const { return _it == rhs._it; } bool operator==(const iterator &rhs) const { return _it == rhs._it; }
bool operator!=(const iterator &rhs) const { return _it != rhs._it; } bool operator!=(const iterator &rhs) const { return _it != rhs._it; }
const T &operator*() const { return *_it; } const T &operator*() const { return *_it; }
const T *operator->() const { return &*_it; } const T *operator->() const { return &*_it; }
private: private:
typename std::unordered_set<T>::iterator _it; typename std::unordered_set<T>::iterator _it;
}; };
iterator py_iter() const { return this->begin(); } iterator py_iter() const { return this->begin(); }
void py_print(std::ostream &s) const { void py_print(std::ostream &s) const {
s << '{'; s << '{';
if (this->size() > 0) { if (this->size() > 0) {
print_to(*this->begin(), s); print_to(*this->begin(), s);
for (auto it = ++this->begin(); it != this->end(); it++) { for (auto it = ++this->begin(); it != this->end(); it++) {
s << ", "; s << ", ";
print_to(*it, s); print_to(*it, s);
} }
} }
s << '}'; s << '}';
} }
}; };
} }
template <typename T> typon::PySet<T> set(std::initializer_list<T> &&s) { template <typename T> typon::PySet<T> set(std::initializer_list<T> &&s) {
return typon::PySet<T>(std::move(s)); return typon::PySet<T>(std::move(s));
} }
#endif // TYPON_SET_HPP #endif // TYPON_SET_HPP
// //
// Created by Tom on 08/03/2023. // Created by Tom on 08/03/2023.
// //
#ifndef TYPON_STR_HPP #ifndef TYPON_STR_HPP
#define TYPON_STR_HPP #define TYPON_STR_HPP
#include <sstream> #include <sstream>
#include <string> #include <string>
#include <algorithm> #include <algorithm>
using namespace std::literals; using namespace std::literals;
#include "bytes.hpp" #include "bytes.hpp"
#include "print.hpp" #include "print.hpp"
#include "slice.hpp" #include "slice.hpp"
// #include <format> // #include <format>
#include <fmt/format.h> #include <fmt/format.h>
class PyStr : public std::string { class PyStr : public std::string {
public: public:
PyStr() : std::string() {} PyStr() : std::string() {}
PyStr(const std::string &s) : std::string(s) {} PyStr(const std::string &s) : std::string(s) {}
PyStr(std::string &&s) : std::string(std::move(s)) {} PyStr(std::string &&s) : std::string(std::move(s)) {}
constexpr PyStr(const char *s, size_t count) : std::string(s, count) {} constexpr PyStr(const char *s, size_t count) : std::string(s, count) {}
constexpr PyStr(size_t count, char ch) : std::string(count, ch) {} constexpr PyStr(size_t count, char ch) : std::string(count, ch) {}
template<typename... Args> template<typename... Args>
PyStr(Args&&... args) : std::string(std::forward<Args>(args)...) {} PyStr(Args&&... args) : std::string(std::forward<Args>(args)...) {}
template <class InputIterator> template <class InputIterator>
PyStr(InputIterator first, InputIterator last) : std::string(first, last) {} PyStr(InputIterator first, InputIterator last) : std::string(first, last) {}
METHOD(PyBytes, encode, (Self self, const std::string &encoding = "utf-8"), { METHOD(PyBytes, encode, (Self self, const std::string &encoding = "utf-8"), {
return PyBytes(self.begin(), self.end()); return PyBytes(self.begin(), self.end());
}) })
METHOD_GEN((typename... T), PyStr, format, (Self self, T &&...args), { METHOD_GEN((typename... T), PyStr, format, (Self self, T &&...args), {
return PyStr(fmt::format(fmt::runtime(self), std::forward<T>(args)...)); return PyStr(fmt::format(fmt::runtime(self), std::forward<T>(args)...));
}) })
METHOD(bool, startswith, (Self self, const std::string &s), { METHOD(bool, startswith, (Self self, const std::string &s), {
return self.starts_with(s); return self.starts_with(s);
}) })
METHOD(int, find, (Self self, const std::string &s), { METHOD(int, find, (Self self, const std::string &s), {
auto pos = self.std::string::find(s); auto pos = self.std::string::find(s);
return pos == std::string::npos ? -1 : pos; return pos == std::string::npos ? -1 : pos;
}) })
METHOD(bool, isspace, (Self self), { METHOD(bool, isspace, (Self self), {
return self.find_first_not_of(' ') == std::string::npos; return self.find_first_not_of(' ') == std::string::npos;
}) })
METHOD(auto, py_contains, (Self self, const std::string &x), { METHOD(auto, py_contains, (Self self, const std::string &x), {
return self.std::string::find(x) != std::string::npos; return self.std::string::find(x) != std::string::npos;
}) })
PyStr operator[](PySlice slice) const { PyStr operator[](PySlice slice) const {
auto [len, new_slice] = slice.adjust_indices(this->size()); auto [len, new_slice] = slice.adjust_indices(this->size());
PyStr result; PyStr result;
result.reserve(len); result.reserve(len);
if (new_slice.start < new_slice.stop) { if (new_slice.start < new_slice.stop) {
if (new_slice.step > 0) { if (new_slice.step > 0) {
for (auto i = new_slice.start; i < new_slice.stop; for (auto i = new_slice.start; i < new_slice.stop;
i += new_slice.step) { i += new_slice.step) {
result.push_back(this->char_at(i)); result.push_back(this->char_at(i));
} }
} }
} else { } else {
if (new_slice.step < 0) { if (new_slice.step < 0) {
for (auto i = new_slice.start; i > new_slice.stop; for (auto i = new_slice.start; i > new_slice.stop;
i += new_slice.step) { i += new_slice.step) {
result.push_back(this->char_at(i)); result.push_back(this->char_at(i));
} }
} }
} }
return result; return result;
} }
PyStr operator[](ssize_t index) const { PyStr operator[](ssize_t index) const {
if (index < 0) { if (index < 0) {
index += this->size(); index += this->size();
} }
return PyStr(1, std::string::operator[](index)); return PyStr(1, std::string::operator[](index));
} }
char char_at(ssize_t index) const { char char_at(ssize_t index) const {
if (index < 0) { if (index < 0) {
index += this->size(); index += this->size();
} }
return this->begin()[index]; return this->begin()[index];
} }
operator int() const { operator int() const {
return std::stoi(*this); return std::stoi(*this);
} }
}; };
inline constexpr PyStr operator""_ps(const char *s, size_t len) noexcept { inline constexpr PyStr operator""_ps(const char *s, size_t len) noexcept {
return PyStr(s, len); return PyStr(s, len);
} }
template<typename Self> template<typename Self>
PyStr PyBytes::decode_s::operator()(Self self, const std::string &encoding) const { PyStr PyBytes::decode_s::operator()(Self self, const std::string &encoding) const {
return PyStr(self.begin(), self.end()); return PyStr(self.begin(), self.end());
} }
template <typename T> PyStr str(const T &x) { template <typename T> PyStr str(const T &x) {
std::stringstream s; std::stringstream s;
print_to(x, s); print_to(x, s);
return s.str(); return s.str();
} }
template <typename T> PyStr repr(const T &x) { template <typename T> PyStr repr(const T &x) {
std::stringstream s; std::stringstream s;
repr_to(x, s); repr_to(x, s);
return s.str(); return s.str();
} }
template <> struct std::hash<PyStr> { template <> struct std::hash<PyStr> {
std::size_t operator()(const PyStr &s) const noexcept { std::size_t operator()(const PyStr &s) const noexcept {
return std::hash<std::string>()(s); return std::hash<std::string>()(s);
} }
}; };
#endif // TYPON_STR_HPP #endif // TYPON_STR_HPP
// //
// Created by Tom on 09/03/2023. // Created by Tom on 09/03/2023.
// //
#ifndef TYPON_SYS_HPP #ifndef TYPON_SYS_HPP
#define TYPON_SYS_HPP #define TYPON_SYS_HPP
#include <iostream> #include <iostream>
#include "builtins.hpp" #include "builtins.hpp"
namespace py_sys { namespace py_sys {
struct sys_t { struct sys_t {
static constexpr auto &stdin = std::cin; static constexpr auto &stdin = std::cin;
static constexpr auto &stdout = std::cout; static constexpr auto &stdout = std::cout;
static constexpr auto &stderr = std::cerr; static constexpr auto &stderr = std::cerr;
typon::PyList<PyStr> argv; typon::PyList<PyStr> argv;
FUNCTION(void, exit, (int code), { FUNCTION(void, exit, (int code), {
std::exit(code); std::exit(code);
}) })
} all; } all;
auto& get_all() { auto& get_all() {
return all; return all;
} }
} // namespace py_sys } // namespace py_sys
#endif // TYPON_SYS_HPP #endif // TYPON_SYS_HPP
Subproject commit 9194f47b6a9fc96b3d77b932033b9e3395be4cb8
...@@ -11,7 +11,8 @@ from pathlib import Path ...@@ -11,7 +11,8 @@ from pathlib import Path
import sys import sys
compiler_path = Path(__file__).parent compiler_path = Path(__file__).parent
runtime_path = compiler_path.parent / "rt" stdlib_path = compiler_path.parent / "include"
runtime_path = compiler_path.parent / "runtime" / "rt" / "include"
sys.path.insert(0, str(compiler_path)) sys.path.insert(0, str(compiler_path))
...@@ -40,7 +41,8 @@ if args.cpp_flags: ...@@ -40,7 +41,8 @@ if args.cpp_flags:
import sysconfig import sysconfig
include_dirs = [ include_dirs = [
str(runtime_path / "include"), str(stdlib_path),
str(runtime_path),
sysconfig.get_path("include"), sysconfig.get_path("include"),
sysconfig.get_path("platinclude"), sysconfig.get_path("platinclude"),
pybind11.commands.get_include() pybind11.commands.get_include()
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment