Skip to content

S
slapos
Commit 2562f56f authored by Jérome Perrin's avatar Jérome Perrin
Browse files
Options

software/grafana: version up and include loki/promtail 

This enable vieiwing applications logs directly in grafana, as long as
loki agent can access the log file and the regular expressions to parse
the logs are defined

Other changes:
 - provision data sources automatically so that user do not have to add
   them manually
 - update TODO in README
 - introduce a SR test
 - switch to dep to install dependencies, because that's how these
  packages manage their dependencies.

Versions up:
  grafana: v6.3.0
  telegraf: v1.11.1-0
  influxdb: v1.7.8rc1
New:
  loki: v0.3.0
parent e70c9e52
Hide whitespace changes
Inline Side-by-side
Showing with 565 additions and 844 deletions
software/grafana/README.md
View file @ 2562f56f
# grafana / telegraf / influxdb # grafana / telegraf / influxdb
This is an experimental integration, mainly to evaluate these solutions.
## Custom telegraf plugins ## Custom telegraf plugins
See https://github.com/influxdata/telegraf to learn about plugins. See https://github.com/influxdata/telegraf to learn about plugins.
Useful plugins in this context are probably Useful plugins in this context are probably
[exec](https://github.com/influxdata/telegraf/tree/1.5.1/plugins/inputs/exec) [exec](https://github.com/influxdata/telegraf/tree/1.11.1/plugins/inputs/exec),
[logparser](https://github.com/influxdata/telegraf/tree/1.11.1/plugins/inputs/logparser)
or or
[httpjson](https://github.com/influxdata/telegraf/tree/1.5.1/plugins/inputs/httpjson). [http](https://github.com/influxdata/telegraf/tree/1.11.1/plugins/inputs/http).
Telegraf will save in the `telegraf` database from the embedded influxdb server. Telegraf will save in the `telegraf` database from the embedded influxdb server.
## Grafana ## Grafana
You'll have to add yourself the influxdb data source in grafana, using the A default user is created, username and password are published as connection
parameters published by the slapos instance. parameters. You can add more users in grafana interface.
http://docs.grafana.org/features/datasources/influxdb/
When adding datasource, use *proxy* option, otherwise Grafana makes your Datasources should be automatically added.
browser query influxdb directly, which also uses a self signed certificate.
One workaround is to configure your browser to also accept influxdb certificate
before using grafana, but using proxy seems easier.
## Influxdb ## Influxdb
...@@ -34,8 +31,32 @@ One important thing to notice is that the backup protocol is enabled on ipv4 ...@@ -34,8 +31,32 @@ One important thing to notice is that the backup protocol is enabled on ipv4
provided by slapos, so make sure this ip is not reachable from untrusted provided by slapos, so make sure this ip is not reachable from untrusted
sources. sources.
## TODO # Ingesting/Visualizing logs
Eventhough main feature is visualizing metrics, Grafana has a feature called "Explore" to view logs for a time frame.
The following backend can be used:
## Loki
See `TestLoki` in test for an example.
## Influxdb
Influxdb logs only have tags and there does not seem to be a way to search (except than tag and time frame).
To inject log files containing:
```
INFO the message
WARN another message
```
use config like:
```
[[inputs.logparser]]
files = ["/tmp/x*.log", "/tmp/aaa.log"]
* influxdb and telegraf runs with very low priority, this could become an option [inputs.logparser.grok]
* make one partition for each service and use switch software type measurement = "logs"
* make it easier to add custom configuration (how ?) patterns = ['^%{WORD:level:tag} %{GREEDYDATA:message:string}']
```
software/grafana/buildout.hash.cfg
View file @ 2562f56f
...@@ -15,7 +15,7 @@ ...@@ -15,7 +15,7 @@
[instance-profile] [instance-profile]
filename = instance.cfg.in filename = instance.cfg.in
md5sum = 461d515da03de5e422e6f75189d09184 md5sum = 8abfd8ce01c375257e94f64824478f5c
[influxdb-config-file] [influxdb-config-file]
filename = influxdb-config-file.cfg.in filename = influxdb-config-file.cfg.in
...@@ -28,3 +28,15 @@ md5sum = a1a9c22c2a7829c66a49fc2504604d21 ...@@ -28,3 +28,15 @@ md5sum = a1a9c22c2a7829c66a49fc2504604d21
[grafana-config-file] [grafana-config-file]
filename = grafana-config-file.cfg.in filename = grafana-config-file.cfg.in
md5sum = 8244d430905b968795c7946049bed9e3 md5sum = 8244d430905b968795c7946049bed9e3
[grafana-provisioning-config-file]
filename = grafana-provisioning-config-file.cfg.in
md5sum = 3aa0f1ed752b2a59ea2b5e7c1733daf3
[loki-config-file]
filename = loki-config-file.cfg.in
md5sum = 02ba5acf23fcf88f5594919f46838533
[promtail-config-file]
filename = promtail-config-file.cfg.in
md5sum = c77788d0a3cc654ad9393eb4b1f31e94
software/grafana/gowork.cfg
View file @ 2562f56f
...@@ -3,844 +3,39 @@ ...@@ -3,844 +3,39 @@
# list of go git repositories to fetch # list of go git repositories to fetch
[gowork.goinstall] [gowork.goinstall]
depends_gitfetch = depends_gitfetch =
${go_collectd.org:recipe} ${go_github.com_golang_dep:recipe}
${go_github.com_BurntSushi_toml:recipe}
${go_github.com_Microsoft_go-winio:recipe}
${go_github.com_RoaringBitmap_roaring:recipe}
${go_github.com_Shopify_sarama:recipe}
${go_github.com_Sirupsen_logrus:recipe}
${go_github.com_StackExchange_wmi:recipe}
${go_github.com_aerospike_aerospike-client-go:recipe}
${go_github.com_amir_raidman:recipe}
${go_github.com_apache_thrift:recipe}
${go_github.com_aws_aws-sdk-go:recipe}
${go_github.com_beorn7_perks:recipe}
${go_github.com_bmizerany_pat:recipe}
${go_github.com_boltdb_bolt:recipe}
${go_github.com_bsm_sarama-cluster:recipe}
${go_github.com_cenkalti_backoff:recipe}
${go_github.com_cespare_xxhash:recipe}
${go_github.com_couchbase_go-couchbase:recipe}
${go_github.com_couchbase_gomemcached:recipe}
${go_github.com_couchbase_goutils:recipe}
${go_github.com_davecgh_go-spew:recipe}
${go_github.com_dgrijalva_jwt-go:recipe}
${go_github.com_dgryski_go-bitstream:recipe}
${go_github.com_docker_docker:recipe}
${go_github.com_docker_go-connections:recipe}
${go_github.com_eapache_go-resiliency:recipe}
${go_github.com_eapache_go-xerial-snappy:recipe}
${go_github.com_eapache_queue:recipe}
${go_github.com_eclipse_paho.mqtt.golang:recipe}
${go_github.com_glycerine_go-unsnap-stream:recipe}
${go_github.com_go-ini_ini:recipe}
${go_github.com_go-logfmt_logfmt:recipe}
${go_github.com_go-ole_go-ole:recipe}
${go_github.com_go-sql-driver_mysql:recipe}
${go_github.com_gobwas_glob:recipe}
${go_github.com_gogo_protobuf:recipe}
${go_github.com_golang_protobuf:recipe}
${go_github.com_golang_snappy:recipe}
${go_github.com_google_go-cmp:recipe}
${go_github.com_gorilla_mux:recipe}
${go_github.com_grafana_grafana:recipe} ${go_github.com_grafana_grafana:recipe}
${go_github.com_hailocab_go-hostpool:recipe} ${go_github.com_grafana_loki:recipe}
${go_github.com_hashicorp_consul:recipe}
${go_github.com_influxdata_influxdb:recipe} ${go_github.com_influxdata_influxdb:recipe}
${go_github.com_influxdata_influxql:recipe}
${go_github.com_influxdata_tail:recipe}
${go_github.com_influxdata_telegraf:recipe} ${go_github.com_influxdata_telegraf:recipe}
${go_github.com_influxdata_toml:recipe}
${go_github.com_influxdata_usage-client:recipe}
${go_github.com_influxdata_wlog:recipe}
${go_github.com_influxdata_yamux:recipe}
${go_github.com_influxdata_yarpc:recipe}
${go_github.com_jackc_pgx:recipe}
${go_github.com_jmespath_go-jmespath:recipe}
${go_github.com_jwilder_encoding:recipe}
${go_github.com_kardianos_govendor:recipe}
${go_github.com_kardianos_osext:recipe}
${go_github.com_kardianos_service:recipe}
${go_github.com_kballard_go-shellquote:recipe}
${go_github.com_matttproud_golang_protobuf_extensions:recipe}
${go_github.com_miekg_dns:recipe}
${go_github.com_mitchellh_mapstructure:recipe}
${go_github.com_multiplay_go-ts3:recipe}
${go_github.com_naoina_go-stringutil:recipe}
${go_github.com_nats-io_go-nats:recipe}
${go_github.com_nats-io_nats:recipe}
${go_github.com_nats-io_nuid:recipe}
${go_github.com_nsqio_go-nsq:recipe}
${go_github.com_opencontainers_runc:recipe}
${go_github.com_opentracing-contrib_go-observer:recipe}
${go_github.com_opentracing_opentracing-go:recipe}
${go_github.com_openzipkin_zipkin-go-opentracing:recipe}
${go_github.com_peterh_liner:recipe}
${go_github.com_philhofer_fwd:recipe}
${go_github.com_pierrec_lz4:recipe}
${go_github.com_pierrec_xxHash:recipe}
${go_github.com_pkg_errors:recipe}
${go_github.com_pmezard_go-difflib:recipe}
${go_github.com_prometheus_client_golang:recipe}
${go_github.com_prometheus_client_model:recipe}
${go_github.com_prometheus_common:recipe}
${go_github.com_prometheus_procfs:recipe}
${go_github.com_rcrowley_go-metrics:recipe}
${go_github.com_retailnext_hllpp:recipe}
${go_github.com_samuel_go-zookeeper:recipe}
${go_github.com_satori_go.uuid:recipe}
${go_github.com_shirou_gopsutil:recipe}
${go_github.com_shirou_w32:recipe}
${go_github.com_soniah_gosnmp:recipe}
${go_github.com_sparrc_gdm:recipe}
${go_github.com_streadway_amqp:recipe}
${go_github.com_stretchr_objx:recipe}
${go_github.com_stretchr_testify:recipe}
${go_github.com_tidwall_gjson:recipe}
${go_github.com_tidwall_match:recipe}
${go_github.com_tinylib_msgp:recipe}
${go_github.com_vjeantet_grok:recipe}
${go_github.com_wvanbergen_kafka:recipe}
${go_github.com_wvanbergen_kazoo-go:recipe}
${go_github.com_xlab_treeprint:recipe}
${go_github.com_yuin_gopher-lua:recipe}
${go_github.com_zensqlmonitor_go-mssqldb:recipe}
${go_go.uber.org_atomic:recipe}
${go_go.uber.org_multierr:recipe}
${go_go.uber.org_zap:recipe}
${go_golang.org_x_crypto:recipe}
${go_golang.org_x_net:recipe}
${go_golang.org_x_sys:recipe}
${go_golang.org_x_text:recipe}
${go_golang.org_x_time:recipe}
${go_golang.org_x_tools:recipe}
${go_gopkg.in_asn1-ber.v1:recipe}
${go_gopkg.in_fatih_pool.v2:recipe}
${go_gopkg.in_fsnotify.v1:recipe}
${go_gopkg.in_gorethink_gorethink.v3:recipe}
${go_gopkg.in_ldap.v2:recipe}
${go_gopkg.in_mgo.v2:recipe}
${go_gopkg.in_olivere_elastic.v5:recipe}
${go_gopkg.in_tomb.v1:recipe}
${go_gopkg.in_yaml.v2:recipe}
[go_collectd.org] [go_github.com_golang_dep]
<= go-git-package <= go-git-package
go.importpath = collectd.org go.importpath = github.com/golang/dep
repository = https://github.com/collectd/go-collectd repository = https://github.com/golang/dep
revision = v0.3.0-17-g606bd390f3 revision = 1f7c19e5f5
[go_github.com_BurntSushi_toml]
<= go-git-package
go.importpath = github.com/BurntSushi/toml
repository = https://github.com/BurntSushi/toml
revision = v0.2.0-45-ga368813c5e
[go_github.com_Microsoft_go-winio]
<= go-git-package
go.importpath = github.com/Microsoft/go-winio
repository = https://github.com/Microsoft/go-winio
revision = ce2922f643
[go_github.com_RoaringBitmap_roaring]
<= go-git-package
go.importpath = github.com/RoaringBitmap/roaring
repository = https://github.com/RoaringBitmap/roaring
revision = v0.2.8-174-g0a6691af7c
[go_github.com_Shopify_sarama]
<= go-git-package
go.importpath = github.com/Shopify/sarama
repository = https://github.com/Shopify/sarama
revision = 3b1b38866a
[go_github.com_Sirupsen_logrus]
<= go-git-package
go.importpath = github.com/Sirupsen/logrus
repository = https://github.com/Sirupsen/logrus
revision = 61e43dc76f
[go_github.com_StackExchange_wmi]
<= go-git-package
go.importpath = github.com/StackExchange/wmi
repository = https://github.com/StackExchange/wmi
revision = f3e2bae1e0
[go_github.com_aerospike_aerospike-client-go]
<= go-git-package
go.importpath = github.com/aerospike/aerospike-client-go
repository = https://github.com/aerospike/aerospike-client-go
revision = v1.6.4-277-g95e1ad7791
[go_github.com_amir_raidman]
<= go-git-package
go.importpath = github.com/amir/raidman
repository = https://github.com/amir/raidman
revision = c74861fe6a
[go_github.com_apache_thrift]
<= go-git-package
go.importpath = github.com/apache/thrift
repository = https://github.com/apache/thrift
revision = 4aaa92ece8
[go_github.com_aws_aws-sdk-go]
<= go-git-package
go.importpath = github.com/aws/aws-sdk-go
repository = https://github.com/aws/aws-sdk-go
revision = c861d27d03
[go_github.com_beorn7_perks]
<= go-git-package
go.importpath = github.com/beorn7/perks
repository = https://github.com/beorn7/perks
revision = 4c0e84591b
[go_github.com_bmizerany_pat]
<= go-git-package
go.importpath = github.com/bmizerany/pat
repository = https://github.com/bmizerany/pat
revision = 6226ea591a
[go_github.com_boltdb_bolt]
<= go-git-package
go.importpath = github.com/boltdb/bolt
repository = https://github.com/boltdb/bolt
revision = 9da3174536
[go_github.com_bsm_sarama-cluster]
<= go-git-package
go.importpath = github.com/bsm/sarama-cluster
repository = https://github.com/bsm/sarama-cluster
revision = v1.0.0-164-gabf039439f
[go_github.com_cenkalti_backoff]
<= go-git-package
go.importpath = github.com/cenkalti/backoff
repository = https://github.com/cenkalti/backoff
revision = b02f2bbce1
[go_github.com_cespare_xxhash]
<= go-git-package
go.importpath = github.com/cespare/xxhash
repository = https://github.com/cespare/xxhash
revision = e4e2bd419c
[go_github.com_couchbase_go-couchbase]
<= go-git-package
go.importpath = github.com/couchbase/go-couchbase
repository = https://github.com/couchbase/go-couchbase
revision = bfe555a140
[go_github.com_couchbase_gomemcached]
<= go-git-package
go.importpath = github.com/couchbase/gomemcached
repository = https://github.com/couchbase/gomemcached
revision = 4a25d2f4e1
[go_github.com_couchbase_goutils]
<= go-git-package
go.importpath = github.com/couchbase/goutils
repository = https://github.com/couchbase/goutils
revision = 5823a0cbaa
[go_github.com_davecgh_go-spew]
<= go-git-package
go.importpath = github.com/davecgh/go-spew
repository = https://github.com/davecgh/go-spew
revision = v1.1.0-9-gecdeabc654
[go_github.com_dgrijalva_jwt-go]
<= go-git-package
go.importpath = github.com/dgrijalva/jwt-go
repository = https://github.com/dgrijalva/jwt-go
revision = dbeaa9332f
[go_github.com_dgryski_go-bitstream]
<= go-git-package
go.importpath = github.com/dgryski/go-bitstream
repository = https://github.com/dgryski/go-bitstream
revision = 7d46cd22db
[go_github.com_docker_docker]
<= go-git-package
go.importpath = github.com/docker/docker
repository = https://github.com/docker/docker
revision = v17.03.2-ce-0-gf5ec1e2936
[go_github.com_docker_go-connections]
<= go-git-package
go.importpath = github.com/docker/go-connections
repository = https://github.com/docker/go-connections
revision = 990a1a1a70
[go_github.com_eapache_go-resiliency]
<= go-git-package
go.importpath = github.com/eapache/go-resiliency
repository = https://github.com/eapache/go-resiliency
revision = b86b1ec0dd
[go_github.com_eapache_go-xerial-snappy]
<= go-git-package
go.importpath = github.com/eapache/go-xerial-snappy
repository = https://github.com/eapache/go-xerial-snappy
revision = bb955e01b9
[go_github.com_eapache_queue]
<= go-git-package
go.importpath = github.com/eapache/queue
repository = https://github.com/eapache/queue
revision = 44cc805cf1
[go_github.com_eclipse_paho.mqtt.golang]
<= go-git-package
go.importpath = github.com/eclipse/paho.mqtt.golang
repository = https://github.com/eclipse/paho.mqtt.golang
revision = d4f545eb10
[go_github.com_glycerine_go-unsnap-stream]
<= go-git-package
go.importpath = github.com/glycerine/go-unsnap-stream
repository = https://github.com/glycerine/go-unsnap-stream
revision = 62a9a9eb44
[go_github.com_go-ini_ini]
<= go-git-package
go.importpath = github.com/go-ini/ini
repository = https://github.com/go-ini/ini
revision = 9144852efb
[go_github.com_go-logfmt_logfmt]
<= go-git-package
go.importpath = github.com/go-logfmt/logfmt
repository = https://github.com/go-logfmt/logfmt
revision = v0.3.0-0-g390ab7935e
[go_github.com_go-ole_go-ole]
<= go-git-package
go.importpath = github.com/go-ole/go-ole
repository = https://github.com/go-ole/go-ole
revision = be49f7c077
[go_github.com_go-sql-driver_mysql]
<= go-git-package
go.importpath = github.com/go-sql-driver/mysql
repository = https://github.com/go-sql-driver/mysql
revision = v1.0-470-g2e00b5cd70
[go_github.com_gobwas_glob]
<= go-git-package
go.importpath = github.com/gobwas/glob
repository = https://github.com/gobwas/glob
revision = v0.2.2-0-gbea32b9cd2
[go_github.com_gogo_protobuf]
<= go-git-package
go.importpath = github.com/gogo/protobuf
repository = https://github.com/gogo/protobuf
revision = 160de10b25
[go_github.com_golang_protobuf]
<= go-git-package
go.importpath = github.com/golang/protobuf
repository = https://github.com/golang/protobuf
revision = 1e59b77b52
[go_github.com_golang_snappy]
<= go-git-package
go.importpath = github.com/golang/snappy
repository = https://github.com/golang/snappy
revision = 553a641470
[go_github.com_google_go-cmp]
<= go-git-package
go.importpath = github.com/google/go-cmp
repository = https://github.com/google/go-cmp
revision = f94e52cad9
[go_github.com_gorilla_mux]
<= go-git-package
go.importpath = github.com/gorilla/mux
repository = https://github.com/gorilla/mux
revision = 392c28fe23
[go_github.com_grafana_grafana] [go_github.com_grafana_grafana]
<= go-git-package <= go-git-package
go.importpath = github.com/grafana/grafana go.importpath = github.com/grafana/grafana
repository = https://github.com/grafana/grafana repository = https://github.com/grafana/grafana
revision = v4.6.0-beta1-1360-g9606a34e0a revision = v6.3.0-0-g830da0fda0
[go_github.com_hailocab_go-hostpool]
<= go-git-package
go.importpath = github.com/hailocab/go-hostpool
repository = https://github.com/hailocab/go-hostpool
revision = e80d13ce29
[go_github.com_hashicorp_consul] [go_github.com_grafana_loki]
<= go-git-package <= go-git-package
go.importpath = github.com/hashicorp/consul go.importpath = github.com/grafana/loki
repository = https://github.com/hashicorp/consul repository = https://github.com/grafana/loki
revision = v0.7.3-35-g63d2fc6823 revision = v0.3.0-0-gc673380bb3
[go_github.com_influxdata_influxdb] [go_github.com_influxdata_influxdb]
<= go-git-package <= go-git-package
go.importpath = github.com/influxdata/influxdb go.importpath = github.com/influxdata/influxdb
repository = https://github.com/influxdata/influxdb repository = https://github.com/influxdata/influxdb
revision = v1.4.0rc0-328-g938db68198 revision = v1.7.8rc1-0-ga7afa2ecf1
[go_github.com_influxdata_influxql]
<= go-git-package
go.importpath = github.com/influxdata/influxql
repository = https://github.com/influxdata/influxql
revision = 851636b092
[go_github.com_influxdata_tail]
<= go-git-package
go.importpath = github.com/influxdata/tail
repository = https://github.com/influxdata/tail
revision = v0-95-ga395bf99fe
[go_github.com_influxdata_telegraf] [go_github.com_influxdata_telegraf]
<= go-git-package <= go-git-package
go.importpath = github.com/influxdata/telegraf go.importpath = github.com/influxdata/telegraf
repository = https://github.com/influxdata/telegraf repository = https://github.com/influxdata/telegraf
revision = 1.5.0-rc1-73-g90b6b760d1 revision = 1.11.1-0-gf91de60d37
[go_github.com_influxdata_toml]
<= go-git-package
go.importpath = github.com/influxdata/toml
repository = https://github.com/influxdata/toml
revision = 5d1d907f22
[go_github.com_influxdata_usage-client]
<= go-git-package
go.importpath = github.com/influxdata/usage-client
repository = https://github.com/influxdata/usage-client
revision = 6d38953763
[go_github.com_influxdata_wlog]
<= go-git-package
go.importpath = github.com/influxdata/wlog
repository = https://github.com/influxdata/wlog
revision = 7c63b0a71e
[go_github.com_influxdata_yamux]
<= go-git-package
go.importpath = github.com/influxdata/yamux
repository = https://github.com/influxdata/yamux
revision = 1f58ded512
[go_github.com_influxdata_yarpc]
<= go-git-package
go.importpath = github.com/influxdata/yarpc
repository = https://github.com/influxdata/yarpc
revision = fdd7e84bf3
[go_github.com_jackc_pgx]
<= go-git-package
go.importpath = github.com/jackc/pgx
repository = https://github.com/jackc/pgx
revision = 63f58fd32e
[go_github.com_jmespath_go-jmespath]
<= go-git-package
go.importpath = github.com/jmespath/go-jmespath
repository = https://github.com/jmespath/go-jmespath
revision = 0.2.2-14-gbd40a432e4
[go_github.com_jwilder_encoding]
<= go-git-package
go.importpath = github.com/jwilder/encoding
repository = https://github.com/jwilder/encoding
revision = b4e1701a28
[go_github.com_kardianos_govendor]
<= go-git-package
go.importpath = github.com/kardianos/govendor
repository = https://github.com/kardianos/govendor
revision = 274337c49c
[go_github.com_kardianos_osext]
<= go-git-package
go.importpath = github.com/kardianos/osext
repository = https://github.com/kardianos/osext
revision = c2c54e542f
[go_github.com_kardianos_service]
<= go-git-package
go.importpath = github.com/kardianos/service
repository = https://github.com/kardianos/service
revision = 6d3a0ee7d3
[go_github.com_kballard_go-shellquote]
<= go-git-package
go.importpath = github.com/kballard/go-shellquote
repository = https://github.com/kballard/go-shellquote
revision = d8ec1a69a2
[go_github.com_matttproud_golang_protobuf_extensions]
<= go-git-package
go.importpath = github.com/matttproud/golang_protobuf_extensions
repository = https://github.com/matttproud/golang_protobuf_extensions
revision = v1.0.0-2-gc12348ce28
[go_github.com_miekg_dns]
<= go-git-package
go.importpath = github.com/miekg/dns
repository = https://github.com/miekg/dns
revision = 99f84ae56e
[go_github.com_mitchellh_mapstructure]
<= go-git-package
go.importpath = github.com/mitchellh/mapstructure
repository = https://github.com/mitchellh/mapstructure
revision = d0303fe809
[go_github.com_multiplay_go-ts3]
<= go-git-package
go.importpath = github.com/multiplay/go-ts3
repository = https://github.com/multiplay/go-ts3
revision = 07477f49b8
[go_github.com_naoina_go-stringutil]
<= go-git-package
go.importpath = github.com/naoina/go-stringutil
repository = https://github.com/naoina/go-stringutil
revision = 6b638e95a3
[go_github.com_nats-io_go-nats]
<= go-git-package
go.importpath = github.com/nats-io/go-nats
repository = https://github.com/nats-io/go-nats
revision = v1.2.0-73-gea9585611a
[go_github.com_nats-io_nats]
<= go-git-package
go.importpath = github.com/nats-io/nats
repository = https://github.com/nats-io/nats
revision = v1.2.0-73-gea9585611a
[go_github.com_nats-io_nuid]
<= go-git-package
go.importpath = github.com/nats-io/nuid
repository = https://github.com/nats-io/nuid
revision = 289cccf02c
[go_github.com_nsqio_go-nsq]
<= go-git-package
go.importpath = github.com/nsqio/go-nsq
repository = https://github.com/nsqio/go-nsq
revision = v1.0.7-0-geee57a3ac4
[go_github.com_opencontainers_runc]
<= go-git-package
go.importpath = github.com/opencontainers/runc
repository = https://github.com/opencontainers/runc
revision = v0.0.5-426-g89ab7f2ccc
[go_github.com_opentracing-contrib_go-observer]
<= go-git-package
go.importpath = github.com/opentracing-contrib/go-observer
repository = https://github.com/opentracing-contrib/go-observer
revision = a52f234244
[go_github.com_opentracing_opentracing-go]
<= go-git-package
go.importpath = github.com/opentracing/opentracing-go
repository = https://github.com/opentracing/opentracing-go
revision = 1361b9cd60
[go_github.com_openzipkin_zipkin-go-opentracing]
<= go-git-package
go.importpath = github.com/openzipkin/zipkin-go-opentracing
repository = https://github.com/openzipkin/zipkin-go-opentracing
revision = 1cafbdfde9
[go_github.com_peterh_liner]
<= go-git-package
go.importpath = github.com/peterh/liner
repository = https://github.com/peterh/liner
revision = 3681c2a912
[go_github.com_philhofer_fwd]
<= go-git-package
go.importpath = github.com/philhofer/fwd
repository = https://github.com/philhofer/fwd
revision = bb6d471dc9
[go_github.com_pierrec_lz4]
<= go-git-package
go.importpath = github.com/pierrec/lz4
repository = https://github.com/pierrec/lz4
revision = v1.0-0-g5c9560bfa9
[go_github.com_pierrec_xxHash]
<= go-git-package
go.importpath = github.com/pierrec/xxHash
repository = https://github.com/pierrec/xxHash
revision = 5a004441f8
[go_github.com_pkg_errors]
<= go-git-package
go.importpath = github.com/pkg/errors
repository = https://github.com/pkg/errors
revision = v0.8.0-0-g645ef00459
[go_github.com_pmezard_go-difflib]
<= go-git-package
go.importpath = github.com/pmezard/go-difflib
repository = https://github.com/pmezard/go-difflib
revision = v1.0.0-0-g792786c740
[go_github.com_prometheus_client_golang]
<= go-git-package
go.importpath = github.com/prometheus/client_golang
repository = https://github.com/prometheus/client_golang
revision = v0.9.0-pre1-21-g180b8fdc22
[go_github.com_prometheus_client_model]
<= go-git-package
go.importpath = github.com/prometheus/client_model
repository = https://github.com/prometheus/client_model
revision = model-0.0.2-16-g99fa1f4be8
[go_github.com_prometheus_common]
<= go-git-package
go.importpath = github.com/prometheus/common
repository = https://github.com/prometheus/common
revision = 89604d1970
[go_github.com_prometheus_procfs]
<= go-git-package
go.importpath = github.com/prometheus/procfs
repository = https://github.com/prometheus/procfs
revision = b15cd069a8
[go_github.com_rcrowley_go-metrics]
<= go-git-package
go.importpath = github.com/rcrowley/go-metrics
repository = https://github.com/rcrowley/go-metrics
revision = 1f30fe9094
[go_github.com_retailnext_hllpp]
<= go-git-package
go.importpath = github.com/retailnext/hllpp
repository = https://github.com/retailnext/hllpp
revision = v1.0.0-2-g6e8b6d3944
[go_github.com_samuel_go-zookeeper]
<= go-git-package
go.importpath = github.com/samuel/go-zookeeper
repository = https://github.com/samuel/go-zookeeper
revision = 1d7be4effb
[go_github.com_satori_go.uuid]
<= go-git-package
go.importpath = github.com/satori/go.uuid
repository = https://github.com/satori/go.uuid
revision = v1.1.0-8-g5bf94b69c6
[go_github.com_shirou_gopsutil]
<= go-git-package
go.importpath = github.com/shirou/gopsutil
repository = https://github.com/shirou/gopsutil
revision = v2.16.10-243-g384a55110a
[go_github.com_shirou_w32]
<= go-git-package
go.importpath = github.com/shirou/w32
repository = https://github.com/shirou/w32
revision = 3c9377fc67
[go_github.com_soniah_gosnmp]
<= go-git-package
go.importpath = github.com/soniah/gosnmp
repository = https://github.com/soniah/gosnmp
revision = v1.0-204-g5ad50dc75a
[go_github.com_sparrc_gdm]
<= go-git-package
go.importpath = github.com/sparrc/gdm
repository = https://github.com/sparrc/gdm
revision = 1.4-4-g81089dabfa
[go_github.com_streadway_amqp]
<= go-git-package
go.importpath = github.com/streadway/amqp
repository = https://github.com/streadway/amqp
revision = 63795daa9a
[go_github.com_stretchr_objx]
<= go-git-package
go.importpath = github.com/stretchr/objx
repository = https://github.com/stretchr/objx
revision = 1a9d0bb9f5
[go_github.com_stretchr_testify]
<= go-git-package
go.importpath = github.com/stretchr/testify
repository = https://github.com/stretchr/testify
revision = v1.0-187-g4d4bfba8f1
[go_github.com_tidwall_gjson]
<= go-git-package
go.importpath = github.com/tidwall/gjson
repository = https://github.com/tidwall/gjson
revision = 0623bd8fbd
[go_github.com_tidwall_match]
<= go-git-package
go.importpath = github.com/tidwall/match
repository = https://github.com/tidwall/match
revision = 173748da73
[go_github.com_tinylib_msgp]
<= go-git-package
go.importpath = github.com/tinylib/msgp
repository = https://github.com/tinylib/msgp
revision = 428e467e72
[go_github.com_vjeantet_grok]
<= go-git-package
go.importpath = github.com/vjeantet/grok
repository = https://github.com/vjeantet/grok
revision = d73e972b60
[go_github.com_wvanbergen_kafka]
<= go-git-package
go.importpath = github.com/wvanbergen/kafka
repository = https://github.com/wvanbergen/kafka
revision = bc265fedb9
[go_github.com_wvanbergen_kazoo-go]
<= go-git-package
go.importpath = github.com/wvanbergen/kazoo-go
repository = https://github.com/wvanbergen/kazoo-go
revision = 9689573521
[go_github.com_xlab_treeprint]
<= go-git-package
go.importpath = github.com/xlab/treeprint
repository = https://github.com/xlab/treeprint
revision = 06dfc6fa17
[go_github.com_yuin_gopher-lua]
<= go-git-package
go.importpath = github.com/yuin/gopher-lua
repository = https://github.com/yuin/gopher-lua
revision = 66c871e454
[go_github.com_zensqlmonitor_go-mssqldb]
<= go-git-package
go.importpath = github.com/zensqlmonitor/go-mssqldb
repository = https://github.com/zensqlmonitor/go-mssqldb
revision = ffe5510c6f
[go_go.uber.org_atomic]
<= go-git-package
go.importpath = go.uber.org/atomic
repository = https://github.com/uber-go/atomic
revision = v1.3.1-0-g8474b86a5a
[go_go.uber.org_multierr]
<= go-git-package
go.importpath = go.uber.org/multierr
repository = https://github.com/uber-go/multierr
revision = v1.1.0-1-gfb7d312c2c
[go_go.uber.org_zap]
<= go-git-package
go.importpath = go.uber.org/zap
repository = https://github.com/uber-go/zap
revision = v1.7.1-11-gf85c78b1dd
[go_golang.org_x_crypto]
<= go-git-package
go.importpath = golang.org/x/crypto
repository = https://go.googlesource.com/crypto
revision = b3c9a1d25c
[go_golang.org_x_net]
<= go-git-package
go.importpath = golang.org/x/net
repository = https://go.googlesource.com/net
revision = ab555f366c
[go_golang.org_x_sys]
<= go-git-package
go.importpath = golang.org/x/sys
repository = https://go.googlesource.com/sys
revision = 810d700034
[go_golang.org_x_text]
<= go-git-package
go.importpath = golang.org/x/text
repository = https://go.googlesource.com/text
revision = e19ae14969
[go_golang.org_x_time]
<= go-git-package
go.importpath = golang.org/x/time
repository = https://go.googlesource.com/time
revision = 6dc17368e0
[go_golang.org_x_tools]
<= go-git-package
go.importpath = golang.org/x/tools
repository = https://go.googlesource.com/tools
revision = fbec762f83
[go_gopkg.in_asn1-ber.v1]
<= go-git-package
go.importpath = gopkg.in/asn1-ber.v1
repository = https://gopkg.in/asn1-ber.v1
revision = 4e86f43671
[go_gopkg.in_fatih_pool.v2]
<= go-git-package
go.importpath = gopkg.in/fatih/pool.v2
repository = https://gopkg.in/fatih/pool.v2
revision = 6e328e6789
[go_gopkg.in_fsnotify.v1]
<= go-git-package
go.importpath = gopkg.in/fsnotify.v1
repository = https://gopkg.in/fsnotify.v1
revision = a8a77c9133
[go_gopkg.in_gorethink_gorethink.v3]
<= go-git-package
go.importpath = gopkg.in/gorethink/gorethink.v3
repository = https://gopkg.in/gorethink/gorethink.v3
revision = v2.2.1-35-g7ab832f7b6
[go_gopkg.in_ldap.v2]
<= go-git-package
go.importpath = gopkg.in/ldap.v2
repository = https://gopkg.in/ldap.v2
revision = 8168ee085e
[go_gopkg.in_mgo.v2]
<= go-git-package
go.importpath = gopkg.in/mgo.v2
repository = https://gopkg.in/mgo.v2
revision = 3f83fa5005
[go_gopkg.in_olivere_elastic.v5]
<= go-git-package
go.importpath = gopkg.in/olivere/elastic.v5
repository = https://gopkg.in/olivere/elastic.v5
revision = v5.0.41-0-g3113f9b9ad
[go_gopkg.in_tomb.v1]
<= go-git-package
go.importpath = gopkg.in/tomb.v1
repository = https://gopkg.in/tomb.v1
revision = dd632973f1
[go_gopkg.in_yaml.v2]
<= go-git-package
go.importpath = gopkg.in/yaml.v2
repository = https://gopkg.in/yaml.v2
revision = 4c78c975fe
software/grafana/grafana-provisioning-config-file.cfg.in 0 → 100644
View file @ 2562f56f
# https://grafana.com/docs/administration/provisioning/#example-datasource-config-file
apiVersion: 1
datasources:
- name: telegraf
type: influxdb
access: proxy
url: {{ influxdb['url'] }}
user: {{ influxdb['auth-username'] }}
database: telegraf
isDefault: true
jsonData:
tlsSkipVerify: true
secureJsonData:
password: {{ influxdb['auth-password'] }}
version: 1
editable: false
- name: loki
type: loki
access: proxy
url: {{ loki['url'] }}
version: 1
editable: false
software/grafana/instance-input-schema.json
View file @ 2562f56f
...@@ -28,6 +28,11 @@ ...@@ -28,6 +28,11 @@
"description": "Name used in From: header of emails", "description": "Name used in From: header of emails",
"default": "Grafana", "default": "Grafana",
"type": "string" "type": "string"
},
"promtail-extra-scrape-config": {
"description": "Raw promtail config (experimental parameter, see https://github.com/grafana/loki/blob/master/docs/promtail.md#scrape-configs for detail)",
"default": "",
"type": "string"
} }
} }
} }
software/grafana/instance.cfg.in
View file @ 2562f56f
...@@ -41,11 +41,14 @@ grafana-data-dir = ${:grafana-dir}/data ...@@ -41,11 +41,14 @@ grafana-data-dir = ${:grafana-dir}/data
grafana-logs-dir = ${:var}/log grafana-logs-dir = ${:var}/log
grafana-plugins-dir = ${:grafana-dir}/plugins grafana-plugins-dir = ${:grafana-dir}/plugins
grafana-provisioning-config-dir = ${:grafana-dir}/provisioning-config grafana-provisioning-config-dir = ${:grafana-dir}/provisioning-config
grafana-provisioning-datasources = ${:grafana-provisioning-config-dir}/datasources grafana-provisioning-datasources-dir = ${:grafana-provisioning-config-dir}/datasources
grafana-provisioning-dashboards = ${:grafana-provisioning-config-dir}/dashboards grafana-provisioning-dashboards-dir = ${:grafana-provisioning-config-dir}/dashboards
telegraf-dir = ${:srv}/telegraf telegraf-dir = ${:srv}/telegraf
telegraf-extra-config-dir = ${:telegraf-dir}/extra-config telegraf-extra-config-dir = ${:telegraf-dir}/extra-config
loki-dir = ${:srv}/loki
loki-storage-boltdb-dir = ${:loki-dir}/index/
loki-storage-filesystem-dir = ${:loki-dir}/chunks/
promtail-dir = ${:srv}/promtail
# macros # macros
[generate-certificate] [generate-certificate]
...@@ -75,7 +78,11 @@ extensions = jinja2.ext.do ...@@ -75,7 +78,11 @@ extensions = jinja2.ext.do
module = check_port_listening module = check_port_listening
name = ${:_buildout_section_name_}.py name = ${:_buildout_section_name_}.py
[check-url-available-promise]
recipe = slapos.cookbook:check_url_available
path = ${directory:promise}/${:_buildout_section_name_}
dash_path = {{ dash_bin }}
curl_path = {{ curl_bin }}
[influxdb] [influxdb]
ipv6 = ${instance-parameter:ipv6-random} ipv6 = ${instance-parameter:ipv6-random}
...@@ -133,6 +140,7 @@ data-dir = ${directory:grafana-data-dir} ...@@ -133,6 +140,7 @@ data-dir = ${directory:grafana-data-dir}
logs-dir = ${directory:grafana-logs-dir} logs-dir = ${directory:grafana-logs-dir}
plugins-dir = ${directory:grafana-plugins-dir} plugins-dir = ${directory:grafana-plugins-dir}
provisioning-config-dir = ${directory:grafana-provisioning-config-dir} provisioning-config-dir = ${directory:grafana-provisioning-config-dir}
provisioning-datasources-dir = ${directory:grafana-provisioning-datasources-dir}
admin-user = ${grafana-password:username} admin-user = ${grafana-password:username}
admin-password = ${grafana-password:passwd} admin-password = ${grafana-password:passwd}
secret-key = ${grafana-secret-key:passwd} secret-key = ${grafana-secret-key:passwd}
...@@ -160,6 +168,15 @@ context = ...@@ -160,6 +168,15 @@ context =
section grafana grafana section grafana grafana
section apache_frontend apache-frontend section apache_frontend apache-frontend
key slapparameter_dict slap-configuration:configuration key slapparameter_dict slap-configuration:configuration
depends =
${grafana-provisioning-config-file:rendered}
[grafana-provisioning-config-file]
<= config-file
rendered = ${grafana:provisioning-datasources-dir}/datasource.yaml
context =
section influxdb influxdb
section loki loki
[grafana-listen-promise] [grafana-listen-promise]
<= check-port-listening-promise <= check-port-listening-promise
...@@ -183,6 +200,53 @@ context = ...@@ -183,6 +200,53 @@ context =
section telegraf telegraf section telegraf telegraf
[loki]
recipe = slapos.cookbook:wrapper
command-line =
bash -c 'nice -19 chrt --idle 0 ionice -c3 {{ loki_bin }} -config.file=${loki-config-file:rendered}'
wrapper-path = ${directory:service}/loki
storage-boltdb-dir = ${directory:loki-storage-boltdb-dir}
storage-filesystem-dir = ${directory:loki-storage-filesystem-dir}
ip = ${instance-parameter:ipv4-random}
port = 3100
url = http://${:ip}:${:port}
[loki-config-file]
<= config-file
context =
section loki loki
[loki-listen-promise]
<= check-url-available-promise
url = ${loki:url}/ready
[promtail]
recipe = slapos.cookbook:wrapper
command-line =
bash -c 'nice -19 chrt --idle 0 ionice -c3 {{ promtail_bin }} -config.file=${promtail-config-file:rendered}'
wrapper-path = ${directory:service}/promtail
dir = ${directory:promtail-dir}
http_port = 19080
ip = ${instance-parameter:ipv4-random}
url = http://${:ip}:${:http_port}
[promtail-config-file]
<= config-file
context =
section promtail promtail
section loki loki
key slapparameter_dict slap-configuration:configuration
[promtail-listen-promise]
<= check-port-listening-promise
hostname= ${promtail:ip}
port = ${promtail:http_port}
[apache-frontend] [apache-frontend]
<= slap-connection <= slap-connection
recipe = slapos.cookbook:requestoptional recipe = slapos.cookbook:requestoptional
...@@ -201,6 +265,8 @@ instance-promises = ...@@ -201,6 +265,8 @@ instance-promises =
${influxdb-listen-promise:path} ${influxdb-listen-promise:path}
${influxdb-password-promise:wrapper-path} ${influxdb-password-promise:wrapper-path}
${grafana-listen-promise:path} ${grafana-listen-promise:path}
${loki-listen-promise:path}
${promtail-listen-promise:path}
[publish-connection-parameter] [publish-connection-parameter]
...@@ -213,4 +279,6 @@ telegraf-extra-config-dir = ${telegraf:extra-config-dir} ...@@ -213,4 +279,6 @@ telegraf-extra-config-dir = ${telegraf:extra-config-dir}
grafana-url = ${grafana:url} grafana-url = ${grafana:url}
grafana-username = ${grafana:admin-user} grafana-username = ${grafana:admin-user}
grafana-password = ${grafana:admin-password} grafana-password = ${grafana:admin-password}
loki-url = ${loki:url}
promtail-url = ${promtail:url}
url = ${apache-frontend:connection-secure_access} url = ${apache-frontend:connection-secure_access}
software/grafana/loki-config-file.cfg.in 0 → 100644
View file @ 2562f56f
auth_enabled: false
server:
http_listen_port: {{ loki['port'] }}
ingester:
lifecycler:
address: {{ loki['ip'] }}
ring:
kvstore:
store: inmemory
replication_factor: 1
chunk_idle_period: 15m
schema_config:
configs:
- from: 2018-04-15
store: boltdb
object_store: filesystem
schema: v9
index:
prefix: index_
period: 168h
storage_config:
boltdb:
directory: {{ loki['storage-boltdb-dir'] }}
filesystem:
directory: {{ loki['storage-filesystem-dir'] }}
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
max_look_back_period: 0
table_manager:
chunk_tables_provisioning:
inactive_read_throughput: 0
inactive_write_throughput: 0
provisioned_read_throughput: 0
provisioned_write_throughput: 0
index_tables_provisioning:
inactive_read_throughput: 0
inactive_write_throughput: 0
provisioned_read_throughput: 0
provisioned_write_throughput: 0
retention_deletes_enabled: false
retention_period: 0
software/grafana/promtail-config-file.cfg.in 0 → 100644
View file @ 2562f56f
# https://github.com/grafana/loki/blob/master/docs/logentry/processing-log-lines.md
server:
http_listen_port: {{ promtail['http_port'] }}
# XXX this external_url does not work ... promtail still listen on all IPs
external_url: {{ promtail['url'] }}
grpc_listen_port: 0
positions:
filename: {{ promtail['dir'] }}/positions.yaml
clients:
- url: {{ loki['url'] }}/api/prom/push
scrape_configs:
- job_name: test
static_configs:
- targets:
- localhost
labels:
job: grafanalogs
__path__: ./var/log/*log
{{ slapparameter_dict.get('promtail-extra-scrape-config', '') }}
software/grafana/software.cfg
View file @ 2562f56f
...@@ -6,6 +6,8 @@ extends = ...@@ -6,6 +6,8 @@ extends =
../../component/make/buildout.cfg ../../component/make/buildout.cfg
../../component/golang/buildout.cfg ../../component/golang/buildout.cfg
../../component/openssl/buildout.cfg ../../component/openssl/buildout.cfg
../../component/curl/buildout.cfg
../../component/dash/buildout.cfg
buildout.hash.cfg buildout.hash.cfg
gowork.cfg gowork.cfg
...@@ -17,6 +19,9 @@ parts = ...@@ -17,6 +19,9 @@ parts =
influxdb-config-file influxdb-config-file
telegraf-config-file telegraf-config-file
grafana-config-file grafana-config-file
grafana-provisioning-config-file
loki-config-file
promtail-config-file
[nodejs] [nodejs]
...@@ -31,7 +36,7 @@ md5sum = 46790033c23803387890f545e4040690 ...@@ -31,7 +36,7 @@ md5sum = 46790033c23803387890f545e4040690
[gowork] [gowork]
# All the softwares installed in the go work have "non standard" installation # All the softwares installed in the go workspace have "non standard" installation
# methods, so we install them in specific parts with custom commands. # methods, so we install them in specific parts with custom commands.
# They will be installed because they are dependencies of ${gowork.goinstall} # They will be installed because they are dependencies of ${gowork.goinstall}
install = install =
...@@ -41,6 +46,8 @@ influx-bin = ${:bin}/influx ...@@ -41,6 +46,8 @@ influx-bin = ${:bin}/influx
influxd-bin = ${:bin}/influxd influxd-bin = ${:bin}/influxd
grafana-bin = ${:bin}/grafana-server grafana-bin = ${:bin}/grafana-server
grafana-homepath = ${go_github.com_grafana_grafana:location} grafana-homepath = ${go_github.com_grafana_grafana:location}
loki-bin = ${:bin}/loki
promtail-bin = ${:bin}/promtail
# use recent go # use recent go
golang = ${golang1.12:location} golang = ${golang1.12:location}
...@@ -48,21 +55,26 @@ gcc-bin-directory = ${golang1.12:gcc-bin-directory} ...@@ -48,21 +55,26 @@ gcc-bin-directory = ${golang1.12:gcc-bin-directory}
[gowork.goinstall] [gowork.goinstall]
command = : command = :
depends = depends =
${influxdb-install:recipe} ${influxdb-install:recipe}
${telegraf-install:recipe} ${telegraf-install:recipe}
${grafana-install:recipe} ${grafana-install:recipe}
${loki-install:recipe}
${promtail-install:recipe}
[influxdb-install] [influxdb-install]
<= gowork.goinstall <= gowork.goinstall
command = bash -c ". ${gowork:env.sh} && \ command = bash -c ". ${gowork:env.sh} && \
go install -v github.com/golang/dep/cmd/dep && \
cd ${gowork:directory}/src/github.com/influxdata/influxdb && \ cd ${gowork:directory}/src/github.com/influxdata/influxdb && \
dep ensure && \
go install ./cmd/..." go install ./cmd/..."
update-command = update-command =
[telegraf-install] [telegraf-install]
<= gowork.goinstall <= gowork.goinstall
command = bash -c ". ${gowork:env.sh} && \ command = bash -c ". ${gowork:env.sh} && \
go install -v github.com/golang/dep/cmd/dep && \
cd ${gowork:directory}/src/github.com/influxdata/telegraf && \ cd ${gowork:directory}/src/github.com/influxdata/telegraf && \
${make:location}/bin/make && ${make:location}/bin/make &&
cp telegraf ${gowork:bin}" cp telegraf ${gowork:bin}"
...@@ -74,12 +86,32 @@ update-command = ...@@ -74,12 +86,32 @@ update-command =
command = bash -c "export PATH=${nodejs:location}/bin/:$PATH && \ command = bash -c "export PATH=${nodejs:location}/bin/:$PATH && \
. ${gowork:env.sh} && \ . ${gowork:env.sh} && \
cd ${gowork:directory}/src/github.com/grafana/grafana && \ cd ${gowork:directory}/src/github.com/grafana/grafana && \
${gowork:golang}/bin/go run build.go setup && \ go run build.go setup && \
${gowork:golang}/bin/go run build.go build && \ go run build.go build && \
${yarn:location}/bin/yarn install --pure-lockfile && \ ${yarn:location}/bin/yarn install --pure-lockfile && \
${nodejs:location}/bin/npm run build" ${nodejs:location}/bin/npm run build"
update-command = update-command =
[loki-install]
<= gowork.goinstall
# loki also uses nodejs
command = bash -c "export PATH=${nodejs:location}/bin/:$PATH && \
. ${gowork:env.sh} && \
go install -v github.com/golang/dep/cmd/dep && \
cd ${gowork:directory}/src/github.com/grafana/loki && \
go install ./cmd/loki"
update-command =
[promtail-install]
<= gowork.goinstall
# CGO_ENABLED is to disable systemd support (did not compile in my case)
command = bash -c "export CGO_ENABLED=0 && \
. ${gowork:env.sh} && \
go install -v github.com/golang/dep/cmd/dep && \
cd ${gowork:directory}/src/github.com/grafana/loki && \
go install ./cmd/promtail"
update-command =
[download-file-base] [download-file-base]
recipe = slapos.recipe.build:download recipe = slapos.recipe.build:download
...@@ -96,6 +128,15 @@ mode = 0644 ...@@ -96,6 +128,15 @@ mode = 0644
[grafana-config-file] [grafana-config-file]
<= download-file-base <= download-file-base
[grafana-provisioning-config-file]
<= download-file-base
[loki-config-file]
<= download-file-base
[promtail-config-file]
<= download-file-base
[instance-profile] [instance-profile]
recipe = slapos.recipe.template:jinja2 recipe = slapos.recipe.template:jinja2
template = ${:_profile_base_location_}/${:filename} template = ${:_profile_base_location_}/${:filename}
...@@ -110,10 +151,14 @@ context = ...@@ -110,10 +151,14 @@ context =
key influx_bin gowork:influx-bin key influx_bin gowork:influx-bin
key grafana_bin gowork:grafana-bin key grafana_bin gowork:grafana-bin
key grafana_homepath gowork:grafana-homepath key grafana_homepath gowork:grafana-homepath
key loki_bin gowork:loki-bin
key promtail_bin gowork:promtail-bin
key curl_bin :curl-bin
key dash_bin :dash-bin
key monitor_template monitor2-template:rendered key monitor_template monitor2-template:rendered
curl-bin = ${curl:location}/bin/curl
dash-bin = ${dash:location}/bin/dash
[versions] [versions]
slapos.recipe.template = 4.2 slapos.recipe.template = 4.2
inotifyx = 0.2.2 inotifyx = 0.2.2
software/grafana/test/README.md 0 → 100644
View file @ 2562f56f
Tests for Grafana software release
software/grafana/test/setup.py 0 → 100644
View file @ 2562f56f
##############################################################################
#
# Copyright (c) 2018 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from setuptools import setup, find_packages
version = '0.0.1.dev0'
name = 'slapos.test.grafana'
long_description = open("README.md").read()
setup(
name=name,
version=version,
description="Test for SlapOS' Grafana",
long_description=long_description,
long_description_content_type='text/markdown',
maintainer="Nexedi",
maintainer_email="info@nexedi.com",
url="https://lab.nexedi.com/nexedi/slapos",
packages=find_packages(),
install_requires=[
'slapos.core',
'slapos.libnetworkcache',
'erp5.util',
'requests',
'supervisor',
'psutil',
],
zip_safe=True,
test_suite='test',
)
software/grafana/test/test.py 0 → 100644
View file @ 2562f56f
##############################################################################
#
# Copyright (c) 2019 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import os
import textwrap
import logging
import tempfile
import time
import requests
from slapos.testing.testcase import makeModuleSetUpAndTestCaseClass
setUpModule, SlapOSInstanceTestCase = makeModuleSetUpAndTestCaseClass(
os.path.abspath(
os.path.join(os.path.dirname(__file__), '..', 'software.cfg')))
class GrafanaTestCase(SlapOSInstanceTestCase):
"""Base test case for grafana.
Since the instances takes timte to start and stop,
we increate as lot the number of retries.
"""
report_max_retry = 30
instance_max_retry = 30
class TestGrafana(GrafanaTestCase):
def setUp(self):
self.grafana_url = self.computer_partition.getConnectionParameterDict(
)['grafana-url']
def test_grafana_available(self):
resp = requests.get(self.grafana_url, verify=False)
self.assertEqual(requests.codes.ok, resp.status_code)
def test_grafana_api(self):
# check API is usable
api_org_url = '{self.grafana_url}/api/org'.format(**locals())
resp = requests.get(api_org_url, verify=False)
self.assertEqual(requests.codes.unauthorized, resp.status_code)
connection_params = self.computer_partition.getConnectionParameterDict()
resp = requests.get(
api_org_url,
verify=False,
auth=requests.auth.HTTPBasicAuth(
connection_params['grafana-username'],
connection_params['grafana-password'],
))
self.assertEqual(requests.codes.ok, resp.status_code)
self.assertEqual(1, resp.json()['id'])
def test_grafana_datasource_povisinonned(self):
# data sources are provisionned
connection_params = self.computer_partition.getConnectionParameterDict()
resp = requests.get(
'{self.grafana_url}/api/datasources'.format(**locals()),
verify=False,
auth=requests.auth.HTTPBasicAuth(
connection_params['grafana-username'],
connection_params['grafana-password'],
))
self.assertEqual(requests.codes.ok, resp.status_code)
self.assertEqual(
sorted(['influxdb', 'loki']),
sorted([ds['type'] for ds in resp.json()]))
class TestInfluxDb(GrafanaTestCase):
def setUp(self):
self.influxdb_url = self.computer_partition.getConnectionParameterDict(
)['influxdb-url']
def test_influxdb_available(self):
ping_url = '{self.influxdb_url}/ping'.format(**locals())
resp = requests.get(ping_url, verify=False)
self.assertEqual(requests.codes.no_content, resp.status_code)
def test_influxdb_api(self):
query_url = '{self.influxdb_url}/query'.format(**locals())
connection_params = self.computer_partition.getConnectionParameterDict()
for i in range(5):
# retry, as it may take a little delay to create databases
resp = requests.get(
query_url,
verify=False,
params=dict(
q='SHOW DATABASES',
u=connection_params['influxdb-username'],
p=connection_params['influxdb-password']))
self.assertEqual(requests.codes.ok, resp.status_code)
result, = resp.json()['results']
if result['series']:
break
time.sleep(0.5 * i)
self.assertIn(
[connection_params['influxdb-database']], result['series'][0]['values'])
class TestTelegraf(GrafanaTestCase):
def test_telegraf_running(self):
with self.slap.instance_supervisor_rpc as supervisor:
all_process_info = supervisor.getAllProcessInfo()
process_info, = [p for p in all_process_info if 'telegraf' in p['name']]
self.assertEqual('RUNNING', process_info['statename'])
class TestLoki(GrafanaTestCase):
@classmethod
def getInstanceParameterDict(cls):
cls._logfile = tempfile.NamedTemporaryFile(suffix='log')
return {
'promtail-extra-scrape-config':
textwrap.dedent(
r'''
- job_name: {cls.__name__}
pipeline_stages:
- regex:
expression: "^(?P<timestamp>.*) - (?P<name>\\S+) - (?P<level>\\S+) - (?P<message>.*)"
- timestamp:
format: 2006-01-02T15:04:05Z00:00
source: timestamp
- labels:
level:
name:
static_configs:
- targets:
- localhost
labels:
job: {cls.__name__}
__path__: {cls._logfile.name}
''').format(**locals())
}
@classmethod
def tearDownClass(cls):
cls._logfile.close()
super(TestLoki, cls).tearDownClass()
def setUp(self):
self.loki_url = self.computer_partition.getConnectionParameterDict(
)['loki-url']
def test_loki_available(self):
self.assertEqual(
requests.codes.ok,
requests.get('{self.loki_url}/ready'.format(**locals()),
verify=False).status_code)
def test_log_ingested(self):
# create a logger logging to the file that we have
# configured in instance parameter.
test_logger = logging.getLogger(self.id()) # type: logging.Logger
test_handler = logging.FileHandler(filename=self._logfile.name)
test_handler.setFormatter(
logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s'))
test_logger.addHandler(test_handler)
test_logger.info("testing message")
test_logger.info("testing another message")
test_logger.warning("testing warn")
# Check our messages have been ingested
# we retry a few times, because there's a short delay until messages are
# ingested and returned.
for i in range(10):
resp = requests.get(
'{self.loki_url}/api/prom/query?query={{job="TestLoki"}}'.format(
**locals()),
verify=False).json()
if not resp or len(resp['streams']) < 2:
time.sleep(0.5 * i)
continue
warn_stream, = [stream for stream in resp['streams'] if 'level="WARNING"' in stream['labels']]
self.assertIn("testing warn", warn_stream['entries'][0]['line'])
info_stream, = [stream for stream in resp['streams'] if 'level="INFO"' in stream['labels']]
self.assertTrue(
[
line for line in info_stream['entries']
if "testing message" in line['line']
])
self.assertTrue(
[
line for line in info_stream['entries']
if "testing another message" in line['line']
])
# The labels we have configued are also available
resp = requests.get(
'{self.loki_url}/api/prom/label'.format(**locals()),
verify=False).json()
self.assertIn('level', resp['values'])
self.assertIn('name', resp['values'])
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7