docker ELK环境搭建

本文介绍的组件搭建，均为docker环境，如需查看其他环境搭建，请移步官网进行查看

1.elasticsearch 组件列表

2. docker elasticsearch start

elasticsearch官方文档

elasticsearch简介

Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.

拉取elasticsearch镜像

docker pull docker.elastic.co/elasticsearch/elasticsearch:6.5.4

启动elasticsearch

docker run -it -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.5.4

3. docker start logstash

logstash官方文档

logstash简介

Logstash is an open source data collection engine with real-time pipelining capabilities. Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice. Cleanse and democratize all your data for diverse advanced downstream analytics and visualization use cases.
While Logstash originally drove innovation in log collection, its capabilities extend well beyond that use case. Any type of event can be enriched and transformed with a broad array of input, filter, and output plugins, with many native codecs further simplifying the ingestion process. Logstash accelerates your insights by harnessing a greater volume and variety of data

拉取logstash镜像

docker pull docker.elastic.co/logstash/logstash:6.5.4

启动logstash

docker run  -it -d --name logstash -v /usr/local/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml -v /usr/local/logstash/pipeline/:/usr/share/logstash/pipeline/  -v /usr/local/logstash/lib/:/usr/local/logstash/lib/ docker.elastic.co/logstash/logstash:6.5.4

参数说明
-v /usr/local/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml将本地logstash的配置文件挂载到容器内部
-v /usr/local/logstash/pipeline/:/usr/share/logstash/pipeline/将logstash管道配置文件挂载到容器内部
-v /usr/local/logstash/lib/:/usr/local/logstash/lib/在使用logstash全量增量导入数据的时候，需要使用jdbc进行导入，所以将jar的目录挂载到容器内部

demo数据导入配置文件

input {
  jdbc { 
    jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/test"
    jdbc_user => "user"
    jdbc_password => "password"
    # The path to our downloaded jdbc driver
    jdbc_driver_library => "/usr/local/logstash/lib/mysql/mysql-connector-java-5.1.47.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    # our query
    schedule => "* * * * *"
    statement => "select* from test where id> :sql_last_value"
    use_column_value => true
    # not lowercase column name
    lowercase_column_names => false
    tracking_column => id
}
}

output {
  stdout { codec => json_lines }
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
      index => "test"
  document_type => "data"
  document_id => "%{id}"
  }
}

格式化后的sql，需开启logstash自动转义，默认为关闭

statement => "select
                   id,
                   name,
                   user_id as userId
               from test 
               where id> :sql_last_value"

image.png

设置方法：修改logstash.yml配置文件，开启自动转义
config.support_escapes: true

logstash config说明文档

4. docker start kibane

kibane官方文档

kibana简介

Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps.

Kibana makes it easy to understand large volumes of data. Its simple, browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.

Setting up Kibana is a snap. You can install Kibana and start exploring your Elasticsearch indices in minutes — no code, no additional infrastructure required.

拉取kibana

docker pull docker.elastic.co/kibana/kibana:6.5.4

启动kibana

docker run --name kibana -e ELASTICSEARCH_URL=http://127.0.0.1:9200 -p 5601:5601 -d docker.elastic.co/kibana/kibana:6.5.4

至此elk的基本组件就搭好了，尽情玩耍把!

docker ELK环境搭建