Parsing Logs with Logstash

发表于 | 分类于 |

步骤

  • deb:

    1
    2
    # curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.4.2-amd64.deb
    # sudo dpkg -i filebeat-5.4.2-amd64.deb

  • 安装后可以看到如下

Parsing Logs with Logstash

filebeat config

  • 在安装完成之后你需要配置filebeat，通过修改filebeat.yml，以nginx的日志为例
1
2
# cd /etc/filebeat
# vi filebeat.yml
  • filebeat.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- input_type: log
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
     - /var/log/nginx/*.log
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]
  • 在数据源的机器上执行如下命令:
    1
    2
    # cd /usr/share/filebeat/bin/
    # ./filebeat -e -c /etc/filebeat/filebeat.yml -d "publish"

logstash config

  • 创建一个logstash的配置文件
1
2
# cd /etc/logstash/conf.d
# vi first-pipeline.conf

  • first-pipeline.conf

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    input {
        beats {
            port => "5044"
        }
    }
    # The filter part of this file is commented out to indicate that it is
    # optional.
    # filter {
    #
    # }
    output {
        stdout { codec => rubydebug }
    }

  • 确认logstash配置文件是否写错

1
2
# cd /usr/share/logstash/
# bin/logstash  --path.settings /etc/logstash  -f /etc/logstash/conf.d/first-pipeline.conf --config.test_and_exit
  • 获取ok ( The –config.test_and_exit option parses your configuration file and reports any errors.)
1
2
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK
  • 运行logstash
1
# bin/logstash  --path.settings /etc/logstash  -f /etc/logstash/conf.d/first-pipeline.conf --config.reload.automatic
  • 在filebeat开启的时候可以获得类似于如下的信息：
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    {
        "@timestamp" => 2017-06-22T14:58:32.169Z,
            "offset" => 13300,
          "@version" => "1",
        "input_type" => "log",
              "beat" => {
            "hostname" => "iZm5e7jlki70utmw22zj76Z",
                "name" => "iZm5e7jlki70utmw22zj76Z",
             "version" => "5.4.2"
        },
              "host" => "iZm5e7jlki70utmw22zj76Z",
            "source" => "/var/log/nginx/access.log",
           "message" => "115.61.84.162 - - [22/Jun/2017:22:06:06 +0800] \"GET http://open.163.com/ HTTP/1.1\" 200 612 \"http://open.163.com/\" \"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)\"",
              "type" => "log",
              "tags" => [
            [0] "beats_input_codec_plain_applied"
        ]
    }

  • 修改first-pipeline.conf，使用grok
1
2
3
4
5
6
7
8
9
10
11
12
13
input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}
output {
    stdout { codec => rubydebug }
}
  • 删除日志记录的节点，我们可以重头读取日志
1
2
# rm /usr/share/filebeat/bin/data/registry
# bin/logstash  --path.settings /etc/logstash  -f /etc/logstash/conf.d/first-pipeline.conf --config.reload.automatic
  • 可以看到我们获取到了的日志改变了，获取了更加详细的日志

  • 启用goip
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    stdout { codec => rubydebug }
}
  • 同样删除registry，重启程序，可以看到信息又更新了

indexing data into elasticsearch

  • 将first-pipeline.conf 再度修改，将output指向elasticsearch

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    input {
        beats {
            port => "5044"
        }
    }
    filter {
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}"}
        }
        geoip {
            source => "clientip"
        }
    }
    #output {
    #    stdout { codec => rubydebug }
    #}
    output {
        elasticsearch {
            hosts => [ "localhost:9200" ]
        }
    }

  • 同样删除registry，重启程序，然后再启动elasticsearch

1
2
3
# systemctl start elasticsearch.service
# curl -XGET 'http://localhost:9200/logstash-2017.06.23/_search?pretty&q=response=200'
# curl -XGET 'http://localhost:9200/logstash-2017.06.23/_search?pretty&q=geoip.city_name=Zhengzhou'
  • 命令中logstash-DATE，替换 DATE 变成正确的时间, 格式如下 YYYY.MM.DD，通过请求，我们可以获得类似如下的数据

  • 完整版
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
{
  "took" : 11,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 2.1282318,
    "hits" : [
      {
        "_index" : "logstash-2017.06.23",
        "_type" : "log",
        "_id" : "AVzTKaFXQUT1Lre_3sa9",
        "_score" : 2.1282318,
        "_source" : {
          "request" : "http://www.dajie.com/",
          "agent" : "\"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)\"",
          "geoip" : {
            "city_name" : "Zhengzhou",
            "timezone" : "Asia/Shanghai",
            "ip" : "115.61.84.162",
            "latitude" : 34.6836,
            "country_name" : "China",
            "country_code2" : "CN",
            "continent_code" : "AS",
            "country_code3" : "CN",
            "region_name" : "Henan",
            "location" : {
              "lon" : 113.5325,
              "lat" : 34.6836
            },
            "region_code" : "41",
            "longitude" : 113.5325
          },
          "offset" : 182,
          "auth" : "-",
          "ident" : "-",
          "input_type" : "log",
          "verb" : "GET",
          "source" : "/var/log/nginx/access.log",
          "message" : "115.61.84.162 - - [23/Jun/2017:06:50:10 +0800] \"GET http://www.dajie.com/ HTTP/1.1\" 200 612 \"http://www.dajie.com/\" \"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)\"",
          "type" : "log",
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "referrer" : "\"http://www.dajie.com/\"",
          "@timestamp" : "2017-06-23T04:15:32.219Z",
          "response" : "200",
          "bytes" : "612",
          "clientip" : "115.61.84.162",
          "@version" : "1",
          "beat" : {
            "hostname" : "iZm5e7jlki70utmw22zj76Z",
            "name" : "iZm5e7jlki70utmw22zj76Z",
            "version" : "5.4.2"
          },
          "host" : "iZm5e7jlki70utmw22zj76Z",
          "httpversion" : "1.1",
          "timestamp" : "23/Jun/2017:06:50:10 +0800"
        }
      },
      {
        "_index" : "logstash-2017.06.23",
        "_type" : "log",
        "_id" : "AVzTOcnEQUT1Lre_3sbu",
        "_score" : 2.1282318,
        "_source" : {
          "request" : "http://www.dajie.com/",
          "agent" : "\"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)\"",
          "geoip" : {
            "city_name" : "Zhengzhou",
            "timezone" : "Asia/Shanghai",
            "ip" : "115.61.84.162",
            "latitude" : 34.6836,
            "country_name" : "China",
            "country_code2" : "CN",
            "continent_code" : "AS",
            "country_code3" : "CN",
            "region_name" : "Henan",
            "location" : {
              "lon" : 113.5325,
              "lat" : 34.6836
            },
            "region_code" : "41",
            "longitude" : 113.5325
          },
          "offset" : 182,
          "auth" : "-",
          "ident" : "-",
          "input_type" : "log",
          "verb" : "GET",
          "source" : "/var/log/nginx/access.log",
          "message" : "115.61.84.162 - - [23/Jun/2017:06:50:10 +0800] \"GET http://www.dajie.com/ HTTP/1.1\" 200 612 \"http://www.dajie.com/\" \"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)\"",
          "type" : "log",
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "referrer" : "\"http://www.dajie.com/\"",
          "@timestamp" : "2017-06-23T04:33:13.588Z",
          "response" : "200",
          "bytes" : "612",
          "clientip" : "115.61.84.162",
          "@version" : "1",
          "beat" : {
            "hostname" : "iZm5e7jlki70utmw22zj76Z",
            "name" : "iZm5e7jlki70utmw22zj76Z",
            "version" : "5.4.2"
          },
          "host" : "iZm5e7jlki70utmw22zj76Z",
          "httpversion" : "1.1",
          "timestamp" : "23/Jun/2017:06:50:10 +0800"
        }
      }
    ]
  }
}
  • 去web上查看结果

参考资料：

坚持原创技术分享，您的支持将鼓励我继续创作！