如同logstash,默认会生成filebeat-*的索引以及使用解压目录下的fields.yml(二进制压缩包)

这里的需求是解析生成的特定文件,每行记录为json格式的数据,需要:

  • 不同的文件生成不同的索引;
  • 类型需要定义,原始查询中对于字符串未使用.keyword,因为默认模板使用的是小于1024是keyword,反之是text,如果要当作keyword使用则是fieldName.keyword;

配置

setup.template.name: custom_name
setup.template.pattern: custom_name_*
setup.template.enabled: true
setup.template.overwrite: true
setup.template.fields: customFields.yml
setup.ilm.enabled: false
processors:
- drop_fields:
    fields: [log, host, input, agent, ecs]
    ignore_missing: false
filebeat.inputs:
- close_removed: true
  close_inactive: 5m
  type: log
  tags: [t1]
  clean_removed: true
  enabled: true
  json: {keys_under_root: true, overwrite_keys: true}
  paths: [/data/t1_*]
- close_removed: true
  close_inactive: 5m
  type: log
  tags: [t2]
  clean_removed: true
  enabled: true
  json: {keys_under_root: true, overwrite_keys: true}
  paths: [/data/t2_*]
output.elasticsearch:
  indices:
  - index: custom_name_t1
    when.contains: {tags: t1}
  - when.contains: {tags: t2}
    index: custom_name_t2
  hosts: ['127.0.0.1:9200']

setup.template.name 设置一个新的模板,模板的名称
setup.template.pattern 模板匹配那些索引
setup.template.enabled: false 关掉默认的模板配置
setup.template.overwrite: false 是否覆盖现有模板
when.contains: 包含
keys_under_root: true 将field展开到最外层(the custom fields are stored as top-level fields in the output document)

自定义模板请查看:Configuration-template

文件路径请查看:Configure project paths

字段定义

setup.template.fields: customFields.yml(路径请查看上述链接),文件格式如下:

- key: custom_name
  title: custom_name
  description: >
    custom fields
  fields:
  # some desc
    - name: t1
      type: keyword
    - name: t2
      type: keyword
    - name: t3
      type: ip
    - name: t4
      type: integer
    - name: t5
      type: ip
    - name: t6
      type: integer
    - name: t7
      type: ip
    - name: t8
      type: text
    - name: t9
      type: date
    - name: t10
      type: long

启动

$ ./filebeat -e -c config/customConf.yml
Logo

为开发者提供学习成长、分享交流、生态实践、资源工具等服务,帮助开发者快速成长。

更多推荐