今天，同事突然告诉我查不到了日志了。
于是登上日志服务器查看logstash日志发现：

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"********-2021-
01-16", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x15df4cd0>], :response=>{"index"=>
{"_index"=>"********-2021-01-16", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>
{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but 
this cluster currently has [999]/[1000] maximum shards open;"}}}}

重点在于

reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;

百度一下发现是elasticsearch7默认分片只有1000个，目前已经用完了，导致已经没法创建新的索引了。
然后查询到可以设置分片数量大小，于是记录一下：

临时设置

curl -XPUT -H "Content-Type:application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster": { "max_shards_per_node": 10000 } } }'

永久设置

curl -XPUT -H "Content-Type:application/json" http://localhost:9200/_cluster/settings -d '{ "persistent": { "cluster": { "max_shards_per_node": 10000 } } }'

设置为10000，我想应该可以顶个一两年了。