ES 全量数据查询
1、场景引入queryForList默认查询10条每次,最多可设置为10000条,意味着该api最多只能查询10000条数数据elasticsearchTemplate.queryForList(query, ComparisonPlatformGoodsItemSearchView.class);类似同步之类的逻辑,es特定的业务索引几百万乃至上千万的数据,显然,仅仅通过elasticsearc
·
1、场景引入
queryForList默认查询10条每次,最多可设置为10000条,意味着该api最多只能查询10000条数数据
elasticsearchTemplate.queryForList(query, ComparisonPlatformGoodsItemSearchView.class);
类似同步之类的逻辑,es特定的业务索引几百万乃至上千万的数据,显然,仅仅通过elasticsearchTemplate.queryForList()是无法满足的。
2、解决方案:
第一种:通过scroll(类似mysql游标)
官方手册:https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.scroll
(注意:spring-data-elasticsearch不同版本api名称不一样,4.x.x和3.x.x存在不一样,诸如:searchScrollStart/scrollStart、searchScrollContinue/scrollContinue等)
代码示例:
public void createPurchaseActivityConfigurationGoodsSearchData(PurchaseActivityConfiguration paramBean) {
BoolQueryBuilder boolQuery = QueryBuilders.boolQuery()
.must(QueryBuilders.matchQuery("brandId", paramBean.getBrandId()))
.must(QueryBuilders.matchQuery("comparisonCategoryId", paramBean.getCategoryId()));
NativeSearchQuery query = new NativeSearchQueryBuilder().withQuery(boolQuery).build();
List<ComparisonPlatformGoodsItemSearchView> goodsItemViews = new ArrayList<>();
ScrolledPage scroll = elasticsearchTemplate.startScroll(SCROLL_TIME, query, ComparisonPlatformGoodsItemSearchView.class);
while (scroll.hasContent()) {
goodsItemViews.addAll(scroll.getContent());
scroll = elasticsearchTemplate.continueScroll(scroll.getScrollId(), SCROLL_TIME, ComparisonPlatformGoodsItemSearchView.class);
}
elasticsearchTemplate.clearScroll(scroll.getScrollId());
if (ObjectUtil.isNotNull(paramBean.getCommissionRatioThreshold())) {
purchaseHighSubsidyGoodsSearchRepository.saveAll(covertPurchaseHighSubsidyGoodsSearchView(goodsItemViews));
log.info(">>>>>> createPurchaseHighSubsidyGoodsSearchData success");
}
}
第二种:通过Stream
更多推荐
已为社区贡献1条内容
所有评论(0)