1、场景引入


queryForList默认查询10条每次,最多可设置为10000条,意味着该api最多只能查询10000条数数据
elasticsearchTemplate.queryForList(query, ComparisonPlatformGoodsItemSearchView.class);

类似同步之类的逻辑,es特定的业务索引几百万乃至上千万的数据,显然,仅仅通过elasticsearchTemplate.queryForList()是无法满足的。

2、解决方案:

第一种:通过scroll(类似mysql游标)

官方手册:https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.scroll

(注意:spring-data-elasticsearch不同版本api名称不一样,4.x.x和3.x.x存在不一样,诸如:searchScrollStart/scrollStart、searchScrollContinue/scrollContinue等)

代码示例:

public void createPurchaseActivityConfigurationGoodsSearchData(PurchaseActivityConfiguration paramBean) {
        BoolQueryBuilder boolQuery = QueryBuilders.boolQuery()
                .must(QueryBuilders.matchQuery("brandId", paramBean.getBrandId()))
                .must(QueryBuilders.matchQuery("comparisonCategoryId", paramBean.getCategoryId()));

        NativeSearchQuery query = new NativeSearchQueryBuilder().withQuery(boolQuery).build();
        List<ComparisonPlatformGoodsItemSearchView> goodsItemViews = new ArrayList<>();


        ScrolledPage scroll = elasticsearchTemplate.startScroll(SCROLL_TIME, query, ComparisonPlatformGoodsItemSearchView.class);
        while (scroll.hasContent()) {
            goodsItemViews.addAll(scroll.getContent());
            scroll = elasticsearchTemplate.continueScroll(scroll.getScrollId(), SCROLL_TIME, ComparisonPlatformGoodsItemSearchView.class);
        }
        elasticsearchTemplate.clearScroll(scroll.getScrollId());



        if (ObjectUtil.isNotNull(paramBean.getCommissionRatioThreshold())) {
            purchaseHighSubsidyGoodsSearchRepository.saveAll(covertPurchaseHighSubsidyGoodsSearchView(goodsItemViews));
            log.info(">>>>>> createPurchaseHighSubsidyGoodsSearchData success");
        }
    }

 

第二种:通过Stream

Logo

华为开发者空间,是为全球开发者打造的专属开发空间,汇聚了华为优质开发资源及工具,致力于让每一位开发者拥有一台云主机,基于华为根生态开发、创新。

更多推荐