springboot利用druid和mybatis批量数据写入clickhouse时,刚开始的时候,完全按照mysql的写法,出现无法写入的问题。经过不断的尝试,发现所使用的驱动包,驱动类配置以及sql写法上都需要特别注意,不然批量写入会抛出各种异常导致写入失败。

虽然clickhouse的官方说明中能够支持大部分的sql语法,在某些特定的场景下,比如批量写数据,还是需要做一些特别的配置。

这里用的clickhouse的版本是:version 22.2.2.1

1. 添加依赖

这个是可以实现批量写入的版本。

<!--clickhouse-->
<dependency>
    <groupId>com.clickhouse</groupId>
    <artifactId>clickhouse-jdbc</artifactId>
    <version>0.3.2-patch8</version>
</dependency>

刚开始的时候添加的是下面这个版本:

<!--
<dependency>
    <groupId>ru.yandex.clickhouse</groupId>
    <artifactId>clickhouse-jdbc</artifactId>
    <version>0.2.4</version>
</dependency>
-->

使用这个版本的时候,批量写入数据总是报错。

2. 配置连接信息

url: jdbc:clickhouse://192.168.17.81:8123/default
username: username
password: password
# driverClassName: ru.yandex.clickhouse.ClickHouseDriver
driverClassName: com.clickhouse.jdbc.ClickHouseDriver

之前配置为ru.yandex.clickhouse.ClickHouseDriver时,批量写入报错。

3. mapper.xml中的sql写法

<insert id="insertBatchCkPerm" useGeneratedKeys="false" parameterType="java.util.List">
        insert into plat_access_perm(access_id, developer_id, developer_name, access_key, access_secret)
        FORMAT Values
        <foreach collection="list" item="item" index="index" separator=",">
        (#{item.accessId},#{item.developerId},#{item.developerName},#{item.accessKey},#{item.accessSecret})
        </foreach>
</insert>

这里需要特别说明的是:按照mysql的写法,只需要values关键字即可,但是这里使用values的时候,单条记录写入是可以的,但是多条记录就会报错,报的是空指针错误。需要修改为FORMAT Values

useGeneratedKeys="false" 设置时为了避免自动生成主键id,clickhouse不支持。

4. mapper接口的写法

public int insertBatchCkPerm(List<PlatAccessPerm> list);

5. 遇到过的报错信息

报错信息如下:

### Error updating database. Cause: java.sql.SQLFeatureNotSupportedException ### The error may exist in file

Cause: java.sql.SQLFeatureNotSupportedException ; null; nested exception is java.sql.SQLFeatureNotSupportedException

错误消息:

nested exception is org.apache.ibatis.executor.ExecutorException: Error preparing statement. Cause: java.lang.NullPointerException

错误堆栈:

org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.executor.ExecutorException: Error preparing statement.  Cause: java.lang.NullPointerException
	at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:96)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:441)
	at com.sun.proxy.$Proxy84.insert(Unknown Source)
	at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:272)
	at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62)
	at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:145)
	at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
	at com.sun.proxy.$Proxy193.insertBatchCkPerm(Unknown Source)
	at com.ftsafe.operate.service.impl.PlatAccessPermServiceImpl.insertBatchCkPerm(PlatAccessPermServiceImpl.java:90)
	at com.ftsafe.operate.service.impl.PlatAccessPermServiceImpl$$FastClassBySpringCGLIB$$71c31b4d.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)	
	......		
Caused by: org.apache.ibatis.executor.ExecutorException: Error preparing statement.  Cause: java.lang.NullPointerException
	at org.apache.ibatis.executor.statement.BaseStatementHandler.prepare(BaseStatementHandler.java:97)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.prepare(RoutingStatementHandler.java:59)
	at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:87)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:49)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)	
	... 133 common frames omitted
Caused by: java.lang.NullPointerException: null
	at com.clickhouse.jdbc.internal.ClickHouseConnectionImpl.prepareStatement(ClickHouseConnectionImpl.java:618)
	at com.clickhouse.jdbc.ClickHouseConnection.prepareStatement(ClickHouseConnection.java:104)
	at com.alibaba.druid.filter.FilterChainImpl.connection_prepareStatement(FilterChainImpl.java:535)
	at com.alibaba.druid.filter.FilterAdapter.connection_prepareStatement(FilterAdapter.java:908)
	at com.alibaba.druid.filter.FilterEventAdapter.connection_prepareStatement(FilterEventAdapter.java:116)
	at com.alibaba.druid.filter.FilterChainImpl.connection_prepareStatement(FilterChainImpl.java:531)
	at com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl.prepareStatement(ConnectionProxyImpl.java:326)
	at com.alibaba.druid.pool.DruidPooledConnection.prepareStatement(DruidPooledConnection.java:362)
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.ibatis.logging.jdbc.ConnectionLogger.invoke(ConnectionLogger.java:55)
	at com.sun.proxy.$Proxy97.prepareStatement(Unknown Source)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.instantiateStatement(PreparedStatementHandler.java:86)
	at org.apache.ibatis.executor.statement.BaseStatementHandler.prepare(BaseStatementHandler.java:88)
	... 151 common frames omitted
13:22:56.656 [http-nio-80-exec-8] WARN  o.s.w.s.m.m.a.ExceptionHandlerExceptionResolver - [logException,208] - Resolved [org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.executor.ExecutorException: Error preparing statement.  Cause: java.lang.NullPointerException]

以上错误时,批量插入多条数据时,没有添加FORMAT时出现的报错信息

Logo

为开发者提供学习成长、分享交流、生态实践、资源工具等服务,帮助开发者快速成长。

更多推荐