cover

springboot自定义ClassLoader实现同一个jar支持多版本的使用场景【附源码】

springboot自定义ClassLoader实现同一个jar支持多版本加载使用的方案，demo 源码验证在同一个系统既要支持hive3.1.0版本同时又要支持hive2.1.1版本的数据源功能;

lify365

3646人浏览 · 2022-11-22 11:26:04

lify365 · 2022-11-22 11:26:04 发布

springboot自定义ClassLoader实现同一个jar支持多版本的使用场景

背景

最近业务提出一个业务场景：系统目前支持hive3.1.0版本的数据源适配,但是有个别部门使用的数据源是hive2.1.1版本,但是hive3.1.0版本的驱动无法支持连接hive2.1.1的hive数据源;这就提出了新的目标：在同一个系统既要支持hive3.1.0版本同时又要支持hive2.1.1版本的数据源功能;
思路分析

1.java的执行都是通过类加载运行的,其加载运行过程如下：
加载 >> 验证 >> 准备 >> 解析 >> 初始化 >> 使用 >> 卸载
(不在详解,需要了解的百度一下吧),其多版本的兼容运行也离不开多版本的jar加载其运行：对于不同的hive版本分别classLoader不同的支持版本的lib jar就可以实现该目标;
2.java类加载器有引导类(Launcher)、扩展类(ExtClassLoader)、应用程序类加载器(AppClassLoader)以及支持自定义加载器,其JVM类的加载类有一个双亲委派的机制：加载某个类时会先委托父加载器寻找目标类，找不到再委托上层父加载器加载，如果所有父加载器在自己的加载类路径下都找不到目标类，则在自己的类加载路径中查找并载入目标类。正是该机制导致了系统无法支持同一个jar的多版本加载;解决该问题就只能自定义加载类，打破双亲委派机制,通过指定的lib jar路径去加载需要class 以达到支持不同版本功能;

经过如上的理论分析,实现方案有了理论支持,下面就直接开始代码实现上面的思路.
代码实现和验证

1.demo环境和前提准备：

    springboot 2.5.6
    mysql-connector-java 5.1.35
    hive 3.1.X需要的依赖包如：/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/
    hive2.1.1需要的依赖包路径如：/Users/lifangyu/soft/driver-lib/hive-jdbc-2.1.1/
    
    hive2.1.1 版本数据源信息
        url：jdbc:hive2://127.0.0.1:10001/demo
        user:test
        pwd:test
    hive3.1.X 版本数据源连接信息
        url:jdbc:hive2://127.0.0.1:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
        user:test
        pwd:test

2.自定义加载类实现,打破双亲委派机制,通过自定义指定的jar 文件路径加载类JarClassLoader,源码如下：

package com.bigdata.myClassLoader;

import java.io.*;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.ArrayList;
import java.util.List;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;

/**
 * description 自定义加载类：打破双全委派机制
 *
 * @author Cyber
 * <p> Created By 2022/11/22
 * @version 1.0
 */
public class JarClassLoader extends URLClassLoader {

    private static ThreadLocal<URL[]> threadLocal = new ThreadLocal<>();
    private URL[] allUrl;

    public JarClassLoader(String[] paths) {
        this(paths, JarClassLoader.class.getClassLoader());
    }

    public JarClassLoader(String[] paths, ClassLoader parent) {
        super(getURLs(paths), parent);
        // 当前线程防止重复读取文件信息，可使用其他缓存代替
        allUrl = threadLocal.get();

    }

    public JarClassLoader(URL[] urls, ClassLoader parent) {
        super(urls, parent);
    }

    /**
     * description 通过文件目录获取目录下所有的jar全路径信息
     *
     * @param paths 文件路径
     * @return java.net.URL[]
     * @author Cyber
     * <p> Created by 2022/11/22
     */
    private static URL[] getURLs(String[] paths) {
        if (null == paths || 0 == paths.length) {
            throw new RuntimeException("jar包路径不能为空.");
        }
        List<String> dirs = new ArrayList<String>();
        for (String path : paths) {
            dirs.add(path);
            JarClassLoader.collectDirs(path, dirs);
        }
        List<URL> urls = new ArrayList<URL>();
        for (String path : dirs) {
            urls.addAll(doGetURLs(path));
        }
        URL[] threadLocalurls = urls.toArray(new URL[0]);
        threadLocal.set(threadLocalurls);
        return threadLocalurls;
    }

    /**
     * description 递归获取文件目录下的根目录
     *
     * @param path      文件路径
     * @param collector 根目录
     * @return void
     * @author Cyber
     * <p> Created by 2022/11/22
     */
    private static void collectDirs(String path, List<String> collector) {
        if (null == path || "".equalsIgnoreCase(path)) {
            return;
        }
        File current = new File(path);
        if (!current.exists() || !current.isDirectory()) {
            return;
        }
        for (File child : current.listFiles()) {
            if (!child.isDirectory()) {
                continue;
            }
            collector.add(child.getAbsolutePath());
            collectDirs(child.getAbsolutePath(), collector);
        }
    }

    private static List<URL> doGetURLs(final String path) {
        if (null == path || "".equalsIgnoreCase(path)) {
            throw new RuntimeException("jar包路径不能为空.");
        }
        File jarPath = new File(path);
        if (!jarPath.exists() || !jarPath.isDirectory()) {
            throw new RuntimeException("jar包路径必须存在且为目录.");
        }

        FileFilter jarFilter = new FileFilter() {
            /**
             * description  判断是否是jar文件
             * @param pathname jar 全路径文件
             * @return boolean
             * @author Cyber
             * <p> Created by 2022/11/22
             */
            @Override
            public boolean accept(File pathname) {
                return pathname.getName().endsWith(".jar");
            }
        };
        File[] allJars = new File(path).listFiles(jarFilter);
        List<URL> jarURLs = new ArrayList<URL>(allJars.length);
        for (int i = 0; i < allJars.length; i++) {
            try {
                jarURLs.add(allJars[i].toURI().toURL());
            } catch (Exception e) {
                throw new RuntimeException("系统加载jar包出错", e);
            }
        }
        return jarURLs;
    }

    /**
     * description 重新loadClass加载过程，打破双亲委派机制，采用逆向双亲委派
     *
     * @param className 加载的类名
     * @return java.lang.Class<?>
     * @author Cyber
     * <p> Created by 2022/11/22
     */
    @Override
    public Class<?> loadClass(String className) throws ClassNotFoundException {
        if (allUrl != null) {
            String classPath = className.replace(".", "/");
            classPath = classPath.concat(".class");
            for (URL url : allUrl) {
                byte[] data = null;
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                InputStream is = null;
                try {
                    File file = new File(url.toURI());
                    if (file != null && file.exists()) {
                        JarFile jarFile = new JarFile(file);
                        if (jarFile != null) {
                            JarEntry jarEntry = jarFile.getJarEntry(classPath);
                            if (jarEntry != null) {
                                is = jarFile.getInputStream(jarEntry);
                                int c = 0;
                                while (-1 != (c = is.read())) {
                                    baos.write(c);
                                }
                                data = baos.toByteArray();
                                System.out.println("********找到classPath=" + classPath + "的jar=" + url.toURI().getPath() + "*******");
                                return this.defineClass(className, data, 0, data.length);
                            }
                        }
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                } finally {
                    try {
                        if (is != null) {
                            is.close();
                        }
                        baos.close();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                }

            }
        }
        // 未找到的情况下通过父类加载器加载
        return super.loadClass(className);
    }
}

3.当前线程ContextClassLoader的装载和卸载处理

package com.bigdata.myClassLoader;

/**
 * description 自定义classloader加载器当前线程的ContextClassLoader处理
 *
 * @author Cyber
 * <p> Created By 2022/11/22
 * @version 1.0
 */
public class JarClassLoaderSwapper {

    private ClassLoader storeClassLoader = null;

    private JarClassLoaderSwapper() {
    }

    public static JarClassLoaderSwapper newCurrentThreadClassLoaderSwapper() {
        return new JarClassLoaderSwapper();
    }

    /**
     * description 保存当前classLoader，并将当前线程的classLoader设置为所给classLoader
     *
     * @param classLoader
     * @return java.lang.ClassLoader
     * @author Cyber
     * <p> Created by 2022/11/22
     */
    public ClassLoader setCurrentThreadClassLoader(ClassLoader classLoader) {
        this.storeClassLoader = Thread.currentThread().getContextClassLoader();
        Thread.currentThread().setContextClassLoader(classLoader);
        return this.storeClassLoader;
    }

    /**
     * description 将当前线程的类加载器设置为保存的类加载
     *
     * @param
     * @return java.lang.ClassLoader
     * @author Cyber
     * <p> Created by 2022/11/22
     */
    public ClassLoader restoreCurrentThreadClassLoader() {
        ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
        Thread.currentThread().setContextClassLoader(this.storeClassLoader);
        return classLoader;
    }
}

结果验证

测试类：JarClassLoaderHiveTest

package com.bigdata.hive;

import com.bigdata.myClassLoader.JarClassLoader;
import com.bigdata.myClassLoader.JarClassLoaderSwapper;
import lombok.extern.slf4j.Slf4j;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.lang.reflect.InvocationTargetException;
import java.sql.*;
import java.util.Properties;

/**
 * description hive多版本自定义加载类测试
 *
 * @author Cyber
 * <p> Created By 2022/11/22
 * @version 1.0
 */
public class JarClassLoaderHiveTest {

    public static void main(String[] args) throws ClassNotFoundException, InstantiationException, IllegalAccessException, NoSuchMethodException, InvocationTargetException, SQLException {
        
        String jarUrl2 = "/Users/lifangyu/soft/driver-lib/hive-jdbc-2.1.1/"; //自己定义的测试jar包，不同版本打印内容不同
        /**
         * description lib/目录下的日子文件删除以防冲突
         * log4j-1.2-api-2.10.0.jar
         * log4j-api-2.10.0.jar
         * log4j-core-2.10.0.jar
         * log4j-slf4j-impl-2.10.0.jar
         * log4j-web-2.10.0.jar
         */
        String url = "jdbc:hive2://127.0.0.1:10001/ty";
        String user = "test";
        String pwd = "test";
        // String sql = "show databases";
        // String sql = "show tables";
        String sql = "select * from user_level_demo1 limit 10";
        testHiveJdbc(jarUrl2, url, user, pwd, sql);

        
        String jarUrl3 = "/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/";
        //自己定义的测试jar包，不同版本打印内容不同
        /**
         * description lib/目录下的日子文件删除以防冲突
         * log4j-1.2-api-2.10.0.jar
         * log4j-api-2.10.0.jar
         * log4j-core-2.10.0.jar
         * log4j-slf4j-impl-2.10.0.jar
         * log4j-web-2.10.0.jar
         */
        url = "jdbc:hive2://127.0.0.1:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2";
        user = "test";
        pwd = "test";
        String sql3 = "SELECT x.* FROM dc_dwa.dwa_d_bd_blend x where x.pro_id = 10 and x.contact_no='13009502690'";
        testHiveJdbc(jarUrl3, url, user, pwd, sql3);
    }

    private static void testHiveJdbc(String jarUrl, String url, String user, String pwd, String sql) throws InstantiationException, IllegalAccessException, ClassNotFoundException, SQLException {
        long start = System.currentTimeMillis();
        JarClassLoader jarLoader = new JarClassLoader(new String[]{jarUrl});
        JarClassLoaderSwapper classLoaderSwapper = JarClassLoaderSwapper.newCurrentThreadClassLoaderSwapper();
        classLoaderSwapper.setCurrentThreadClassLoader(jarLoader);
        Class<?> aClass = Thread.currentThread().getContextClassLoader().loadClass("org.apache.hive.jdbc.HiveDriver");
        classLoaderSwapper.restoreCurrentThreadClassLoader();

        Driver driver = (Driver) aClass.newInstance();
        Properties properties = new Properties();
        properties.put("user", user);
        properties.put("password", pwd);
        Connection conn = driver.connect(url, properties);
        PreparedStatement pstmt = (PreparedStatement) conn.prepareStatement(sql);
        ResultSet rs = pstmt.executeQuery();
        int col = rs.getMetaData().getColumnCount();//列数
        System.out.println("============================:" + jarUrl);
        while (rs.next()) {//一行一行输出
            for (int i = 1; i <= col; i++) {
                System.out.print(rs.getString(i) + "\t");//输出
                if ((i == 2) && (rs.getString(i).length() < 8)) {//输出制表符
                    System.out.print("\t");
                }
            }
            System.out.println("");
        }
        System.out.println("============================耗时：" + (System.currentTimeMillis() - start) + "ms");
    }
}

查看测试输出结果如下：

********找到classPath=com/google/common/base/Preconditions.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-2.1.1/guava-14.0.1.jar*******
********找到classPath=org/apache/hive/service/cli/ColumnBasedSet$1.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-2.1.1/hive-service-2.1.1.jar*******
********找到classPath=org/apache/hadoop/hive/serde2/thrift/ColumnBuffer$1.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-2.1.1/hive-serde-2.1.1.jar*******
********找到classPath=org/apache/hive/jdbc/HiveBaseResultSet$1.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-2.1.1/hive-jdbc-2.1.1.jar*******
1014040324171307	18547200338	V0150200	0472	null	40AAAAAA	1	0	1	202205	null	202203	202205	010	1	
1014040924175631	18547304560	V0150300	0473	null	50AAAAAA	3	0	1	202205	null	202203	202205	010	1	
1014041524181440	18647010211	V0152302	0470	null	40AAAAAA	2	0	1	202205	null	202203	202205	010	1	
1014042424194176	18547225591	V0150200	0472	null	40AAAAAA	2	0	1	202205	null	202203	202205	010	1	
1014042524195969	18604775277	V0152700	0477	null	40AAAAAA	2	0	1	202205	null	202112	202205	010	1	
1014042624197401	18648538891	V0152301	0475	null	40AAAAAA	2	0	1	202205	null	202203	202205	010	1	
1014042924201464	18647164248	V0150400	0476	null	40AAAAAA	1	0	1	202205	null	202111	202205	010	1	
1014042924203019	18604717645	V0150100	0471	null	50AAAAAA	4	0	1	202205	null	202203	202205	010	1	
1014043024203640	18547030504	V0152302	0470	null	40AAAAAA	2	0	1	202205	null	202203	202205	010	1	
1014043024203736	18647896194	V0152800	0478	null	40AAAAAA	2	0	1	202205	null	202205	202205	010	1	
============================耗时：7822ms
********找到classPath=org/apache/hive/jdbc/HiveDriver.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/hive-jdbc-3.1.2.jar*******
********找到classPath=org/apache/hive/jdbc/ZooKeeperHiveClientException.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/hive-jdbc-3.1.2.jar*******
********找到classPath=org/apache/hive/jdbc/HiveConnection.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/hive-jdbc-3.1.2.jar*******
********找到classPath=org/apache/thrift/TException.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/libthrift-0.9.3.jar*******
********找到classPath=org/apache/http/client/HttpClient.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/httpclient-4.5.2.jar*******
********找到classPath=org/apache/thrift/transport/TTransport.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/libthrift-0.9.3.jar*******
... ...
********找到classPath=org/apache/thrift/TBaseHelper$NestedStructureComparator.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/libthrift-0.9.3.jar*******
********找到classPath=org/apache/hive/service/cli/ColumnBasedSet$1.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/hive-service-3.1.2.jar*******
********找到classPath=org/apache/hadoop/hive/serde2/thrift/ColumnBuffer$1.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/hive-serde-3.1.2.jar*******
********找到classPath=org/apache/hive/jdbc/HiveBaseResultSet$1.class的jar=/Users/lifangyu/soft/driver-lib/hive-jdbc-3.1.2/hive-jdbc-3.1.2.jar*******
10	13009502690	40AAAAAA	047103676479	202211	15
10	13009502690	40AAAAAA	047103676479	202209	22	
10	13009502690	40AAAAAA	047103676479	202209	24	
============================耗时：59360ms

通过以上测试即可看到hive2.1.1版本和hive3.1.X版本运行分别使用的是各自相关依赖包加载运行，没有依赖冲突;

总结思考

以上只是这一种场景的思考解决,在实际的项目中,有关该类问题jar包冲突导致的种种问题都可以使用该思想去解决，当然以上只是抛砖引玉的解决思路，在生产使用还建议结合项目实际情况结合适配器等设计模式进行封装和优化以完成生产力使用;

华为云开发者联盟

为开发者提供学习成长、分享交流、生态实践、资源工具等服务，帮助开发者快速成长。

更多推荐

cover

解锁HDC 2024之旅：从购票到报名，全程攻略

华为云开发者联盟

cover

从原始边列表到邻接矩阵Python实现图数据处理的完整指南

华为云开发者联盟

cover

华为云云原生FinOps解决方案，释放云原生最大价值

华为云开发者联盟

所有评论(0)

查看更多评论

lify365

已为社区贡献1条内容