OpenCV Android Studio

在Android Studio工程中使用Native的方式集成OpenCV

为什么要使用Native方式集成OpenCV

如果我们要处理的图片计算量不大，或者对处理速度不关注的时候，我们完全可以采用Java的来调用OpenCV。采用Java来调用OpenCV的集成方法非常简单，具体集成方法可以参考我的这个教程：https://panxsoft.coding.net/s/89438a3d-52b9-47e6-9c6b-23841f86cd8f

但是如果我们要进行实时处理，或者需要和其他的视觉库一起来使用，那么Java的处理方式就比较不友好了或者效率也不行(虽然OpenCV最终都是在C/C++上执行的，但是直接用C/C++来开发关键算法，明显会使得app的运行效率大大提升)。为了更快的执行效率，或者更好的与其他的视觉库融合，推荐采用Native的融合方式。

环境

Android Studio 3.2.1

NDK R16C

OpenCV 3.3.0

CMAKE 3.6.4

注意

关于NDK的安装

在国内由于网络的原因有时候通过Android Studio下载NDK的时候会造成下载不得部分文件缺失，这样NDK安装时候成功就不知道，在编译的时候回出现莫名其妙的错误(我就被这种错误深深的坑过)。所以安装NDK最好的办法是在Android官网上下载完整的安装包再解压到自己的电脑上。并在Android Studio中设置NDK的目录。

工程配置

更新Android SDK并安装NDK

下载OpenCV SDK的Android版本

去OpenCV官网下载OpenCV的Android版本:https://opencv.org/

创建一个新的Android Studio工程

选择Include C++ Support

选择一个Empty Activity

在C++ Support中勾选 -fexception和 -frtti

导入OpenCV Library Module

New -> Import Module

选择$(OpenCV for android SDK 所在目录)/sdk/java

一路next即可

修改OpenCV Library Module的build.gradle和你的app的build.gradle一致

例如：我的工程的的app的build.gradle的如下:

compileSdkVersion 28

defaultConfig {

minSdkVersion 18

targetSdkVersion 28

}

则应该将opencv library的build.gradle也修改为与app的build.gradle一致。下面是我修改之后

android {

compileSdkVersion 28 //与app的build.gradle一致

// buildToolsVersion "28.0.3" 这一行可以注视点

defaultConfig {

minSdkVersion 18 //与app的build.gradle一致

targetSdkVersion 28 //与app的build.gradle一致

}

buildTypes {

release {

minifyEnabled false

proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.txt'

}

}

}

在 app module dependency中添加OpenCV Module

File -> Project structure -> Module app -> Dependencies tab -> New module dependency -> choose OpenCV library module

在工程的app/src/main目录下创建名为jniLibs的文件夹，并将$(OpenCV for android SDK 所在目录)/sdk/native/libs下面的文件夹拷贝到该文件夹路径下

为了是最终打包的APK尽可能的小，可以只拷贝对应ABI的文件.

将(OpenCV for android SDK 所在目录)/sdk/native/jni/include文件夹拷贝到app/src/main/cpp文件下

网上也有人说不用拷贝，在 CMakeLists.txt 设置就行了，但是我实际这样操作话会出现莫名其妙的错误，目前还不知道错误原因，等弄明白了再更新

如果在创建Android Studio工程的时候选择Include C++ Support，则在app/src/main文件夹下面会自动出现cpp文件夹。如果是自己手工添加C++ Support的话，存放C++文件的文件夹是自己定义的，总之就是将include文件放在你存放C++文件的文件夹下.

配置CMakeLists.txt文件

配置CMakeLists.txt有两种方法，选择一种适合你自己的

第一种：这种方式比较简单，一般不会出现什么错误，但是打包的*.so文件会稍微大一点，初学者推荐始终这种配置方式

cmake_minimum_required(VERSION 3.4.1)

# OpenCV 配置

include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include)

add_library( lib_opencv SHARED IMPORTED )

set_target_properties(lib_opencv PROPERTIES IMPORTED_LOCATION ${CMAKE_CURRENT_SOURCE_DIR}/src/main/jniLibs/${ANDROID_ABI}/libopencv_java3.so)

# Creates and names a library, sets it as either STATIC

# or SHARED, and provides the relative paths to its source code.

# You can define multiple libraries, and CMake builds it for you.

# Gradle automatically packages shared libraries with your APK.

add_library( # Sets the name of the library.

native-lib

# Sets the library as a shared library.

SHARED

# Provides a relative path to your source file(s).

# Associated headers in the same location as their source

# file are automatically included.

src/main/cpp/native-lib.cpp )

# Searches for a specified prebuilt library and stores the path as a

# variable. Because system libraries are included in the search path by

# default, you only need to specify the name of the public NDK library

# you want to add. CMake verifies that the library exists before

# completing its build.

find_library( # Sets the name of the path variable.

log-lib

# Specifies the name of the NDK library that

# you want CMake to locate.

log )

# Specifies libraries CMake should link to your target library. You

# can link multiple libraries, such as libraries you define in the

# build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.

native-lib

# OpenCV lib

lib_opencv

# Links the target library to the log library

# included in the NDK.

${log-lib} )

第二种：将opencv中的*.so文件一个一个指定打包。这种方法可以最大限度的减少打包的*.so文件的大小。但是对于OpenCV有一定的要求，初学者不建议采用这种�配置方式。

# NDK的最小版本

cmake_minimum_required(VERSION 3.4.1)

# 显示CMake Build的输出信息

set(CMAKE_VERBOSE_MAKEFILE on)

set(libs "${CMAKE_SOURCE_DIR}/src/main/jniLibs")

include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include)

add_library(libopencv_java3 SHARED IMPORTED )

set_target_properties(libopencv_java3 PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_java3.so")

add_library(libopencv_calib3d STATIC IMPORTED )

set_target_properties(libopencv_calib3d PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_calib3d.a")

add_library(libopencv_core STATIC IMPORTED )

set_target_properties(libopencv_core PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_core.a")

add_library(libopencv_features2d STATIC IMPORTED )

set_target_properties(libopencv_features2d PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_features2d.a")

add_library(libopencv_flann STATIC IMPORTED )

set_target_properties(libopencv_flann PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_flann.a")

add_library(libopencv_highgui STATIC IMPORTED )

set_target_properties(libopencv_highgui PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_highgui.a")

add_library(libopencv_imgcodecs STATIC IMPORTED )

set_target_properties(libopencv_imgcodecs PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_imgcodecs.a")

add_library(libopencv_imgproc STATIC IMPORTED )

set_target_properties(libopencv_imgproc PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_imgproc.a")

add_library(libopencv_ml STATIC IMPORTED )

set_target_properties(libopencv_ml PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_ml.a")

add_library(libopencv_objdetect STATIC IMPORTED )

set_target_properties(libopencv_objdetect PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_objdetect.a")

add_library(libopencv_photo STATIC IMPORTED )

set_target_properties(libopencv_photo PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_photo.a")

add_library(libopencv_shape STATIC IMPORTED )

set_target_properties(libopencv_shape PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_shape.a")

add_library(libopencv_stitching STATIC IMPORTED )

set_target_properties(libopencv_stitching PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_stitching.a")

add_library(libopencv_superres STATIC IMPORTED )

set_target_properties(libopencv_superres PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_superres.a")

add_library(libopencv_video STATIC IMPORTED )

set_target_properties(libopencv_video PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_video.a")

add_library(libopencv_videoio STATIC IMPORTED )

set_target_properties(libopencv_videoio PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_videoio.a")

add_library(libopencv_videostab STATIC IMPORTED )

set_target_properties(libopencv_videostab PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_videostab.a")

add_library(libopencv_ts STATIC IMPORTED )

set_target_properties(libopencv_ts PROPERTIES

IMPORTED_LOCATION "${libs}/${ANDROID_ABI}/libopencv_ts.a")

add_library( # Sets the name of the library.

native-lib

# Sets the library as a shared library.

SHARED

# Provides a relative path to your source file(s).

# Associated headers in the same location as their source

# file are automatically included.

src/main/cpp/native-lib.cpp )

find_library( # Sets the name of the path variable.

log-lib

# Specifies the name of the NDK library that

# you want CMake to locate.

log)

target_link_libraries(native-lib android log

libopencv_java3 libopencv_calib3d libopencv_core libopencv_features2d libopencv_flann libopencv_highgui libopencv_imgcodecs

libopencv_imgproc libopencv_ml libopencv_objdetect libopencv_photo libopencv_shape libopencv_stitching libopencv_superres

libopencv_video libopencv_videoio libopencv_videostab

${log-lib}

)

如何指定编译的ABI可以在app的build.gradle中指定

apply plugin: 'com.android.application'

android {

compileSdkVersion 28

defaultConfig {

applicationId "com.xxxxx.xxxx"

minSdkVersion 18

targetSdkVersion 28

versionCode 1

versionName "1.0"

testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"

externalNativeBuild {

cmake {

cppFlags "-std=c++11 -frtti -fexceptions" //CMake编译支持

abiFilters 'armeabi-v7a', 'x86', 'x86_64', 'arm64-v8a', 'armeabi' //指定需要编译的ABI

}

}

}

buildTypes {

release {

minifyEnabled false

proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'

}

}

externalNativeBuild {

cmake {

path "CMakeLists.txt"

}

}

sourceSets{

main{

jniLibs.srcDirs = ['src/main/jniLibs']

}

}

}

dependencies {

implementation fileTree(include: ['*.jar'], dir: 'libs')

implementation 'com.android.support:appcompat-v7:28.0.0'

implementation 'com.android.support.constraint:constraint-layout:1.1.3'

testImplementation 'junit:junit:4.12'

androidTestImplementation 'com.android.support.test:runner:1.0.2'

androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.2'

implementation project(':openCVLibrary330')

}

Demo检查

要检查我们是否配置成功，可以先Build一下，如果一切正常则配置成功。否则根据提示修正错误即可。如果一切正常，我们来做一个从工程：打开摄像头并使用OpenCV提供Canndy算子来检测摄像头画面中对象轮廓。效果图如下：

效果图

为app的AndroidManifest.xml文件添加摄像头访问权限

修改activity_main.xml文件添加摄像头画面SurfaceView

xmlns:app="http://schemas.android.com/apk/res-auto"

xmlns:tools="http://schemas.android.com/tools"

android:layout_width="match_parent"

android:layout_height="match_parent"

tools:context=".MainActivity">

android:id="@+id/camera_surface"

android:layout_width="match_parent"

android:layout_height="match_parent" />

修改app/src/main/cpp下的native-lib.cpp文件，添加Canndy处理方法

#include

#include

#include

#include

#include

using namespace std;

using namespace cv;

extern "C" JNIEXPORT jstring JNICALL Java_com_seventythree_cvtest_MainActivity_stringFromJNI(

JNIEnv *env,

jobject /* this */) {

std::string hello = "Hello from C++";

return env->NewStringUTF(hello.c_str());

}

/**

* @brief 使用Canndy算子检测图像中的对象轮廓

* @param matAddrGray, Mat图像的内存地址

*/

extern "C" JNIEXPORT void JNICALL Java_com_seventythree_cvtest_MainActivity_CanndyDetect(

JNIEnv *env,

jobject thiz,

jlong matAddrGray) {

Mat &grayMat = *(Mat *) matAddrGray;

Canny(grayMat, grayMat, 50, 100);

}

修改MainActivity.java

package com.你自己的包名;

import android.Manifest;

import android.content.pm.PackageManager;

import android.support.annotation.NonNull;

import android.support.v4.app.ActivityCompat;

import android.support.v7.app.AppCompatActivity;

import android.os.Bundle;

import android.util.Log;

import android.view.SurfaceView;

import android.view.WindowManager;

import android.widget.TextView;

import android.widget.Toast;

import org.opencv.android.BaseLoaderCallback;

import org.opencv.android.CameraBridgeViewBase;

import org.opencv.android.LoaderCallbackInterface;

import org.opencv.android.OpenCVLoader;

import org.opencv.core.Mat;

public class MainActivity extends AppCompatActivity implements CameraBridgeViewBase.CvCameraViewListener2 {

private static final String TAG = "MainActivity";

private CameraBridgeViewBase mCameraBridgeViewBase;

private BaseLoaderCallback _baseLoaderCallback = new BaseLoaderCallback(this)

{

@Override

public void onManagerConnected(int status)

{

switch (status)

{

case LoaderCallbackInterface.SUCCESS:

{

Log.i(TAG, "OpenCV库加载成功");

// opencv初始化之后加载ndk打包的模块

System.loadLibrary("native-lib");

mCameraBridgeViewBase.enableView();

}break;

default:

{

super.onManagerConnected(status);

}break;

}

}

};

// // Used to load the 'native-lib' library on application startup.

// static {

// System.loadLibrary("native-lib");

// }

@Override

protected void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);

// 设置windows保持常量

getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);

setContentView(R.layout.activity_main);

// 在Android 6.0 +上可以

ActivityCompat.requestPermissions(MainActivity.this,

new String[]{Manifest.permission.CAMERA},

1);

mCameraBridgeViewBase = (CameraBridgeViewBase) findViewById(R.id.camera_surface);

mCameraBridgeViewBase.setVisibility(SurfaceView.VISIBLE);

mCameraBridgeViewBase.setCvCameraViewListener(this);

}

/**

* 重写onPause

*/

@Override

protected void onPause() {

super.onPause();

disableCamera();

}

/**

* 重写onResume

*/

@Override

protected void onResume() {

super.onResume();

if (!OpenCVLoader.initDebug())

{

Log.d(TAG, "应用内无法找到OpenCV库，使用OpenCV Manager进行初始化!");

OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION, this, _baseLoaderCallback);

}

else

{

Log.d(TAG, "使用应用内的OpenCV库进行初始化");

_baseLoaderCallback.onManagerConnected(LoaderCallbackInterface.SUCCESS);

}

}

/**

* 重修onDestroy

*/

@Override

protected void onDestroy() {

super.onDestroy();

disableCamera();

}

/**

* 摄像头授权回调

* @param requestCode

* @param permissions

* @param grantResults

*/

@Override

public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {

super.onRequestPermissionsResult(requestCode, permissions, grantResults);

switch (requestCode)

{

case 1:

{

// 如果用户取消授权，则result数组为空

if (grantResults.length > 0 && grantResults[0] == PackageManager.PERMISSION_GRANTED)

{

// 授权成功可以做一些你爱做的事了ˇˍˇ

}

else

{

// 授权失败了

Toast.makeText(MainActivity.this, "一定要授权才能使用呀", Toast.LENGTH_SHORT).show();

}

return;

}

// 其他的case大家根据自己的实际需求写吧

}

}

/**

* 禁用摄像头

*/

public void disableCamera()

{

if (mCameraBridgeViewBase !=null)

mCameraBridgeViewBase.disableView();

}

@Override

public void onCameraViewStarted(int width, int height) {

}

@Override

public void onCameraViewStopped() {

}

@Override

public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame)

{

Mat matGray = inputFrame.gray();

CanndyDetect(matGray.getNativeObjAddr());

return matGray;

}

/**

* A native method that is implemented by the 'native-lib' native library,

* which is packaged with this application.

*/

public native String stringFromJNI();

/**

* 对图像进行Canndy边缘检测

* @param matAddr 灰度图像的Mat地址

*/

public native void CanndyDetect(long matAddr);

}

常见问题

1. OpenCV JavaCameraView的画面旋转问题

今天晚上突然发现一个很严重的问题，使用本方法修改之后onCameraFrame中的Mat突然无法与布局中的JavaCameraView绑定了，具体原因正在排查中...

这是由于OpenCV的SDK中的JavaViewCamera.java中未对屏幕方向进行处理的原型。可以自己写适配屏幕方向的方法，这里仅以portrait为例演示如何旋转摄像头画面旋转90°的方法

首先采用反射的方法在JavaCameraView类中添加旋转摄像头的方法

/**

* 使用反射的方法调用摄像头的旋转方法

* @param camera 摄像头对象

* @param angle 旋转角度，逆时针为正，顺时针为负

*/

private void setDisplayOrientation(Camera camera, int angle)

{

Method setOrientation;

try {

setOrientation = camera.getClass().getMethod("setDisplayOrientation", new Class[]{int.class});

if (setOrientation != null)

setOrientation.invoke(camera, new Object[]{angle});

}catch (Exception e1) {}

}

修改JavaCameraView类的initializeCamera方法

首先定位到initializeCamera方法中如下代码块

/* Finally we are ready to start the preview */

Log.d(TAG, "startPreview");

mCamera.startPreview();

将上面的代码块修改成如下样子

/* Finally we are ready to start the preview */

Log.d(TAG, "startPreview");

// 修正摄像头画面 Added By shawnzhang

// mCamera.setDisplayOrientation(90); // 不采用仿射的方法

setDisplayOrientation(mCamera, 90); //旋转摄像头

mCamera.setPreviewDisplay(getHolder()); // 刷新Canvas

// 修正摄像头画面 Added By shawnzhang

mCamera.startPreview();

此时摄像头画面就在portrait方式下正常了，当然你也可以不采用反射的方法，直接采用注释掉的方法来处理，不过OpenCV的Java源码多采用反射的处理方式，这样做也是符合OpenCV的方式而已。

2. OpenCV的摄像头预览画面变形

默认情况下使用JavaCameraView直接打开摄像头的画面是变形的，这是由于JavaCameraView类的initializeCamera方法中调用的calculateCameraFrameSize方法的问题，OpenCV的原始代码如下：

if (sizes != null) {

/* Select the size that fits surface considering maximum size allowed */

Size frameSize = calculateCameraFrameSize(sizes, new JavaCameraSizeAccessor(), width, height);

/* Image format NV21 causes issues in the Android emulators */

if (Build.FINGERPRINT.startsWith("generic")

calculateCameraFrameSize在CameraBridgeViewBase.java中实现，参数sizes是摄像头支持的previewSize, 参数width和height是显示摄像头画面的Frame的宽和高，由调用initializeCamera方法的对象传入，

这样计算最佳FrameSize的参数都有了。在JavaViewCamera中创建一个根据摄像头支持的previewSize和显示摄像头画面的SurefaceView选择最佳FrameSize的方法，代码如下:

/**

* 根据显示摄像头画面的SurfaceView的尺寸选择最合适的FrameSize

* @param supportedSizes 摄像头支持的previewSize列表

* @param surfaceWidth 显示摄像头画面的SurfaceView的宽度

* @param surfaceHeight 显示摄像头画面的SurfaceView的高度

* @return 注意返回值的Size采用的是org.opencv.core.Size 而不是android.haraware.Camera.Size

*/

private Size getBestCameraFrameSize(List supportedSizes,int surfaceWidth, int surfaceHeight)

{

float tmp = 0.0f;

float minDiff = 100.0f;

int bestWidth = 0;

int bestHeight = 0;

float x_d_y = (float)surfaceWidth/(float)surfaceHeight;

Size best = null;

for (android.hardware.Camera.Size size : supportedSizes)

{

tmp = Math.abs(((float)size.height/(float)size.width)-x_d_y);

if (tmp < minDiff)

{

minDiff = tmp;

bestWidth = size.width;

bestHeight = size.height;

}

}

return new Size(bestWidth, bestHeight);

}

然后修改initializeCamera方法中计算最佳FrameSize的代码块如下所示:

if (sizes != null) {

/* Select the size that fits surface considering maximum size allowed */

// Size frameSize = calculateCameraFrameSize(sizes, new JavaCameraSizeAccessor(), width, height);

// 选择最适合的Frame的Size Add By Shawnzhang

Size frameSize = getBestCameraFrameSize(sizes, width, height);

// 选择最适合的Frame的Size Add By Shawnzhang

/* Image format NV21 causes issues in the Android emulators */

if (Build.FINGERPRINT.startsWith("generic")