OpenHarmony VideoEncoder js接口开发
由于本人对于视频编码的知识了解较少，所以提供给app使用的接口主要参考native接口来设计，接口文件index.d.ts如下/*** @since 9* @since 9/*** @since 9* @since 9* @since 9* @since 9* @since 9* @since 9* @since 9* @since 9* @since 9* @since 9* @since 9*
·
OpenHarmony 添加VideoEncoder的ts接口开发记录
一、背景
由于项目中需要将camera导出的yuv数据进行推流，所以需要VideoEncoder来编码yuv数据程h264视频流，但是查看官方网站发现openharmony关于编码和解码这部分只有native接口。所以需要使用native接口实现视频编码并通过napi导出ts接口提供给app使用。而且在3.2release版本中 video encoder的native接口中没有buffer模式的入口，只能通过surface接口把数据传到video encoder中，而surface相关的接口并没有开放到ndk中。所以想要在3.2 release分支下开发video encoder的ts接口，只能将surface相关的接口导出到ndk中才能在app中开发。但是我在查看官方master分支中发现，在master分支上，官方将codec相管的接口都从/foundation/multimedia/player_framework/这个仓库移动到了/foundation/multimedia/av_codec/这个仓库下，并且增加了buffer模式的入口。，而且在3.2 release版本中虽然buffer模式的接口没有开放，但是底层其实还是实现了对应的接口的。所以现在有两个方案：
方案一、使用现成的surface接口，将surface相关的接口导出来即可开发
方案二、使用buffer接口，需要将最新的buffer模式的接口加入到目前的代码中
综合考虑我选择了方案二 项目地址：https://gitee.com/yw2219247615/camera-demo_-ohos
二、定义视频编码的接口
由于本人对于视频编码的知识了解较少，所以提供给app使用的接口主要参考native接口来设计，接口文件index.d.ts如下
//index.d.ts
import media from '@ohos.multimedia.media';
declare namespace avcodec {
/**
* Creates a video encoder instance.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @return Returns VideoEncoder interface
*/
function createVideoEncoder():VideoEncoder
/**
* Manages video encoder. Before calling an VideoEncoder method, you must use createVideoEncoder()
* to create an VideoEncoder instance.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
*/
interface VideoEncoder {
/**
* To prepare the internal resources of the encoder, the Configure interface must be called before
* calling this interface.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @return Returns AV_ERR_OK if the execution is successful,
* otherwise returns a specific error code, refer to {@link CodecErrCode}
*/
prepare():CodecErrCode
/**
* To configure the video encoder, typically, you need to configure the description information of the
* encoded video track. This interface must be called before Prepare is called.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @param an AvFormat instance {@link AvFormat}
* @return Returns AV_ERR_OK if the execution is successful,
* otherwise returns a specific error code, refer to {@link CodecErrCode}
*/
configure(format:AvFormat):CodecErrCode
/**
* Start the encoder, this interface must be called after the Prepare is successful.
* calling this interface.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @return Returns AV_ERR_OK if the execution is successful,
* otherwise returns a specific error code, refer to {@link CodecErrCode}
*/
start():CodecErrCode
/**
* Stop the encoder. After stopping, you can re-enter the Started state through Start.
* calling this interface.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @return Returns AV_ERR_OK if the execution is successful,
* otherwise returns a specific error code, refer to {@link CodecErrCode}
*/
stop():CodecErrCode
/**
* Clear output data buffered in the encoder. After this interface is called, all the Buffer
* indexes previously reported through the asynchronous callback will be invalidated, make sure not to access the
* Buffers corresponding to these indexes. this interface must be called before stop interface.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @return Returns AV_ERR_OK if the execution is successful,
* otherwise returns a specific error code, refer to {@link CodecErrCode}
*/
flush():CodecErrCode
/**
* Reset the encoder. To continue coding, you need to call the configure interface again to
* configure the encoder instance.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @return Returns AV_ERR_OK if the execution is successful,
* otherwise returns a specific error code, refer to {@link CodecErrCode}
*/
reset():CodecErrCode
/**
* Get the description information of the output data of the encoder, refer to {@link AvFormat} for details.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @return Returns a string obj about AvFormat
*/
getOutputDescription():string
/**
* Clear the internal resources of the encoder and destroy the encoder instance
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @return Returns AV_ERR_OK if the execution is successful,
* otherwise returns a specific error code, refer to {@link CodecErrCode}
*/
release():CodecErrCode
/**
* Send frame data to video encoder, this interface must be called after the start is successful.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @param data Image data buffer
* @param dataSize Image data byte size, Indicates the size of valid data in the buffer
* @param timeStamp Image data timestamp
*/
sendFrameData(data:Uint8Array,dataSize:number,timeStamp:number):void
/**
* Register or unregister listens for video encoder events.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @param type Type of the video encoder event to listen for.
* @param callback Callback used to listen for the video encoder output buffer event.
*/
on(type:"OnOutputBufferAvailable",callback:(data: ArrayBuffer, dataSize: number,pts:number,flags:number) => void):void
/**
* Register or unregister listens for video encoder events.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @param type Type of the video encoder event to listen for.
* @param callback Callback used to listen for the video encoder error event.
*/
on(type:"onError",callback:(errCoder: number) => void):void
/**
* Register or unregister listens for video encoder events.
* @since 9
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
* @param type Type of the video encoder event to listen for.
* @param callback Callback used to listen for the video encoder output format change event.
*/
on(type:"OnOutputFormatChanged",callback:(format: AvFormat) => void):void
}
interface AvFormat {
/**设置编码器的类型目前只支持视频编码器*/
mime:media.CodecMimeType;
/**图像帧的宽度 */
width:number;
/**图像帧的高度 */
height:number;
/**输入图像的格式 */
pixelFormat:VideoPixelFormat;
/**图像编码的帧率*/
frameRate:number;
/**图像编码的码率*/
encodeBitRate:number;
/** I 帧采集的间隔 单位：毫秒*/
iFrameIntervalMs:number;
/**编码质量:0 - 100*/
codecQuality:number;
/**编码的profile:
*AVC_PROFILE_BASELINE = 0,
*AVC_PROFILE_HIGH = 4,
*AVC_PROFILE_MAIN = 8,
*/
codecProfile:number;
videoEncodeBitRateMode: VideoEncodeBitrateMode;
}
enum VideoPixelFormat {
/**
* yuv 420 planar.
*/
YUVI420 = 1,
/**
* NV12. yuv 420 semiplanar.
*/
NV12 = 2,
/**
* NV21. yvu 420 semiplanar.
*/
NV21 = 3,
/**
* format from surface.
*/
SURFACE_FORMAT = 4,
/**
* RGBA.
*/
RGBA = 5,
}
enum CodecErrCode {
/**
* the operation completed successfully.
*/
AV_ERR_OK = 0,
/**
* no memory.
*/
AV_ERR_NO_MEMORY = 1,
/**
* opertation not be permitted.
*/
AV_ERR_OPERATE_NOT_PERMIT = 2,
/**
* invalid argument.
*/
AV_ERR_INVALID_VAL = 3,
/**
* IO error.
*/
AV_ERR_IO = 4,
/**
* network timeout.
*/
AV_ERR_TIMEOUT = 5,
/**
* unknown error.
*/
AV_ERR_UNKNOWN = 6,
/**
* media service died.
*/
AV_ERR_SERVICE_DIED = 7,
/**
* the state is not support this operation.
*/
AV_ERR_INVALID_STATE = 8,
/**
* unsupport interface.
*/
AV_ERR_UNSUPPORT = 9,
/**
* extend err start.
*/
AV_ERR_EXTEND_START = 100,
}
enum VideoEncodeBitrateMode {
/**
* constant bit rate mode.
*/
CBR = 0,
/**
* variable bit rate mode.
*/
VBR = 1,
/**
* constant quality mode.
*/
CQ = 2,
}
}
export default avcodec;
上面的文件主要时定义了一些encoder的行为和相关的枚举类，基本也都是从native层拷贝过来的，AvFormat主要用来配置videoencoder的参数。
三、建立NAPI层
这里我定义VideoEncoder的napi层为video_encoder_napi.cpp
首先需要先定义和上层ts接口对应的napi接口如下：
//video_encoder_napi.cpp
napi_value VideoEncoderNapi::Init(napi_env env, napi_value exports) {
AVCODEC_LOGD("%s ", __FUNCTION__);
napi_property_descriptor properties[] = {
//这里的这些宏定义都被我导入到了common/native_common.h中其实都是系统源码中的
DECLARE_NAPI_FUNCTION("configure", Configure),
DECLARE_NAPI_FUNCTION("prepare", Prepare),
DECLARE_NAPI_FUNCTION("start", Start),
DECLARE_NAPI_FUNCTION("stop", Stop),
DECLARE_NAPI_FUNCTION("flush", Flush),
DECLARE_NAPI_FUNCTION("reset", Reset),
DECLARE_NAPI_FUNCTION("getOutputDescription", GetOutputDescription),
DECLARE_NAPI_FUNCTION("sendFrameData", SendFrameData),
DECLARE_NAPI_FUNCTION("release", Release),
DECLARE_NAPI_FUNCTION("on", On)};
napi_property_descriptor staticProperty[] = {
DECLARE_NAPI_STATIC_FUNCTION("createVideoEncoder", CreateVideoEncoder),
};
napi_value constructor = nullptr;
napi_status status = napi_define_class(env, CLASS_NAME.c_str(), NAPI_AUTO_LENGTH, Constructor, nullptr,
sizeof(properties) / sizeof(properties[0]), properties, &constructor);
CHECK_AND_RETURN_RET_LOG(status == napi_ok, nullptr, "Failed to define VideoEncoder class");
status = napi_create_reference(env, constructor, 1, &constructor_);
CHECK_AND_RETURN_RET_LOG(status == napi_ok, nullptr, "Failed to create reference of constructor");
status = napi_set_named_property(env, exports, CLASS_NAME.c_str(), constructor);
CHECK_AND_RETURN_RET_LOG(status == napi_ok, nullptr, "Failed to set constructor");
status = napi_define_properties(env, exports, sizeof(staticProperty) / sizeof(staticProperty[0]), staticProperty);
CHECK_AND_RETURN_RET_LOG(status == napi_ok, nullptr, "Failed to define static function");
AVCODEC_LOGD("Init success");
return exports;
}
在上面的接口中CreateVideoEncoder会触发整个video_encoder_napi的创建，当调用configure时才会通过上层传下来的avformat中的mime真正的创建encoder，并配置相关的参数，剩下的操作基本就是一个搬运工将ts的指令传递到native的encoder即可。
3.1 CreateVideoEncoder
napi_value VideoEncoderNapi::CreateVideoEncoder(napi_env env, napi_callback_info info) {
AVCODEC_LOGD("%s ", __FUNCTION__);
napi_value result = nullptr;
napi_get_undefined(env, &result);
//创建napi对象
napi_value ctor;
napi_status status;
status = napi_get_reference_value(env, constructor_, &ctor);
if (status != napi_ok) {
AVCODEC_LOGE("napi_get_reference_value fail");
return result;
}
status = napi_new_instance(env, ctor, 0, nullptr, &result);
if (status == napi_ok) {
AVCODEC_LOGD("New instance success");
} else {
AVCODEC_LOGD("New instance could not be obtained");
}
return result;
}
CreateVideoEncoder函数非常简单就是创建napi对象，其中会调用到VideoEncoderNapi::Constructor函数，如下；
napi_value VideoEncoderNapi::Constructor(napi_env env, napi_callback_info info) {
AVCODEC_LOGD("%s ", __FUNCTION__);
napi_value result = nullptr;
napi_get_undefined(env, &result);
napi_value jsThis = nullptr;
napi_status status = napi_get_cb_info(env, info, nullptr, nullptr, &jsThis, nullptr);
if (status != napi_ok) {
AVCODEC_LOGE("Failed to retrieve details about the callback");
return result;
}
if (jsThis != nullptr) {
VideoEncoderNapi *videoEncoderNapi = new (std::nothrow) VideoEncoderNapi();
if (videoEncoderNapi != nullptr) {
videoEncoderNapi->env_ = env;
status = napi_wrap(env, jsThis, reinterpret_cast<void *>(videoEncoderNapi),
VideoEncoderNapi::Destructor, nullptr, nullptr);
if (status != napi_ok) {
delete videoEncoderNapi;
AVCODEC_LOGE("Failed to warp native instance!");
return result;
}
}
}
AVCODEC_LOGD("Constructor success");
return jsThis;
}
上面的这些基本都属于模板了。
3.2 Configure
napi_value VideoEncoderNapi::Configure(napi_env env, napi_callback_info info) {
AVCODEC_LOGD("%s ", __FUNCTION__);
napi_value result = nullptr;
napi_get_undefined(env, &result);
//获取arkts传过来参数信息
napi_value thisVar = nullptr;
napi_value args[ARGS1] = {nullptr};
size_t argCount = ARGS1;
napi_status status;
status = napi_get_cb_info(env, info, &argCount, args, &thisVar, nullptr);
CHECK_AND_RETURN_RET_LOG(status == napi_ok, result, "failed to napi_get_cb_info");
//获取AvFormat
AvFormat format = getAvFormat(env, args[PARAM0]);
//获取napi实例
VideoEncoderNapi *videoEncoderNapi = nullptr;
getVideoEncoderNapi(env, thisVar, &videoEncoderNapi);
if (videoEncoderNapi == nullptr)
return result;
int32_t ret;
videoEncoderNapi->videoEncoder = VideoEncoderFactory::CreateByMime(format.codec_mime);
ret = videoEncoderNapi->videoEncoder->configure(&format);
//创建回调接口
videoEncoderNapi->codecCallback = std::make_shared<AvCodecCallbackNapi>();
videoEncoderNapi->videoEncoder->SetCallback(videoEncoderNapi->codecCallback);
AvCodecNapiUtils::createNapiInt32Value(env, ret, result);
return result;
}
在napi层当调用了configure后才会真正创建native层的video encoder可以看到上面通过调用VideoEncoderFactory::CreateByMime创建一个videoEncoder对象并保存在了videoEncoderNapi的videoEncoder指针里，后面我们的操作都会通过这个指针进行。
3.3 Start
napi_value VideoEncoderNapi::Start(napi_env env, napi_callback_info info) {
AVCODEC_LOGD("%s ", __FUNCTION__);
napi_value result = nullptr;
napi_value thisVar = nullptr;
napi_get_undefined(env, &result);
napi_status status;
AVCODEC_NAPI_GET_JS_OBJ_WITH_ZERO_ARGS(env, info, status, thisVar);
CHECK_AND_RETURN_RET_LOG(status == napi_ok, result, "failed to napi_get_cb_info");
VideoEncoderNapi *videoEncoderNapi = nullptr;
getVideoEncoderNapi(env, thisVar, &videoEncoderNapi);
if (videoEncoderNapi == nullptr)
return result;
int32_t ret;
ret = videoEncoderNapi->videoEncoder->start();
AvCodecNapiUtils::createNapiInt32Value(env, ret, result);
return result;
}
可以看到正如上面所说都是通过获取napi实例，接着通过它的videoEncoder来操作video encoder的，其他的流程控制比如stop reset release flush等等都是一个模板。这里就不再细说了
四、封装video encoder
由于video encoder的ndk中都只是一些接口，所以我这边将这些接口封装成了一个类VideoEncoder，在上面的3.2小节中我们知道videoencoder是通过VideoEncoderFactory创建的，当它初始化时就会创建native video encoder并设置callback
VideoEncoder::VideoEncoder(std::string mime){
AVCODEC_LOGD("VideoEncoder create mime =%s",mime.c_str());
venc_ = OH_VideoEncoder_CreateByMime(mime.c_str());
if(venc_ == nullptr){
AVCODEC_LOGD("venc_ is null");
}
setEncoderCallback();
setVideoEncoderMap(venc_,this);
}
int32_t VideoEncoder::setEncoderCallback(){
AVCODEC_CHECK_AND_RETURN_RET_LOG(venc_ != nullptr, AV_ERR_UNKNOWN,"Fatal: OH_VideoEncoder_CreateByMime fail");
struct OH_AVCodecAsyncCallback callback;
callback.onError = VideoEncoder::OnError;
callback.onStreamChanged = VideoEncoder::OnStreamChanged;
callback.onNeedInputData = VideoEncoder::OnNeedInputData;
callback.onNeedOutputData = VideoEncoder::OnNewOutputData;
return OH_VideoEncoder_SetCallback(venc_, callback,nullptr);
}
void VideoEncoder::setVideoEncoderMap(OH_AVCodec* codec,VideoEncoder* encoder){
std::lock_guard<std::mutex> lock(mutex_);
encoderMap_[codec] = encoder;
}
setEncoderCallback主要是设置一些接口给native video encoder来接收输入的buffer(用来填充待编码的数据)和已经编码的buffer数据以及一些错误和变化。setVideoEncoderMap则是将当前创建的native vidoe encoder和我们的VideoEncoder对象对应起来，这是因为OH_AVCodecAsyncCallback的成员函数要求是静态函数，但是如果我们的VideoEncoder对象创建多个，那么无法找到对应的VideoEncoder对象去回调，所以我这边使用了一个map去存储对应关系，这个map的key是OH_AVCodec指针，value是VideoEncoder指针。
4.1 数据流转
当video encoder启动后底层会回调待填充数据的buffer上来，我这里会将这个buffer存储到一个队列中，当上层ts传入数据下来时，我从这个队列中拿出一个buffer将数据拷贝进去，接着调用OH_VideoEncoder_PushInputData将数据发送给native video encoder。代码实现如下：
//native video encoder回调可用的buffer
void VideoEncoder::OnNeedInputData(OH_AVCodec *codec, uint32_t index, OH_AVMemory *data, void *userData){
//从map中获取到encoder对象
VideoEncoder* encoder = GetVideoEncoder(codec);
if (encoder != nullptr) {
//将buffer存放到队列中
encoder->pushInputBufferToQueue(index,data);
}
}
int32_t VideoEncoder::sendFrameData(uint8_t *data, int32_t size, int32_t timestamp){
//当上层的待编码的数据传过来时
AVCODEC_LOGD("sendFrameData data = %d",data);
AVCODEC_CHECK_AND_RETURN_RET_LOG(venc_ != nullptr, AV_ERR_UNKNOWN,"Fatal: OH_VideoEncoder_CreateByMime fail");
std::lock_guard<std::mutex> lock(queueMutex_);
if(indexQueue.size() <= 0){
AVCODEC_LOGE("indexQueue is empty");
return AV_ERR_NO_MEMORY;
}
uint32_t index = indexQueue.front();
auto buffer = inBufferQueue.front();
//从队列中拿出buffer
OH_AVCodecBufferAttr info;
if(size == 0){
info.size = 0;
info.offset = 0;
info.pts = 0;
info.flags = AVCODEC_BUFFER_FLAGS_EOS;
OH_VideoEncoder_PushInputData(venc_, index, info);
AVCODEC_LOGE("send data len == 0");
}
info.size = size;
info.offset = 0;
info.pts = timestamp;
//拷贝数据
memcpy(OH_AVMemory_GetAddr(buffer), data, OH_AVMemory_GetSize(buffer));
//将buffer 推给native video encoder去编码 //这里调用的是buffer模式的发送数据3.2 release分支需要增加此接口并导出
int32_t ret = OH_VideoEncoder_PushInputData(venc_,index,info);
indexQueue.pop();
inBufferQueue.pop();
return ret;
}
当数据编码后会通过VideoEncoder::OnNewOutputData接收到
void VideoEncoder::OnNewOutputData(OH_AVCodec *codec, uint32_t index, OH_AVMemory *data,OH_AVCodecBufferAttr *attr, void *userData){
VideoEncoder* encoder = GetVideoEncoder(codec);
if (encoder != nullptr) {
encoder->avcodecCb->OnOutputBufferAvailable(OH_AVMemory_GetAddr(data), attr->size, attr->pts, attr->flags);
}
OH_VideoEncoder_FreeOutputData(codec,index);
}
这里通过回调接口将数据回调到上层后我们需要调用OH_VideoEncoder_FreeOutputData将buffer归还给native video encoder来维持数据流转。
五、增加buffer模式的发送数据接口
From 72ee572a520604d23dd2aee9064d4fc252462df4 Mon Sep 17 00:00:00 2001
From: yuwei <yuw@guideir.com>
Date: Fri, 2 Jun 2023 11:06:36 +0800
Subject: [PATCH] =?UTF-8?q?feat:=E5=A2=9E=E5=8A=A0video=20encoder=20buffer?=
=?UTF-8?q?=E6=A8=A1=E5=BC=8F=E5=B9=B6=E5=AF=BC=E5=87=BAndk=E6=8E=A5?=
=?UTF-8?q?=E5=8F=A3?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
---
.../capi/avcodec/native_video_encoder.cpp | 67 ++++++++++++++++++-
.../kits/c/native_avcodec_videoencoder.h | 27 ++++++++
.../libnative_media_venc.ndk.json | 4 +-
.../codec/hdi_plugins/hdi_codec_util.cpp | 2 +
4 files changed, 98 insertions(+), 2 deletions(-)
diff --git a/player_framework/frameworks/native/capi/avcodec/native_video_encoder.cpp b/player_framework/frameworks/native/capi/avcodec/native_video_encoder.cpp
index 07b9989..8ac87ea 100644
--- a/player_framework/frameworks/native/capi/avcodec/native_video_encoder.cpp
+++ b/player_framework/frameworks/native/capi/avcodec/native_video_encoder.cpp
@@ -39,10 +39,12 @@ struct VideoEncoderObject : public OH_AVCodec {
const std::shared_ptr<AVCodecVideoEncoder> videoEncoder_;
std::list<OHOS::sptr<OH_AVMemory>> memoryObjList_;
OHOS::sptr<OH_AVFormat> outputFormat_ = nullptr;
+ OHOS::sptr<OH_AVFormat> inputputFormat_ = nullptr;
std::shared_ptr<NativeVideoEncoderCallback> callback_ = nullptr;
std::atomic<bool> isFlushing_ = false;
std::atomic<bool> isStop_ = false;
std::atomic<bool> isEOS_ = false;
+ bool isInputSurfaceMode_ = false;
};
class NativeVideoEncoderCallback : public AVCodecCallback {
@@ -82,7 +84,12 @@ public:
MEDIA_LOGD("At flush, eos or stop, no buffer available");
return;
}
- callback_.onNeedInputData(codec_, index, nullptr, userData_);
+ OH_AVMemory *data = nullptr;
+ if (!videoEncObj->isInputSurfaceMode_) {
+ data = GetInputData(codec_, index);
+ CHECK_AND_RETURN_LOG(data != nullptr, "Data is nullptr, get input data failed");
+ }
+ callback_.onNeedInputData(codec_, index, data, userData_);
}
}
@@ -115,6 +122,30 @@ public:
}
private:
+ OH_AVMemory *GetInputData(struct OH_AVCodec *codec, uint32_t index)
+ {
+ CHECK_AND_RETURN_RET_LOG(codec != nullptr, nullptr, "Codec is nullptr!");
+ CHECK_AND_RETURN_RET_LOG(codec->magic_ == AVMagic::MEDIA_MAGIC_VIDEO_ENCODER, nullptr, "Codec magic error!");
+
+ struct VideoEncoderObject *videoEncObj = reinterpret_cast<VideoEncoderObject *>(codec);
+ CHECK_AND_RETURN_RET_LOG(videoEncObj->videoEncoder_ != nullptr, nullptr, "Video decoder is nullptr!");
+
+ std::shared_ptr<AVSharedMemory> memory = videoEncObj->videoEncoder_->GetInputBuffer(index);
+ CHECK_AND_RETURN_RET_LOG(memory != nullptr, nullptr, "Memory is nullptr, get input buffer failed!");
+
+ for (auto &memoryObj : videoEncObj->memoryObjList_) {
+ if (memoryObj->IsEqualMemory(memory)) {
+ return reinterpret_cast<OH_AVMemory *>(memoryObj.GetRefPtr());
+ }
+ }
+
+ OHOS::sptr<OH_AVMemory> object = new (std::nothrow) OH_AVMemory(memory);
+ CHECK_AND_RETURN_RET_LOG(object != nullptr, nullptr, "AV memory create failed");
+
+ videoEncObj->memoryObjList_.push_back(object);
+ return reinterpret_cast<OH_AVMemory *>(object.GetRefPtr());
+ }
+
OH_AVMemory *GetOutputData(struct OH_AVCodec *codec, uint32_t index)
{
CHECK_AND_RETURN_RET_LOG(codec != nullptr, nullptr, "input codec is nullptr!");
@@ -319,6 +350,7 @@ OH_AVErrCode OH_VideoEncoder_GetSurface(OH_AVCodec *codec, OHNativeWindow **wind
*window = CreateNativeWindowFromSurface(&surface);
CHECK_AND_RETURN_RET_LOG(*window != nullptr, AV_ERR_INVALID_VAL, "CreateNativeWindowFromSurface failed!");
+ videoEncObj->isInputSurfaceMode_ = true;
return AV_ERR_OK;
}
@@ -402,3 +434,36 @@ OH_AVErrCode OH_VideoEncoder_SetCallback(
return AV_ERR_OK;
}
+OH_AVErrCode OH_VideoEncoder_PushInputData(struct OH_AVCodec *codec, uint32_t index, OH_AVCodecBufferAttr attr)
+{
+ CHECK_AND_RETURN_RET_LOG(codec != nullptr, AV_ERR_INVALID_VAL, "Codec is nullptr!");
+ CHECK_AND_RETURN_RET_LOG(codec->magic_ == AVMagic::MEDIA_MAGIC_VIDEO_ENCODER, AV_ERR_INVALID_VAL,
+ "Codec magic error!");
+ CHECK_AND_RETURN_RET_LOG(attr.size >= 0, AV_ERR_INVALID_VAL, "Invalid buffer size!");
+
+ struct VideoEncoderObject *videoEncObj = reinterpret_cast<VideoEncoderObject *>(codec);
+ CHECK_AND_RETURN_RET_LOG(videoEncObj->videoEncoder_ != nullptr, AV_ERR_INVALID_VAL, "Video encoder is nullptr!");
+
+ struct AVCodecBufferInfo bufferInfo;
+ bufferInfo.presentationTimeUs = attr.pts;
+ bufferInfo.size = attr.size;
+ bufferInfo.offset = attr.offset;
+ enum AVCodecBufferFlag bufferFlag = static_cast<enum AVCodecBufferFlag>(attr.flags);
+
+ int32_t ret = videoEncObj->videoEncoder_->QueueInputBuffer(index, bufferInfo, bufferFlag);
+ CHECK_AND_RETURN_RET_LOG(ret == MSERR_OK, AV_ERR_OPERATE_NOT_PERMIT, "Video encoder push input data failed!");
+ if (bufferFlag == AVCODEC_BUFFER_FLAG_EOS) {
+ videoEncObj->isEOS_.store(true);
+ }
+ return AV_ERR_OK;
+}
+
+OH_AVErrCode OH_VideoEncoder_IsValid(OH_AVCodec *codec, bool *isValid)
+{
+ CHECK_AND_RETURN_RET_LOG(codec != nullptr, AV_ERR_INVALID_VAL, "Codec is nullptr!");
+ CHECK_AND_RETURN_RET_LOG(codec->magic_ == AVMagic::MEDIA_MAGIC_VIDEO_ENCODER, AV_ERR_INVALID_VAL,
+ "Codec magic error!");
+ CHECK_AND_RETURN_RET_LOG(isValid != nullptr, AV_ERR_INVALID_VAL, "Input isValid is nullptr!");
+ *isValid = true;
+ return AV_ERR_OK;
+}
diff --git a/player_framework/interfaces/kits/c/native_avcodec_videoencoder.h b/player_framework/interfaces/kits/c/native_avcodec_videoencoder.h
index a429309..2610f00 100644
--- a/player_framework/interfaces/kits/c/native_avcodec_videoencoder.h
+++ b/player_framework/interfaces/kits/c/native_avcodec_videoencoder.h
@@ -207,6 +207,33 @@ OH_AVErrCode OH_VideoEncoder_FreeOutputData(OH_AVCodec *codec, uint32_t index);
*/
OH_AVErrCode OH_VideoEncoder_NotifyEndOfStream(OH_AVCodec *codec);
+/**
+ * @brief Submit the input buffer filled with data to the video encoder.
+ * @syscap SystemCapability.Multimedia.Media.VideoEncoder
+ * @param codec Pointer to an OH_AVCodec instance
+ * @param index Enter the index value corresponding to the Buffer
+ * @param attr Information describing the data contained in the Buffer
+ * @return Returns AV_ERR_OK if the execution is successful,
+ * otherwise returns a specific error code, refer to {@link OH_AVErrCode}
+ * @since 10
+ * @version 4.0
+ */
+OH_AVErrCode OH_VideoEncoder_PushInputData(OH_AVCodec *codec, uint32_t index, OH_AVCodecBufferAttr attr);
+
+/**
+ * @brief Check whether the current codec instance is valid. It can be used fault recovery or app
+ * switchback from the background
+ * @syscap SystemCapability.Multimedia.Media.VideoEncoder
+ * @param codec Pointer to an OH_AVCodec instance
+ * @param isValid Pointer to an bool instance, true: the codec instance is valid, false: the codec
+ * instance is invalid
+ * @return Returns AV_ERR_OK if the execution is successful,
+ * otherwise returns a specific error code, refer to {@link OH_AVErrCode}
+ * @since 10
+ * @version 4.0
+ */
+OH_AVErrCode OH_VideoEncoder_IsValid(OH_AVCodec *codec, bool *isValid);
+
/**
* @brief The bitrate mode of video encoder.
* @syscap SystemCapability.Multimedia.Media.VideoEncoder
diff --git a/player_framework/interfaces/kits/c/video_encoder/libnative_media_venc.ndk.json b/player_framework/interfaces/kits/c/video_encoder/libnative_media_venc.ndk.json
index e2264a4..a62e0c1 100644
--- a/player_framework/interfaces/kits/c/video_encoder/libnative_media_venc.ndk.json
+++ b/player_framework/interfaces/kits/c/video_encoder/libnative_media_venc.ndk.json
@@ -13,5 +13,7 @@
{ "name": "OH_VideoEncoder_SetParameter" },
{ "name": "OH_VideoEncoder_GetSurface" },
{ "name": "OH_VideoEncoder_FreeOutputData" },
- { "name": "OH_VideoEncoder_NotifyEndOfStream" }
+ { "name": "OH_VideoEncoder_NotifyEndOfStream" },
+ { "name": "OH_VideoEncoder_PushInputData" },
+ { "name": "OH_VideoEncoder_IsValid" }
]
2.25.1
六、存在的问题
1.在持续的编码的数据过程中发现内存一直在持续增加，可能出现了内存泄漏
加入官方的patch：https://gitee.com/openharmony/arkui_napi/commit/420a9434606c2b1b6be726eb1b2f6c34b2faff94
并且加入如下的代码
--- a/player_framework/frameworks/js/avcodec/avcodec_callback_napi.cpp
+++ b/player_framework/frameworks/js/avcodec/avcodec_callback_napi.cpp
@@ -38,6 +38,9 @@ void AvCodecCallbackNapi::OnOutputBufferAvailableAsync(BufferInfo &info) const{
int32_t flags = info.flags;
if(onOutputBufferAvailableCb_env == nullptr || onOutputBufferAvailableCb == nullptr) return;
if(data != nullptr && size > 0){
+ napi_handle_scope scope;
+ napi_open_handle_scope(onOutputBufferAvailableCb_env,&scope);
+ if(scope == nullptr) return;
napi_value args[ARGS4] = {0};
napi_create_int32(onOutputBufferAvailableCb_env,size,&args[PARAM1]);
napi_create_int32(onOutputBufferAvailableCb_env,pts,&args[PARAM2]);
@@ -50,6 +53,7 @@ void AvCodecCallbackNapi::OnOutputBufferAvailableAsync(BufferInfo &info) const{
AVCODEC_LOGE("AvCodecCallbackNapi::%s CreateArrayBuffer fail",__FUNCTION__);
free(napiArrBuf);
napiArrBuf = nullptr;
+ napi_close_handle_scope(onOutputBufferAvailableCb_env, scope);
return;
}
napi_value callbackRet = nullptr;
@@ -59,6 +63,7 @@ void AvCodecCallbackNapi::OnOutputBufferAvailableAsync(BufferInfo &info) const{
AVCODEC_LOGE("AvCodecCallbackNapi::%s napi_get_reference_value err[%{public}d]",__FUNCTION__, status);
free(napiArrBuf);
napiArrBuf = nullptr;
+ napi_close_handle_scope(onOutputBufferAvailableCb_env, scope);
return;
}
status = napi_call_function(onOutputBufferAvailableCb_env, nullptr, callback, ARGS4, args, &callbackRet);
@@ -69,6 +74,7 @@ void AvCodecCallbackNapi::OnOutputBufferAvailableAsync(BufferInfo &info) const{
}
free(napiArrBuf);
napiArrBuf = nullptr;
+ napi_close_handle_scope(onOutputBufferAvailableCb_env, scope);
}
}
可以降低内存增加的速度，但是无法彻底解决内存增长的问题，目前怀疑是napi层的接口存在内存泄漏已经向官方提了问题单，需要持续跟踪。
2023.07.26 拉取openharmony的master分支编译验证发现此问题已经解决。
华为开发者空间，是为全球开发者打造的专属开发空间，汇聚了华为优质开发资源及工具，致力于让每一位开发者拥有一台云主机，基于华为根生态开发、创新。
更多推荐
1
0- 0
已为社区贡献7条内容
所有评论(0)