Timelapse Sample. by AdityaKBhadragond14 · Pull Request #352 · Apra-Labs/ApraPipes

AdityaKBhadragond14 · 2024-05-15T11:33:56Z

IMPORTANT: All PRs must be linked to an issue (except for extremely trivial and straightforward changes).

Description
This PR contains the Surveillance summary timelapse based on motion detection sample number 7 from above linked issue.
This sample utilizes the following modules:

Mp4ReaderSource
MotionVectorExtractor
ColorConversionXForm
H264EncoderNVCodec
Mp4WriterSink

The Mp4ReaderSource reads a mp4 file. Mp4ReaderSource next is MotionVectorExtractor which gets the frames which match the given threshold and which are in motion. After MotionVectorExtractor the frames are passed to ColorConversionFormX to convert from BGR tor RGB and then from RGB to YUV420Planar because the H264EncoderNVCodec module only accepts YUV frames. After color conversion next is H264EncoderNVCodec which encodes the frames. After H264EncoderNVCodec next is Mp4WriterSink which writes the frames to disk.
Also in the MotionVectorExtractor I have introduced a flag sendOverlayFrame which when enabled sends the overlay frame else does not send the overlay frames. Since we dont need the overlay frames in this sample setting it to false.
This PR also has the fix for the issue Raw yuv420 output from motion extractor module is having horizontally layered bands on the frame.

Alternative(s) considered

Have you considered any alternatives? And if so, why have you chosen the approach in this PR?

Type

Feature and Bug Fix.

Screenshots (if applicable)

Checklist

I have read the Contribution Guidelines
I have written Unit Tests
I have discussed my proposed solution with code owners in the linked issue(s) and we have agreed upon the general approach

…Queues and seek feature

mraduldubey

formatting and other comments, restructure the format of tests, new impl for tests is required

mraduldubey · 2024-05-20T08:11:27Z

 		return ModuleProps::getSerializeSize() + sizeof(sendDecodedFrame) + sizeof(motionVectorThreshold);
 	}
 	bool sendDecodedFrame = false;
+	bool sendOverlayFrame = false;


should be added in serialization part below

mraduldubey · 2024-05-20T08:15:07Z

 			{
 				outFrame = makeFrameWithPinId(0, motionVectorPinId);
 			}
+			if (sendOverlayFrame == false) {


formatting

put a comment on what is this change about

mraduldubey · 2024-05-20T08:15:37Z

 	memset(&pDstInfo, 0, sizeof(SBufferInfo));
-	outFrame = makeFrameWithPinId(mMotionVectorSize, motionVectorPinId);
-	mMotionVectorData = static_cast<int16_t*>(outFrame->data());
+	if (sendOverlayFrame) 


formatting, in vs code use Ctrl+K, F to auto format

mraduldubey · 2024-05-20T08:16:18Z

 	}
-
-	if ((!sDecParam.bParseOnly) && (pDstInfo.pDst[0] != nullptr) && (mMotionVectorSize != mWidth * mHeight * 8))
+	if(sendOverlayFrame == false){


what is this change about ? why is this reqd in the sample PR

This is required to determine whether the frames are in the motion or not when we set the sendOverlayFrame to false.

mraduldubey · 2024-05-20T08:18:45Z

@@ -0,0 +1,10 @@
+#dependencies
+
+find_package(Threads REQUIRED)


Threads is required here, why ?

Not required forgot to remove this from the CMakeLists was just testing something. Removing it.

mraduldubey · 2024-05-20T08:27:39Z

+  ${LIBRE_INC_DIR}
+  ${NVCODEC_INCLUDE_DIR}
+)
+target_link_libraries(


If I remove this then how do I link the aprapipes lib with the samples?

remove the libraries that are not required by the sample boss

mraduldubey · 2024-05-20T08:27:49Z

+find_library(APRA_PIPES_LIB NAMES aprapipes)
+add_library(example_aprapipes STATIC ${SOURCE})
+add_executable(runner_test ${UT_FILE})
+target_include_directories ( example_aprapipes PRIVATE


call it samples

Renaming it to timelapsesample would be better as there will be multiple samples with this name.

mraduldubey · 2024-05-20T08:28:49Z

+int main() {
+  LoggerProps loggerProps;
+  loggerProps.logLevel = boost::log::trivial::severity_level::info;
+  Logger::setLogLevel(boost::log::trivial::severity_level::info);


both 79 & 80 are not required. one is enough

mraduldubey · 2024-05-20T08:29:43Z

+  bool stopPipeline();
+
+private:
+  PipeLine pipeline;


add a method called test, and call from test added in timelapse-sample/ - it should use step - not the pipeline

mraduldubey · 2024-05-20T08:30:25Z

+
+	timelapsePipeline -> startPipeline();
+
+	timelapsePipeline -> stopPipeline();


this is not a test. it has no checks. also, need to use step() instead

Done added an assertion to check the frame size.

mohammedzakikochargi · 2024-06-03T15:48:31Z

-void DetailFfmpeg::getMotionVectors(frame_container& frames, frame_sp& outFrame, frame_sp& decodedFrame)
-{
+
+bool discardNoisyFrames(int black_pixel_percent, cv::Mat image) {


We should also check if the pixels are all white

mohammedzakikochargi · 2024-06-03T15:49:55Z

-	{
+	if ((!sDecParam.bParseOnly) && (pDstInfo.pDst[0] != nullptr) &&
+		(mMotionVectorSize != mWidth * mHeight * 8) &&
+		(abs(avgX) > threshold || abs(avgY) > threshold)) {


Add a new bool prop and enable avg based thresholding only then

I discussed this with Mradul he suggested this should not be sample specific. The improvement should be for the module and not sample specific.

mohammedzakikochargi · 2024-06-03T15:50:33Z

-
-		cv::cvtColor(yuvImgCV, bgrImg, cv::COLOR_YUV2BGR_I420);
-		frames.insert(make_pair(rawFramePinId, decodedFrame));
+		if (!discardNoisyFrames(97, bgrImg)) {


This should be also checked in the ffmpeg strategy

AdityaKBhadragond14 · 2024-06-05T08:58:23Z

I have addressed the comments please have a look @mohammedzakikochargi .

…ed comments.

mraduldubey

make some minor changes

mraduldubey · 2024-06-18T08:33:50Z

+  ${Boost_LIBRARIES}
+  ${FFMPEG_LIBRARIES}
+  ${OpenCV_LIBRARIES}
+  ${JETSON_LIBS}


do you need all libs in the aprapipessampleut ? I dont think so.

mraduldubey · 2024-06-18T08:47:11Z

+  ${LIBRE_INC_DIR}
+  ${NVCODEC_INCLUDE_DIR}
+)
+target_link_libraries(


remove the libraries that are not required by the sample boss

mraduldubey · 2024-06-18T08:49:43Z

+  mColorchange1 = boost::shared_ptr<ColorConversion>(new ColorConversion(
+      ColorConversionProps(ColorConversionProps::BGR_TO_RGB)));
+  //convert frames from RGB to YUV420PLANAR
+  //the two step color change is done because H264Encoder takes YUV data and we don't have direct BGR to YUV.


delete after YUV data

mraduldubey · 2024-06-18T08:50:47Z

+  return true;
+}
+
+bool TimelapsePipeline::stopPipeline() {


format this file with Ctrl+k,f

mraduldubey · 2024-06-18T08:51:01Z

+  //the two step color change is done because H264Encoder takes YUV data and we don't have direct BGR to YUV.
+  mColorchange2 = boost::shared_ptr<ColorConversion>(new ColorConversion(
+      ColorConversionProps(ColorConversionProps::RGB_TO_YUV420PLANAR)));
+  mSync = boost::shared_ptr<Module>(


why is sync required

sync is required by the encoder.

Further points:

They should be very easy to build and run

MUST not assume hadware GPUs for them to run / have counterpart software implementations

sample example for valve module as well as seek and flush queue features

…raPipes into ns/samples_timelapse

Sample: Face Detection

Running pipeline for face detection wip as per the structure changes in cmake changes in constructor added sample example to create thumbnail from mp4 video reduced thread sleep seconds added new sample which will start video from beginning using flushAllQueues and seek feature refactored CMake list and renamed files using resolved time stamp when seeking timelapse sample timelaspe file changes samples WIP. added main to sample cpp file. removed sample test file from tests. Improved motion vector data code. using absolute path. Reverting unwanted changes. Fixed the ghost channel issue in MotionVectorExtractor. revert vcpkg. changes. taking video paths from terminal and modified cmake. vcpkg baseline revert. reset vcpkg head. vcpkg revert vcpkg revert. update submodule update vcpkg final Corrected cmake for samples. Corrected base cmake for samples. added test thumbnail sample test for play mp4 from beginning updated main CMake refactored test and CMake file refactored CMake and file structure for thumbnail samples refactored CMake and file structure for play mp4 from beginning sample base CMake changes 1. Fixed the props for Mp4ReaderSource and Mp4WriterSink 2. Fixed the memory leak issue in MotionVectorExtractor Corrected cmake and test for timelapse sample. 1. Removed the sendOverlayFrames prop to remove overlay frames, instead setting the RAW_IMAGE output pin to MotionVectorExtractor. 2. Enhanced the threshold logic to get the frames based on average of the vectors. 3. Removed not required tests. Algo to remove noisy black frames. files format. Addressed comments. Added readme file for samples. code refactoring and renaming. Addressed comments. removed not required libs and file formatting. restored files relay sample. base cmake changes. Corrected linking libraries in timelapse sample cmake. rename.

…agond14/ApraPipes into ab/timelapse_sample # Conflicts: # base/CMakeLists.txt

AdityaKBhadragond14 and others added 14 commits May 8, 2024 12:55

timelapse sample

e7b8c0b

timelaspe file changes

0b02ac2

wip sample

55235e4

samples WIP.

c116aad

added main to sample cpp file.

213bf2c

removed sample test file from tests.

5cb1025

added sample example to create thumbnail from mp4 video

9c70041

Improved motion vector data code.

2f3adb6

reduced thread sleep seconds

738763b

using absolute path.

5315e62

Reverting unwanted changes.

b89e3a6

added new sample which will start video from beginning using flushAll…

ab3f182

…Queues and seek feature

refactored CMake list and renamed files

f1d8cc0

using resolved time stamp when seeking

da51b44

mraduldubey reviewed May 20, 2024

View reviewed changes

SatwikSShanbhag and others added 15 commits May 20, 2024 15:20

Merge branch 'ab/timelapse_sample' into ss/sample_example

ad28405

Fixed the ghost channel issue in MotionVectorExtractor.

ad0752f

revert vcpkg.

005cd59

changes.

e9ccde5

taking video paths from terminal and modified cmake.

a5df32a

vcpkg baseline revert.

cce5ba2

reset vcpkg head.

87c26d2

vcpkg revert

4627b36

vcpkg revert.

31d8bf5

update submodule

518a75e

update vcpkg final

b45c423

Merge branch 'Apra-Labs:main' into ab/timelapse_sample

95bdf12

Corrected cmake for samples.

e0f0e91

Corrected base cmake for samples.

dc1c0a6

Running pipeline for face detection

29287b1

mohammedzakikochargi reviewed Jun 3, 2024

View reviewed changes

Addressed comments.

f07bc4e

AdityaKBhadragond14 requested a review from mohammedzakikochargi June 5, 2024 08:57

nsabale7 and others added 2 commits June 14, 2024 14:27

changes in cmake

a2102f1

Added readme file for samples. code refactoring and renaming. Address…

06f679e

…ed comments.

AdityaKBhadragond14 requested a review from mraduldubey June 17, 2024 10:29

mraduldubey requested changes Jun 18, 2024

View reviewed changes

removed not required libs and file formatting.

133d93b

AdityaKBhadragond14 requested a review from mraduldubey June 18, 2024 10:45

SatwikSShanbhag and others added 13 commits June 18, 2024 19:37

Merge branch 'ab/timelapse_sample' into ss/sample_example

5888659

restored files

fab05b3

Merge branch 'main' into ab/timelapse_sample

d920279

Merge branch 'ab/timelapse_sample' into ss/sample_example

7478f05

Merge pull request #2 from SatwikSShanbhag/ss/sample_example

7f79097

sample example for valve module as well as seek and flush queue features

Merge branch 'ab/timelapse_sample' into ns/samples_timelapse

1c2fa9d

changes in constructor

064cce5

Merge branch 'ns/samples_timelapse' of https://github.com/nsabale7/Ap…

197b75d

…raPipes into ns/samples_timelapse

Merge pull request #3 from nsabale7/ns/samples_timelapse

1031260

Sample: Face Detection

relay sample.

8cbb00b

base cmake changes.

61690e5

Corrected linking libraries in timelapse sample cmake.

5903117

Merge branch 'main' into ab/timelapse_sample

17a8fd1

joiskash assigned mraduldubey Jul 24, 2024

AdityaKBhadragond14 and others added 5 commits July 24, 2024 18:06

rename.

2b046f6

Set VPCPK PLATFORM TOOLSET to v143 to avoid issues related to v144.

6ab76c7

Merge branch 'main' of https://github.com/Apra-Labs/ApraPipes

14a4805

Merge branch 'ab/timelapse_sample' of https://github.com/AdityaKBhadr…

73f9fed

…agond14/ApraPipes into ab/timelapse_sample # Conflicts: # base/CMakeLists.txt

mraduldubey mentioned this pull request Oct 27, 2025

Aprapipes samples #428

Open

3 tasks


		timelapsePipeline -> startPipeline();

		timelapsePipeline -> stopPipeline();

Conversation

AdityaKBhadragond14 commented May 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mraduldubey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mohammedzakikochargi Jun 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AdityaKBhadragond14 commented Jun 5, 2024

Uh oh!

mraduldubey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AdityaKBhadragond14 commented May 15, 2024 •

edited

Loading

mohammedzakikochargi Jun 3, 2024 •

edited

Loading