Add integration tests by joaoantoniocardoso · Pull Request #422 · mavlink/mavlink-camera-manager

joaoantoniocardoso · 2024-07-01T23:45:25Z

First step towards #378

How to test locally

1. Install v4l2loopback:

Ubuntu: sudo apt-get install -y v4l2loopback-dkms
Arch: sudo pacman -S v4l2loopback-dkms v4l2loopback-utils

2. Add a v4l2loopback device:

sudo modprobe v4l2loopback video_nr=42 && \
sudo v4l2loopback-ctl delete /dev/video42 && \
sudo v4l2loopback-ctl add --exclusive-caps 0 --max-openers 2 --name potato --verbose /dev/video42

3. Get qrtimestamp plugins:

wget -q https://github.com/patrickelectric/qrtimestamp-gst/releases/download/0.1.0/libgstqrtimestamp-x86_64.so.tar -O - | tar -x
export GST_PLUGIN_PATH="$PWD:$GST_PLUGIN_PATH"

4. Run the test

cargo test --release --locked --test v4l2_latency_and_jitter

5. Clean up

sudo modprobe -r v4l2loopback

Test output

Results for ~100 frames at ~30 FPS, in milliseconds:

	Baseline	Run	Added latency
UDP H264	25.99	34.80	8.81
UDP MJPEG	26.71	34.94	8.23
UDP YUYV	26.99	36.61	9.62
RTSP H264	27.27	35.00	7.73
RTSP MJPEG	26.66	35.18	8.53
RTSP YUYV	26.90	37.52	10.62

Details

2024-12-12T21:34:42.214270Z  INFO src/lib/logger/manager.rs:193: mavlink-camera-manager, version: 0.2.4-fd5634e, build date: 2024-12-12
2024-12-12T21:34:42.215378Z  INFO src/lib/logger/manager.rs:200: Starting at 2024-12-12T21:34:42
2024-12-12T21:34:42.215586Z  INFO src/lib/logger/manager.rs:204: Server running at 0.0.0.0:6020
2024-12-12T21:34:42.231049Z  INFO tests/v4l2_latency_and_jitter.rs:632: Testing for: VideoAndStreamInformation {
    name: "QRTimeStamp - QR",
    stream_information: StreamInformation {
        endpoints: [
            Url {
                scheme: "udp",
                cannot_be_a_base: false,
                username: "",
                password: None,
                host: Some(
                    Domain(
                        "127.0.0.1",
                    ),
                ),
                port: Some(
                    5600,
                ),
                path: "",
                query: None,
                fragment: None,
            },
        ],
        configuration: Video(
            VideoCaptureConfiguration {
                encode: H264,
                height: 320,
                width: 320,
                frame_interval: FrameInterval {
                    numerator: 1,
                    denominator: 30,
                },
            },
        ),
        extended_configuration: Some(
            ExtendedConfiguration {
                thermal: false,
                disable_mavlink: true,
            },
        ),
    },
    video_source: Local(
        VideoSourceLocal {
            name: "potato",
            device_path: "/dev/video42",
            typ: V4L2Loopback(
                "platform:v4l2loopback-042",
            ),
        },
    ),
}
2024-12-12T21:34:42.231486Z  INFO tests/v4l2_latency_and_jitter.rs:634: Building v4lloopback pipeline (video generation with qrtimestampsrc)...
2024-12-12T21:34:42.329457Z  INFO tests/v4l2_latency_and_jitter.rs:638: Building MCM stream...
2024-12-12T21:34:43.239030Z  INFO src/lib/stream/manager.rs:553: Stream 208c433c-fb32-4dab-a23d-f4e85c5c0610 successfully added!
2024-12-12T21:34:43.239163Z  INFO tests/v4l2_latency_and_jitter.rs:645: Getting baseline latency...
2024-12-12T21:34:46.566817Z  INFO tests/v4l2_latency_and_jitter.rs:650: Baseline latency: LatenciesStats {
    latency_min: 5.0,
    latency_max: 30.0,
    latency_mean: 25.989898989898983,
    latency_std_dev: 3.3911497944603344,
    latency_variance: 11.499896928468367,
    jitter_min: -4.0,
    jitter_max: 11.0,
    jitter_mean: 0.21428571428571463,
    jitter_std_dev: 1.9117110595635871,
    jitter_variance: 3.654639175257733,
}
2024-12-12T21:34:46.567015Z  INFO tests/v4l2_latency_and_jitter.rs:652: Building qrtimestamp pipeline (video receiver with qrtimestampsink)...
2024-12-12T21:34:46.574273Z  INFO tests/v4l2_latency_and_jitter.rs:660: Waiting for QrTimestampSink Pipeline to finish...
2024-12-12T21:34:48.138566Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "set state timed-out (5 seconds)"
2024-12-12T21:34:48.140233Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "set state timed-out (5 seconds)"
2024-12-12T21:35:05.627754Z  INFO tests/v4l2_latency_and_jitter.rs:663: Finishing loopback pipeline...
2024-12-12T21:35:05.629443Z  INFO tests/v4l2_latency_and_jitter.rs:666: Finishing MCM stream...
2024-12-12T21:35:05.630675Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:05.736154Z  INFO src/lib/stream/manager.rs:572: Stream 208c433c-fb32-4dab-a23d-f4e85c5c0610 successfully removed!
2024-12-12T21:35:05.736869Z  INFO tests/v4l2_latency_and_jitter.rs:671: Results appended...
2024-12-12T21:35:05.737021Z  INFO tests/v4l2_latency_and_jitter.rs:673: Evaluating results...
2024-12-12T21:35:05.737094Z  INFO tests/v4l2_latency_and_jitter.rs:676: Pipeline latency: LatenciesStats {
    latency_min: 34.0,
    latency_max: 36.0,
    latency_mean: 34.7979797979798,
    latency_std_dev: 0.4512974036336674,
    latency_variance: 0.2036693465264893,
    jitter_min: -1.0,
    jitter_max: 2.0,
    jitter_mean: 2.0816681711721685e-17,
    jitter_std_dev: 0.717958158617738,
    jitter_variance: 0.5154639175257731,
}
2024-12-12T21:35:05.737197Z  INFO tests/v4l2_latency_and_jitter.rs:681: Asserting 8.808080808080817 <= 21...
2024-12-12T21:35:05.737321Z  INFO tests/v4l2_latency_and_jitter.rs:685: Asserting 0.000000000000000020816681711721685 <= 11...
2024-12-12T21:35:05.750789Z  INFO tests/v4l2_latency_and_jitter.rs:632: Testing for: VideoAndStreamInformation {
    name: "QRTimeStamp - QR",
    stream_information: StreamInformation {
        endpoints: [
            Url {
                scheme: "udp",
                cannot_be_a_base: false,
                username: "",
                password: None,
                host: Some(
                    Domain(
                        "127.0.0.1",
                    ),
                ),
                port: Some(
                    5600,
                ),
                path: "",
                query: None,
                fragment: None,
            },
        ],
        configuration: Video(
            VideoCaptureConfiguration {
                encode: Mjpg,
                height: 320,
                width: 320,
                frame_interval: FrameInterval {
                    numerator: 1,
                    denominator: 30,
                },
            },
        ),
        extended_configuration: Some(
            ExtendedConfiguration {
                thermal: false,
                disable_mavlink: true,
            },
        ),
    },
    video_source: Local(
        VideoSourceLocal {
            name: "potato",
            device_path: "/dev/video42",
            typ: V4L2Loopback(
                "platform:v4l2loopback-042",
            ),
        },
    ),
}
2024-12-12T21:35:05.751364Z  INFO tests/v4l2_latency_and_jitter.rs:634: Building v4lloopback pipeline (video generation with qrtimestampsrc)...
2024-12-12T21:35:07.995063Z  INFO tests/v4l2_latency_and_jitter.rs:638: Building MCM stream...
2024-12-12T21:35:08.204992Z  INFO src/lib/stream/manager.rs:553: Stream 59ed3d69-663c-4fdb-9f13-ed94db0b8504 successfully added!
2024-12-12T21:35:08.205221Z  INFO tests/v4l2_latency_and_jitter.rs:645: Getting baseline latency...
2024-12-12T21:35:11.522232Z  INFO tests/v4l2_latency_and_jitter.rs:650: Baseline latency: LatenciesStats {
    latency_min: 24.0,
    latency_max: 29.0,
    latency_mean: 26.70707070707071,
    latency_std_dev: 0.610328202983983,
    latency_variance: 0.37250051535765805,
    jitter_min: -5.0,
    jitter_max: 3.0,
    jitter_mean: 0.010204081632653078,
    jitter_std_dev: 0.8909036066106302,
    jitter_variance: 0.7937092362718285,
}
2024-12-12T21:35:11.522365Z  INFO tests/v4l2_latency_and_jitter.rs:652: Building qrtimestamp pipeline (video receiver with qrtimestampsink)...
2024-12-12T21:35:11.524738Z  INFO tests/v4l2_latency_and_jitter.rs:660: Waiting for QrTimestampSink Pipeline to finish...
2024-12-12T21:35:14.835531Z  INFO tests/v4l2_latency_and_jitter.rs:663: Finishing loopback pipeline...
2024-12-12T21:35:14.836886Z  INFO tests/v4l2_latency_and_jitter.rs:666: Finishing MCM stream...
2024-12-12T21:35:14.838058Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:14.939475Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:14.940660Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:14.977206Z  INFO src/lib/stream/manager.rs:572: Stream 59ed3d69-663c-4fdb-9f13-ed94db0b8504 successfully removed!
2024-12-12T21:35:14.977357Z  INFO tests/v4l2_latency_and_jitter.rs:671: Results appended...
2024-12-12T21:35:14.977423Z  INFO tests/v4l2_latency_and_jitter.rs:673: Evaluating results...
2024-12-12T21:35:14.977477Z  INFO tests/v4l2_latency_and_jitter.rs:676: Pipeline latency: LatenciesStats {
    latency_min: 34.0,
    latency_max: 37.0,
    latency_mean: 34.93939393939396,
    latency_std_dev: 0.5499198794737696,
    latency_variance: 0.30241187384044527,
    jitter_min: -1.0,
    jitter_max: 2.0,
    jitter_mean: 3.469446951953614e-18,
    jitter_std_dev: 0.7321765321469982,
    jitter_variance: 0.5360824742268042,
}
2024-12-12T21:35:14.977588Z  INFO tests/v4l2_latency_and_jitter.rs:681: Asserting 8.23232323232325 <= 13...
2024-12-12T21:35:14.977643Z  INFO tests/v4l2_latency_and_jitter.rs:685: Asserting 0.000000000000000003469446951953614 <= 3...
2024-12-12T21:35:14.992097Z  INFO tests/v4l2_latency_and_jitter.rs:632: Testing for: VideoAndStreamInformation {
    name: "QRTimeStamp - QR",
    stream_information: StreamInformation {
        endpoints: [
            Url {
                scheme: "udp",
                cannot_be_a_base: false,
                username: "",
                password: None,
                host: Some(
                    Domain(
                        "127.0.0.1",
                    ),
                ),
                port: Some(
                    5600,
                ),
                path: "",
                query: None,
                fragment: None,
            },
        ],
        configuration: Video(
            VideoCaptureConfiguration {
                encode: Yuyv,
                height: 320,
                width: 320,
                frame_interval: FrameInterval {
                    numerator: 1,
                    denominator: 30,
                },
            },
        ),
        extended_configuration: Some(
            ExtendedConfiguration {
                thermal: false,
                disable_mavlink: true,
            },
        ),
    },
    video_source: Local(
        VideoSourceLocal {
            name: "potato",
            device_path: "/dev/video42",
            typ: V4L2Loopback(
                "platform:v4l2loopback-042",
            ),
        },
    ),
}
2024-12-12T21:35:14.992450Z  INFO tests/v4l2_latency_and_jitter.rs:634: Building v4lloopback pipeline (video generation with qrtimestampsrc)...
2024-12-12T21:35:15.004949Z  INFO tests/v4l2_latency_and_jitter.rs:638: Building MCM stream...
2024-12-12T21:35:15.219595Z  INFO src/lib/stream/manager.rs:553: Stream 450619fe-a712-4199-951f-d2dc657fa13a successfully added!
2024-12-12T21:35:15.219750Z  INFO tests/v4l2_latency_and_jitter.rs:645: Getting baseline latency...
2024-12-12T21:35:18.535712Z  INFO tests/v4l2_latency_and_jitter.rs:650: Baseline latency: LatenciesStats {
    latency_min: 25.0,
    latency_max: 28.0,
    latency_mean: 26.98989898989899,
    latency_std_dev: 0.5049742263568345,
    latency_variance: 0.2549989692846836,
    jitter_min: -2.0,
    jitter_max: 2.0,
    jitter_mean: 0.020408163265306124,
    jitter_std_dev: 0.6731919938258556,
    jitter_variance: 0.4531874605512308,
}
2024-12-12T21:35:18.535832Z  INFO tests/v4l2_latency_and_jitter.rs:652: Building qrtimestamp pipeline (video receiver with qrtimestampsink)...
2024-12-12T21:35:18.537830Z  INFO tests/v4l2_latency_and_jitter.rs:660: Waiting for QrTimestampSink Pipeline to finish...
2024-12-12T21:35:25.447452Z  INFO tests/v4l2_latency_and_jitter.rs:663: Finishing loopback pipeline...
2024-12-12T21:35:25.448738Z  INFO tests/v4l2_latency_and_jitter.rs:666: Finishing MCM stream...
2024-12-12T21:35:25.449915Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:25.550753Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:25.552025Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:25.586767Z  INFO src/lib/stream/manager.rs:572: Stream 450619fe-a712-4199-951f-d2dc657fa13a successfully removed!
2024-12-12T21:35:25.586869Z  INFO tests/v4l2_latency_and_jitter.rs:671: Results appended...
2024-12-12T21:35:25.586919Z  INFO tests/v4l2_latency_and_jitter.rs:673: Evaluating results...
2024-12-12T21:35:25.586962Z  INFO tests/v4l2_latency_and_jitter.rs:676: Pipeline latency: LatenciesStats {
    latency_min: 35.0,
    latency_max: 68.0,
    latency_mean: 36.6060606060606,
    latency_std_dev: 5.499198794737694,
    latency_variance: 30.24118738404451,
    jitter_min: -29.0,
    jitter_max: 30.0,
    jitter_mean: 7.554897750478585e-17,
    jitter_std_dev: 4.333994134126069,
    jitter_variance: 18.783505154639172,
}
2024-12-12T21:35:25.587033Z  INFO tests/v4l2_latency_and_jitter.rs:681: Asserting 9.616161616161612 <= 32...
2024-12-12T21:35:25.587075Z  INFO tests/v4l2_latency_and_jitter.rs:685: Asserting 0.00000000000000007554897750478585 <= 2...
2024-12-12T21:35:25.597295Z  INFO tests/v4l2_latency_and_jitter.rs:632: Testing for: VideoAndStreamInformation {
    name: "QRTimeStamp - QR",
    stream_information: StreamInformation {
        endpoints: [
            Url {
                scheme: "rtsp",
                cannot_be_a_base: false,
                username: "",
                password: None,
                host: Some(
                    Domain(
                        "127.0.0.1",
                    ),
                ),
                port: Some(
                    8554,
                ),
                path: "/test",
                query: None,
                fragment: None,
            },
        ],
        configuration: Video(
            VideoCaptureConfiguration {
                encode: H264,
                height: 320,
                width: 320,
                frame_interval: FrameInterval {
                    numerator: 1,
                    denominator: 30,
                },
            },
        ),
        extended_configuration: Some(
            ExtendedConfiguration {
                thermal: false,
                disable_mavlink: true,
            },
        ),
    },
    video_source: Local(
        VideoSourceLocal {
            name: "potato",
            device_path: "/dev/video42",
            typ: V4L2Loopback(
                "platform:v4l2loopback-042",
            ),
        },
    ),
}
2024-12-12T21:35:25.597595Z  INFO tests/v4l2_latency_and_jitter.rs:634: Building v4lloopback pipeline (video generation with qrtimestampsrc)...
2024-12-12T21:35:25.611000Z  INFO tests/v4l2_latency_and_jitter.rs:638: Building MCM stream...
2024-12-12T21:35:25.826640Z  INFO src/lib/stream/manager.rs:553: Stream c0768f78-2e40-4350-ab3b-46ec8a9d2950 successfully added!
2024-12-12T21:35:25.826761Z  INFO tests/v4l2_latency_and_jitter.rs:645: Getting baseline latency...
2024-12-12T21:35:29.153783Z  INFO tests/v4l2_latency_and_jitter.rs:650: Baseline latency: LatenciesStats {
    latency_min: 25.0,
    latency_max: 30.0,
    latency_mean: 27.272727272727273,
    latency_std_dev: 1.3983823147945773,
    latency_variance: 1.9554730983302404,
    jitter_min: -4.0,
    jitter_max: 3.0,
    jitter_mean: -2.5773034500226846e-17,
    jitter_std_dev: 1.5860214114335829,
    jitter_variance: 2.5154639175257745,
}
2024-12-12T21:35:29.153956Z  INFO tests/v4l2_latency_and_jitter.rs:652: Building qrtimestamp pipeline (video receiver with qrtimestampsink)...
2024-12-12T21:35:29.172989Z  INFO tests/v4l2_latency_and_jitter.rs:660: Waiting for QrTimestampSink Pipeline to finish...
2024-12-12T21:35:38.921449Z  INFO tests/v4l2_latency_and_jitter.rs:663: Finishing loopback pipeline...
2024-12-12T21:35:38.923241Z  INFO tests/v4l2_latency_and_jitter.rs:666: Finishing MCM stream...
2024-12-12T21:35:38.924244Z  INFO src/lib/stream/manager.rs:572: Stream c0768f78-2e40-4350-ab3b-46ec8a9d2950 successfully removed!
2024-12-12T21:35:38.924372Z  INFO tests/v4l2_latency_and_jitter.rs:671: Results appended...
2024-12-12T21:35:38.924427Z  INFO tests/v4l2_latency_and_jitter.rs:673: Evaluating results...
2024-12-12T21:35:38.924418Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:38.924475Z  INFO tests/v4l2_latency_and_jitter.rs:676: Pipeline latency: LatenciesStats {
    latency_min: 34.0,
    latency_max: 36.0,
    latency_mean: 35.0,
    latency_std_dev: 0.4285714285714286,
    latency_variance: 0.1836734693877551,
    jitter_min: -2.0,
    jitter_max: 1.0,
    jitter_mean: -0.010204081632653062,
    jitter_std_dev: 0.6175257907579325,
    jitter_variance: 0.38133810225120984,
}
2024-12-12T21:35:38.924556Z  INFO tests/v4l2_latency_and_jitter.rs:681: Asserting 7.727272727272727 <= 13...
2024-12-12T21:35:38.924606Z  INFO tests/v4l2_latency_and_jitter.rs:685: Asserting -0.010204081632653062 <= 3...
2024-12-12T21:35:38.937615Z  INFO tests/v4l2_latency_and_jitter.rs:632: Testing for: VideoAndStreamInformation {
    name: "QRTimeStamp - QR",
    stream_information: StreamInformation {
        endpoints: [
            Url {
                scheme: "rtsp",
                cannot_be_a_base: false,
                username: "",
                password: None,
                host: Some(
                    Domain(
                        "127.0.0.1",
                    ),
                ),
                port: Some(
                    8554,
                ),
                path: "/test",
                query: None,
                fragment: None,
            },
        ],
        configuration: Video(
            VideoCaptureConfiguration {
                encode: Mjpg,
                height: 320,
                width: 320,
                frame_interval: FrameInterval {
                    numerator: 1,
                    denominator: 30,
                },
            },
        ),
        extended_configuration: Some(
            ExtendedConfiguration {
                thermal: false,
                disable_mavlink: true,
            },
        ),
    },
    video_source: Local(
        VideoSourceLocal {
            name: "potato",
            device_path: "/dev/video42",
            typ: V4L2Loopback(
                "platform:v4l2loopback-042",
            ),
        },
    ),
}
2024-12-12T21:35:38.937950Z  INFO tests/v4l2_latency_and_jitter.rs:634: Building v4lloopback pipeline (video generation with qrtimestampsrc)...
2024-12-12T21:35:38.946408Z  INFO tests/v4l2_latency_and_jitter.rs:638: Building MCM stream...
2024-12-12T21:35:39.026659Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:39.154905Z  INFO src/lib/stream/manager.rs:553: Stream a35494e6-4586-4dbe-97c1-ccabf9a1dd4f successfully added!
2024-12-12T21:35:39.155047Z  INFO tests/v4l2_latency_and_jitter.rs:645: Getting baseline latency...
test main has been running for over 60 seconds
2024-12-12T21:35:42.472290Z  INFO tests/v4l2_latency_and_jitter.rs:650: Baseline latency: LatenciesStats {
    latency_min: 25.0,
    latency_max: 28.0,
    latency_mean: 26.65656565656566,
    latency_std_dev: 0.5182705967557434,
    latency_variance: 0.2686044114615544,
    jitter_min: -1.0,
    jitter_max: 2.0,
    jitter_mean: 0.010204081632653078,
    jitter_std_dev: 0.739112004948815,
    jitter_variance: 0.5462865558594572,
}
2024-12-12T21:35:42.472413Z  INFO tests/v4l2_latency_and_jitter.rs:652: Building qrtimestamp pipeline (video receiver with qrtimestampsink)...
2024-12-12T21:35:42.474585Z  INFO tests/v4l2_latency_and_jitter.rs:660: Waiting for QrTimestampSink Pipeline to finish...
2024-12-12T21:35:45.822244Z  INFO tests/v4l2_latency_and_jitter.rs:663: Finishing loopback pipeline...
2024-12-12T21:35:45.824481Z  INFO tests/v4l2_latency_and_jitter.rs:666: Finishing MCM stream...
2024-12-12T21:35:45.825657Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:45.927880Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:45.930938Z  INFO src/lib/stream/manager.rs:572: Stream a35494e6-4586-4dbe-97c1-ccabf9a1dd4f successfully removed!
2024-12-12T21:35:45.931147Z  INFO tests/v4l2_latency_and_jitter.rs:671: Results appended...
2024-12-12T21:35:45.931241Z  INFO tests/v4l2_latency_and_jitter.rs:673: Evaluating results...
2024-12-12T21:35:45.931318Z  INFO tests/v4l2_latency_and_jitter.rs:676: Pipeline latency: LatenciesStats {
    latency_min: 35.0,
    latency_max: 36.0,
    latency_mean: 35.18181818181818,
    latency_std_dev: 0.38765744302855326,
    latency_variance: 0.15027829313543603,
    jitter_min: -1.0,
    jitter_max: 1.0,
    jitter_mean: -8.673617379884035e-18,
    jitter_std_dev: 0.5920434645509787,
    jitter_variance: 0.3505154639175259,
}
2024-12-12T21:35:45.931434Z  INFO tests/v4l2_latency_and_jitter.rs:681: Asserting 8.525252525252519 <= 12...
2024-12-12T21:35:45.931513Z  INFO tests/v4l2_latency_and_jitter.rs:685: Asserting -0.000000000000000008673617379884035 <= 2...
2024-12-12T21:35:45.945611Z  INFO tests/v4l2_latency_and_jitter.rs:632: Testing for: VideoAndStreamInformation {
    name: "QRTimeStamp - QR",
    stream_information: StreamInformation {
        endpoints: [
            Url {
                scheme: "rtsp",
                cannot_be_a_base: false,
                username: "",
                password: None,
                host: Some(
                    Domain(
                        "127.0.0.1",
                    ),
                ),
                port: Some(
                    8554,
                ),
                path: "/test",
                query: None,
                fragment: None,
            },
        ],
        configuration: Video(
            VideoCaptureConfiguration {
                encode: Yuyv,
                height: 320,
                width: 320,
                frame_interval: FrameInterval {
                    numerator: 1,
                    denominator: 30,
                },
            },
        ),
        extended_configuration: Some(
            ExtendedConfiguration {
                thermal: false,
                disable_mavlink: true,
            },
        ),
    },
    video_source: Local(
        VideoSourceLocal {
            name: "potato",
            device_path: "/dev/video42",
            typ: V4L2Loopback(
                "platform:v4l2loopback-042",
            ),
        },
    ),
}
2024-12-12T21:35:45.946026Z  INFO tests/v4l2_latency_and_jitter.rs:634: Building v4lloopback pipeline (video generation with qrtimestampsrc)...
2024-12-12T21:35:45.954693Z  INFO tests/v4l2_latency_and_jitter.rs:638: Building MCM stream...
2024-12-12T21:35:46.168292Z  INFO src/lib/stream/manager.rs:553: Stream 4b7fd412-017c-4ac9-86de-8c9ebcd12860 successfully added!
2024-12-12T21:35:46.168431Z  INFO tests/v4l2_latency_and_jitter.rs:645: Getting baseline latency...
2024-12-12T21:35:49.484139Z  INFO tests/v4l2_latency_and_jitter.rs:650: Baseline latency: LatenciesStats {
    latency_min: 26.0,
    latency_max: 28.0,
    latency_mean: 26.898989898989896,
    latency_std_dev: 0.41637280917881647,
    latency_variance: 0.1733663162234591,
    jitter_min: -1.0,
    jitter_max: 2.0,
    jitter_mean: -6.372453585220925e-18,
    jitter_std_dev: 0.6092076990801715,
    jitter_variance: 0.3711340206185567,
}
2024-12-12T21:35:49.484274Z  INFO tests/v4l2_latency_and_jitter.rs:652: Building qrtimestamp pipeline (video receiver with qrtimestampsink)...
2024-12-12T21:35:49.486078Z  INFO tests/v4l2_latency_and_jitter.rs:660: Waiting for QrTimestampSink Pipeline to finish...
2024-12-12T21:35:52.833175Z  INFO tests/v4l2_latency_and_jitter.rs:663: Finishing loopback pipeline...
2024-12-12T21:35:52.834516Z  INFO tests/v4l2_latency_and_jitter.rs:666: Finishing MCM stream...
2024-12-12T21:35:52.835569Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:52.937186Z  WARN src/lib/stream/pipeline/runner.rs:59: task ended with error: "Some(\"Received EndOfStream: Eos { structure: None, source: None }\")"
2024-12-12T21:35:53.015688Z  INFO src/lib/stream/manager.rs:572: Stream 4b7fd412-017c-4ac9-86de-8c9ebcd12860 successfully removed!
2024-12-12T21:35:53.015834Z  INFO tests/v4l2_latency_and_jitter.rs:671: Results appended...
2024-12-12T21:35:53.015901Z  INFO tests/v4l2_latency_and_jitter.rs:673: Evaluating results...
2024-12-12T21:35:53.015954Z  INFO tests/v4l2_latency_and_jitter.rs:676: Pipeline latency: LatenciesStats {
    latency_min: 37.0,
    latency_max: 39.0,
    latency_mean: 37.51515151515151,
    latency_std_dev: 0.5952813837073163,
    latency_variance: 0.3543599257884971,
    jitter_min: -2.0,
    jitter_max: 1.0,
    jitter_mean: -0.020408163265306117,
    jitter_std_dev: 0.7992211651730134,
    jitter_variance: 0.6387544708605092,
}
2024-12-12T21:35:53.016040Z  INFO tests/v4l2_latency_and_jitter.rs:681: Asserting 10.616161616161612 <= 32...
2024-12-12T21:35:53.016090Z  INFO tests/v4l2_latency_and_jitter.rs:685: Asserting -0.020408163265306117 <= 2...
test main ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 71.32s

…aps for RAW

joaoantoniocardoso requested a review from patrickelectric July 1, 2024 23:45

joaoantoniocardoso force-pushed the add_integration_tests branch 5 times, most recently from 76d8a58 to 1677dd8 Compare July 18, 2024 20:36

joaoantoniocardoso force-pushed the add_integration_tests branch 2 times, most recently from 21006ab to 4eda092 Compare July 22, 2024 19:20

joaoantoniocardoso marked this pull request as ready for review July 23, 2024 11:46

joaoantoniocardoso force-pushed the add_integration_tests branch from 4eda092 to 5d4d90c Compare August 3, 2024 02:52

joaoantoniocardoso force-pushed the add_integration_tests branch 20 times, most recently from 07c8b16 to 8b7b654 Compare October 1, 2024 19:22

joaoantoniocardoso added 4 commits December 13, 2024 21:17

src: lib: stream: pipeline: Remove unneeded caps for fake pipeline

8b09315

src: lib: stream: pipeline: Fix JPEG RTP payload number: RFC 2435

6d2c5bd

src: lib: stream: pipeline: Be more specific with the video and RTP c…

c847490

…aps for RAW

src: lib: stream: rtsp: Fix max-buffer-size for RTSP server

f37622d

joaoantoniocardoso force-pushed the add_integration_tests branch 19 times, most recently from be2ddb4 to e95b166 Compare December 14, 2024 02:58

joaoantoniocardoso added 5 commits December 14, 2024 00:54

build.rs: Update comment

b5a9b8f

.github: workflows: Fix deploy for linux

3b39c89

.github: Add latency and jitter v4l2loopback CI tests

b594ae1

src: lib: stream: set H264 config-interval to every IDR frame

4a8a6b8

src: lib: stream: sink: rtsp_sink: Better deal with socket file

c5ad783

joaoantoniocardoso force-pushed the add_integration_tests branch from e95b166 to c5ad783 Compare December 14, 2024 03:54

joaoantoniocardoso marked this pull request as draft March 29, 2025 22:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add integration tests#422

Add integration tests#422
joaoantoniocardoso wants to merge 20 commits into
mavlink:masterfrom
joaoantoniocardoso:add_integration_tests

joaoantoniocardoso commented Jul 1, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

joaoantoniocardoso commented Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to test locally

1. Install v4l2loopback:

2. Add a v4l2loopback device:

3. Get qrtimestamp plugins:

4. Run the test

5. Clean up

Test output

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

joaoantoniocardoso commented Jul 1, 2024 •

edited

Loading