From 221189a919842cbbf076050f1858414f3fd7ff8f Mon Sep 17 00:00:00 2001 From: Shibu Kakkoth Vayalambron Date: Wed, 1 Apr 2026 15:07:52 -0700 Subject: [PATCH 1/2] RDKB-64163: Fix for reporting hang with heavy parallel operations (#307) * Fix reporting hang due to webconfig reload and triggercondition reporting at teh same time from activation (cherry picked from commit 1d59a609d299165dfe22f5147953e8869c7a4450) * Include a summary of custom signals used by telemetry --- .github/README.md | 106 ++++++++++++++++++++++++++++++++++++++ README.md | 106 ++++++++++++++++++++++++++++++++++++++ source/bulkdata/profile.c | 16 +++++- 3 files changed, 227 insertions(+), 1 deletion(-) diff --git a/.github/README.md b/.github/README.md index 6b0e8e7a..eb0cfcae 100644 --- a/.github/README.md +++ b/.github/README.md @@ -165,6 +165,112 @@ See [Profile Configuration Guide](docs/integration/profile-configuration.md) for | `T2_XCONF_URL` | XConf server URL | - | | `T2_REPORT_URL` | Report upload URL | - | +## Runtime Operations + +### Signal Handling + +The Telemetry 2.0 daemon responds to the following signals for runtime control: + +| Signal | Value | Purpose | +|--------|-------|---------| +| **SIGTERM** | 15 | Gracefully terminate the daemon, cleanup resources and exit | +| **SIGINT** | 2 | Interrupt signal - uninitialize services, cleanup and exit | +| **SIGUSR1** | 10 | Trigger log upload with seekmap reset | +| **SIGUSR2** | 12 | Reload configuration from XConf server | +| **LOG_UPLOAD** | 10 | Custom signal to trigger log upload and reset retain seekmap flag | +| **EXEC_RELOAD** | 12 | Custom signal to reload XConf configuration and restart XConf client | +| **LOG_UPLOAD_ONDEMAND** | 29 | Custom signal for on-demand log upload without seekmap reset | +| **SIGIO** | - | I/O signal - repurposed for on-demand log upload | + +**Examples:** + +```bash +# Gracefully stop telemetry +kill -SIGTERM $(pidof telemetry2_0) + +# Trigger log upload +kill -10 $(pidof telemetry2_0) + +# Reload configuration +kill -12 $(pidof telemetry2_0) + +# On-demand log upload +kill -29 $(pidof telemetry2_0) +``` + +**Notes:** +- Custom signal values (10, 12, 29) are defined to avoid conflicts with standard system signals +- Signals SIGUSR1, SIGUSR2, LOG_UPLOAD, EXEC_RELOAD, LOG_UPLOAD_ONDEMAND, and SIGIO are blocked during signal handler execution to prevent race conditions +- Child processes ignore most signals except SIGCHLD, SIGPIPE, SIGALRM, and the log upload/reload signals + +### WebConfig/Profile Reload + +Telemetry 2.0 supports multiple mechanisms for dynamically reloading report profiles and configuration: + +#### 1. Signal-Based XConf Reload + +Trigger XConf configuration reload using signals: + +```bash +# Using custom signal value +kill -12 $(pidof telemetry2_0) +``` + +This stops the XConf client and restarts it to fetch updated configuration from the XConf server. + +#### 2. RBUS-Based Profile Updates + +For WebConfig integration, profiles can be set directly via RBUS (requires `rbuscli`): + +```bash +# Load a temporary profile (JSON format) +rbuscli setv "Device.X_RDKCENTRAL-COM_T2.Temp_ReportProfiles" string '{"profiles":[...]}' + +# Set permanent profiles +rbuscli setv "Device.X_RDKCENTRAL-COM_T2.ReportProfiles" string '{"profiles":[...]}' + +# Set profiles in MessagePack binary format +rbuscli setv "Device.X_RDKCENTRAL-COM_T2.ReportProfilesMsgPack" bytes + +# Clear all profiles +rbuscli setv "Device.X_RDKCENTRAL-COM_T2.ReportProfiles" string '{"profiles":[]}' +``` + +#### 3. DCM Event-Based Reload + +Subscribe to DCM reload events via RBUS (typically used by WebConfig framework): + +```bash +# Publish DCM reload event +rbuscli publish Device.X_RDKCENTREL-COM.Reloadconfig +``` + +#### 4. Using Test Utilities + +The project includes a convenience script for testing profile updates: + +```bash +# Load example profile +./test/set_report_profile.sh example + +# Load DOCSIS reference profile +./test/set_report_profile.sh docsis + +# Clear all profiles +./test/set_report_profile.sh empty + +# Load custom JSON profile +./test/set_report_profile.sh '{"profiles":[...]}' +``` + +**Available RBUS Parameters:** + +- `Device.X_RDKCENTRAL-COM_T2.ReportProfiles` - Persistent report profiles (JSON) +- `Device.X_RDKCENTRAL-COM_T2.ReportProfilesMsgPack` - Persistent profiles (MessagePack binary) +- `Device.X_RDKCENTRAL-COM_T2.Temp_ReportProfiles` - Temporary profiles (JSON) +- `Device.X_RDKCENTRAL-COM_T2.UploadDCMReport` - Trigger on-demand report upload +- `Device.X_RDKCENTRAL-COM_T2.AbortDCMReport` - Abort ongoing report upload + ## Development ### Running Tests diff --git a/README.md b/README.md index 6b0e8e7a..eb0cfcae 100644 --- a/README.md +++ b/README.md @@ -165,6 +165,112 @@ See [Profile Configuration Guide](docs/integration/profile-configuration.md) for | `T2_XCONF_URL` | XConf server URL | - | | `T2_REPORT_URL` | Report upload URL | - | +## Runtime Operations + +### Signal Handling + +The Telemetry 2.0 daemon responds to the following signals for runtime control: + +| Signal | Value | Purpose | +|--------|-------|---------| +| **SIGTERM** | 15 | Gracefully terminate the daemon, cleanup resources and exit | +| **SIGINT** | 2 | Interrupt signal - uninitialize services, cleanup and exit | +| **SIGUSR1** | 10 | Trigger log upload with seekmap reset | +| **SIGUSR2** | 12 | Reload configuration from XConf server | +| **LOG_UPLOAD** | 10 | Custom signal to trigger log upload and reset retain seekmap flag | +| **EXEC_RELOAD** | 12 | Custom signal to reload XConf configuration and restart XConf client | +| **LOG_UPLOAD_ONDEMAND** | 29 | Custom signal for on-demand log upload without seekmap reset | +| **SIGIO** | - | I/O signal - repurposed for on-demand log upload | + +**Examples:** + +```bash +# Gracefully stop telemetry +kill -SIGTERM $(pidof telemetry2_0) + +# Trigger log upload +kill -10 $(pidof telemetry2_0) + +# Reload configuration +kill -12 $(pidof telemetry2_0) + +# On-demand log upload +kill -29 $(pidof telemetry2_0) +``` + +**Notes:** +- Custom signal values (10, 12, 29) are defined to avoid conflicts with standard system signals +- Signals SIGUSR1, SIGUSR2, LOG_UPLOAD, EXEC_RELOAD, LOG_UPLOAD_ONDEMAND, and SIGIO are blocked during signal handler execution to prevent race conditions +- Child processes ignore most signals except SIGCHLD, SIGPIPE, SIGALRM, and the log upload/reload signals + +### WebConfig/Profile Reload + +Telemetry 2.0 supports multiple mechanisms for dynamically reloading report profiles and configuration: + +#### 1. Signal-Based XConf Reload + +Trigger XConf configuration reload using signals: + +```bash +# Using custom signal value +kill -12 $(pidof telemetry2_0) +``` + +This stops the XConf client and restarts it to fetch updated configuration from the XConf server. + +#### 2. RBUS-Based Profile Updates + +For WebConfig integration, profiles can be set directly via RBUS (requires `rbuscli`): + +```bash +# Load a temporary profile (JSON format) +rbuscli setv "Device.X_RDKCENTRAL-COM_T2.Temp_ReportProfiles" string '{"profiles":[...]}' + +# Set permanent profiles +rbuscli setv "Device.X_RDKCENTRAL-COM_T2.ReportProfiles" string '{"profiles":[...]}' + +# Set profiles in MessagePack binary format +rbuscli setv "Device.X_RDKCENTRAL-COM_T2.ReportProfilesMsgPack" bytes + +# Clear all profiles +rbuscli setv "Device.X_RDKCENTRAL-COM_T2.ReportProfiles" string '{"profiles":[]}' +``` + +#### 3. DCM Event-Based Reload + +Subscribe to DCM reload events via RBUS (typically used by WebConfig framework): + +```bash +# Publish DCM reload event +rbuscli publish Device.X_RDKCENTREL-COM.Reloadconfig +``` + +#### 4. Using Test Utilities + +The project includes a convenience script for testing profile updates: + +```bash +# Load example profile +./test/set_report_profile.sh example + +# Load DOCSIS reference profile +./test/set_report_profile.sh docsis + +# Clear all profiles +./test/set_report_profile.sh empty + +# Load custom JSON profile +./test/set_report_profile.sh '{"profiles":[...]}' +``` + +**Available RBUS Parameters:** + +- `Device.X_RDKCENTRAL-COM_T2.ReportProfiles` - Persistent report profiles (JSON) +- `Device.X_RDKCENTRAL-COM_T2.ReportProfilesMsgPack` - Persistent profiles (MessagePack binary) +- `Device.X_RDKCENTRAL-COM_T2.Temp_ReportProfiles` - Temporary profiles (JSON) +- `Device.X_RDKCENTRAL-COM_T2.UploadDCMReport` - Trigger on-demand report upload +- `Device.X_RDKCENTRAL-COM_T2.AbortDCMReport` - Abort ongoing report upload + ## Development ### Running Tests diff --git a/source/bulkdata/profile.c b/source/bulkdata/profile.c index fe7d91fd..298abeab 100644 --- a/source/bulkdata/profile.c +++ b/source/bulkdata/profile.c @@ -1184,7 +1184,7 @@ T2ERROR deleteAllProfiles(bool delFromDisk) T2Error("Profile : %s failed to unregister from scheduler\n", tempProfile->name); } - pthread_mutex_lock(&plMutex); + /* Release plMutex before pthread_join to avoid deadlock */ if (tempProfile->threadExists) { pthread_mutex_lock(&tempProfile->reuseThreadMutex); @@ -1193,6 +1193,9 @@ T2ERROR deleteAllProfiles(bool delFromDisk) pthread_join(tempProfile->reportThread, NULL); tempProfile->threadExists = false; } + + /* Re-acquire plMutex for profile cleanup */ + pthread_mutex_lock(&plMutex); if(tempProfile->grepSeekProfile) { freeGrepSeekProfile(tempProfile->grepSeekProfile); @@ -1284,6 +1287,14 @@ T2ERROR deleteProfile(const char *profileName) } pthread_mutex_unlock(&profile->reportInProgressMutex); + /* Release plMutex before pthread_join to avoid deadlock. + * pthread_join can block indefinitely if the CollectAndReport thread + * is stuck (e.g., waiting on rbusMethodMutex). Holding plMutex during + * pthread_join prevents other threads (timeout callbacks, other profile + * operations) from making progress, creating a deadlock. + */ + pthread_mutex_unlock(&plMutex); + if (profile->threadExists) { pthread_mutex_lock(&profile->reuseThreadMutex); @@ -1293,6 +1304,9 @@ T2ERROR deleteProfile(const char *profileName) profile->threadExists = false; } + /* Re-acquire plMutex for profile cleanup operations */ + pthread_mutex_lock(&plMutex); + if(Vector_Size(profile->triggerConditionList) > 0) { rbusT2ConsumerUnReg(profile->triggerConditionList); From d876cce0e4f7ded4831dd8a7542cf79e0035dc4d Mon Sep 17 00:00:00 2001 From: shibu-kv Date: Wed, 1 Apr 2026 15:11:16 -0700 Subject: [PATCH 2/2] Changelog updates for 1.8.6 release --- CHANGELOG.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bf748f13..76133fa3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,9 +4,18 @@ All notable changes to this project will be documented in this file. Dates are d Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog). +#### [1.8.6](https://github.com/rdkcentral/telemetry/compare/1.8.5...1.8.6) + +> 1 April 2026 + +- RDKB-64163: Fix for reporting hang with heavy parallel operations [`#307`](https://github.com/rdkcentral/telemetry/pull/307) + #### [1.8.5](https://github.com/rdkcentral/telemetry/compare/1.8.4...1.8.5) -- RDKEMW-15233: Previous log look up initialiation corrections [`#299`](https://github.com/rdkcentral/telemetry/pull/299) +> 24 March 2026 + +- RDKEMW-15233:[SERXIONE-8445/XIONE-18418] Develop Support Branch Integ… [`#299`](https://github.com/rdkcentral/telemetry/pull/299) +- Changelog updates for release 1.8.5 [`3d23970`](https://github.com/rdkcentral/telemetry/commit/3d2397047795a728ab1a22ce439c4743cf8a84e1) #### [1.8.4](https://github.com/rdkcentral/telemetry/compare/1.8.3...1.8.4)