Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/en/14-reference/01-components/01-taosd.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ The effective value of charset is UTF-8.

| Parameter Name | Supported Version | Dynamic Modification | Description |
| -------------------------- | ----------------- | ---------------------------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| supportVnodes | | Supported, effective immediately | Maximum number of vnodes supported by a dnode, range 0-4096, default value is twice the number of CPU cores + 5 |
| supportVnodes | | Supported, effective immediately | Maximum number of vnodes supported by a dnode, range 0-1024, default value is twice the number of CPU cores + 5 |
| numOfCommitThreads | | Supported, effective after restart | Maximum number of commit threads, range 1-1024, default value 4 |
| numOfCompactThreads | | Supported, effective after restart | Maximum number of commit threads, range 1-16, default value 2 |
| numOfMnodeReadThreads | | Supported, effective after restart | Number of Read threads for mnode, range 0-1024, default value is one quarter of the CPU cores (not exceeding 4) |
Expand Down
2 changes: 1 addition & 1 deletion docs/en/14-reference/01-components/02-taosc.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ The following configuration parameters only take effect for Native connections.
| smlTsDefaultName | |Supported, effective immediately | Configuration for setting the time column name in schemaless auto table creation, default value "_ts" |
| smlDot2Underline | |Supported, effective immediately | Converts dots in supertable names to underscores in schemaless |
| maxInsertBatchRows | |Supported, effective immediately | Internal parameter, maximum number of rows per batch insert |
| maxSQLLength | v3.3.6.34 |Supported, effective immediately | Maximum length of a single SQL statement; default value: 4,194,304; minimum value: 1,048,576; maximum value: 67,108,864 |
| maxSQLLength |v3.3.6.34, v3.3.8.8|Supported, effective immediately | Maximum length of a single SQL statement; default value: 1,048,576; minimum value: 1,048,576; maximum value: 67,108,864 |

### Region Related

Expand Down
8 changes: 4 additions & 4 deletions docs/en/14-reference/01-components/06-taoskeeper.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,11 @@ Usage of taoskeeper:
-H, --host string http host. Env "TAOS_KEEPER_HOST"
--instanceId int instance ID. Env "TAOS_KEEPER_INSTANCE_ID" (default 64)
--log.compress whether to compress old log. Env "TAOS_KEEPER_LOG_COMPRESS"
--log.keepDays uint log retention days, must be a positive integer. Env "TAOS_KEEPER_LOG_KEEP_DAYS" (default 30)
--log.keepDays uint log retention days, must be a positive integer. Env "TAOS_KEEPER_LOG_KEEP_DAYS" (default 3)
--log.level string log level (trace debug info warning error). Env "TAOS_KEEPER_LOG_LEVEL" (default "info")
--log.path string log path. Env "TAOS_KEEPER_LOG_PATH" (default "/var/log/taos")
--log.reservedDiskSize string reserved disk size for log dir (KB MB GB), must be a positive integer. Env "TAOS_KEEPER_LOG_RESERVED_DISK_SIZE" (default "1GB")
--log.rotationCount uint log rotation count. Env "TAOS_KEEPER_LOG_ROTATION_COUNT" (default 5)
--log.rotationCount uint log rotation count. Env "TAOS_KEEPER_LOG_ROTATION_COUNT" (default 3)
--log.rotationSize string log rotation size(KB MB GB), must be a positive integer. Env "TAOS_KEEPER_LOG_ROTATION_SIZE" (default "1GB")
--log.rotationTime duration deprecated: log rotation time always 24 hours. Env "TAOS_KEEPER_LOG_ROTATION_TIME" (default 24h0m0s)
--logLevel string log level (trace debug info warning error). Env "TAOS_KEEPER_LOG_LEVEL" (default "info")
Expand Down Expand Up @@ -119,9 +119,9 @@ incgroup = false
# path = "/var/log/taos"
level = "info"
# Number of log file rotations before deletion.
rotationCount = 30
rotationCount = 3
# The number of days to retain log files.
keepDays = 30
keepDays = 3
# The maximum size of a log file before rotation.
rotationSize = "1GB"
# If set to true, log files will be compressed.
Expand Down
25 changes: 25 additions & 0 deletions docs/en/14-reference/03-taos-sql/02-database.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ database_option: {
| SS_KEEPLOCAL value
| SS_CHUNKPAGES value
| SS_COMPACT value
| COMPACT_INTERVAL value
| COMPACT_TIME_RANGE value
| COMPACT_TIME_OFFSET value
}
```

Expand Down Expand Up @@ -85,6 +88,28 @@ database_option: {
- SS_CHUNKPAGES: When shared storage is enabled, data files larger than this size will be migrated to shared storage, only available in the enterprise version 3.3.7.0 and later. Minimum is 131072, maximum is 1048576, default is 131072. The unit is TSDB page, which is typically 4KB.
- SS_COMPACT: When shared storage is enabled, if set to 1, file will be compacted before its first migration; if set to 0, compact is skipped. Only available in the enterprise version 3.3.7.0 and later.

:::note

The following parameters are available in TDengine Enterprise only.

:::

- **COMPACT_INTERVAL:** Interval at which to trigger automatic database compaction. The default value is 0, which disables automatic database compaction. To enable automatic database compaction, specify a value between 10m and `KEEP2`. The time unit of the value can be minutes (m), hours (h), or days (d), and the default unit is days.

- Note that time slices start from 1970-01-01T00:00:00Z.

- Automatic database compaction is not triggered when an existing compaction task is already running on the database.

- **COMPACT_TIME_RANGE:** Time range for automatic compact tasks. The default value is `0, 0`, which indicates the range from `-KEEP2` to `-DURATION`. You can specify a custom time range starting at or after `-KEEP2` and ending at or before `-DURATION`. The time unit of the values in this range can be minutes (m), hours (h), or days (d), and the default unit is days.

For example, `-300, -200` would compact data between 300 and 200 days in the past each time automatic compaction is triggered. If the duration parameter of the database is the default 10 days, `-300, -5` would return an error because the second value (5 days in the past) is more recent than the value of `-DURATION` (10 days in the past).

Note that these values are negative numbers, indicating that the time range to be compacted is in the past.

- **COMPACT_TIME_OFFSET:** Time offset relative to local time at which to trigger automatic database compaction. The default value is 0. You can enter an offset between 0 and 23 to trigger compaction after the specified number of hours.

For example, if `COMPACT_INTERVAL` is `1d` and `COMPACT_TIME_OFFSET` is `0`, automatic compact is triggered at 00:00 every day. If `COMPACT_TIME_OFFSET` is `2`, automatic compact is triggered at 02:00 every day.

### Database Creation Example

```sql
Expand Down
2 changes: 1 addition & 1 deletion docs/zh/14-reference/01-components/01-taosd.md
Original file line number Diff line number Diff line change
Expand Up @@ -975,7 +975,7 @@ charset 的有效值是 UTF-8。
- 类型:整数
- 默认值:CPU 核数的 2 倍 + 5
- 最小值:0
- 最大值:4096
- 最大值:1024
- 参数类型:局部配置参数
- 动态修改:仅在企业版支持通过 SQL 修改,立即生效。
- 支持版本:从 v3.0.0.0 版本开始引入
Expand Down
2 changes: 1 addition & 1 deletion docs/zh/14-reference/01-components/02-taosc.md
Original file line number Diff line number Diff line change
Expand Up @@ -339,7 +339,7 @@ taosc 和 taosd 存在许多同名参数,虽然名称相同但作用范围可
- 最小值:1048576
- 最大值:67108864
- 动态修改:支持通过 SQL 修改,立即生效
- 支持版本:从 v3.3.6.34 版本开始引入
- 支持版本:从 v3.3.6.34、v3.3.8.8 版本开始引入
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The English version of this documentation (docs/en/14-reference/01-components/02-taosc.md) was updated to change the default value of maxSQLLength from 4,194,304 to 1,048,576. This change is missing in the Chinese documentation on line 338. Please update it for consistency.


### 区域相关

Expand Down
8 changes: 4 additions & 4 deletions docs/zh/14-reference/01-components/06-taoskeeper.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,11 @@ Usage of taoskeeper:
-H, --host string http host. Env "TAOS_KEEPER_HOST"
--instanceId int instance ID. Env "TAOS_KEEPER_INSTANCE_ID" (default 64)
--log.compress whether to compress old log. Env "TAOS_KEEPER_LOG_COMPRESS"
--log.keepDays uint log retention days, must be a positive integer. Env "TAOS_KEEPER_LOG_KEEP_DAYS" (default 30)
--log.keepDays uint log retention days, must be a positive integer. Env "TAOS_KEEPER_LOG_KEEP_DAYS" (default 3)
--log.level string log level (trace debug info warning error). Env "TAOS_KEEPER_LOG_LEVEL" (default "info")
--log.path string log path. Env "TAOS_KEEPER_LOG_PATH" (default "/var/log/taos")
--log.reservedDiskSize string reserved disk size for log dir (KB MB GB), must be a positive integer. Env "TAOS_KEEPER_LOG_RESERVED_DISK_SIZE" (default "1GB")
--log.rotationCount uint log rotation count. Env "TAOS_KEEPER_LOG_ROTATION_COUNT" (default 5)
--log.rotationCount uint log rotation count. Env "TAOS_KEEPER_LOG_ROTATION_COUNT" (default 3)
--log.rotationSize string log rotation size(KB MB GB), must be a positive integer. Env "TAOS_KEEPER_LOG_ROTATION_SIZE" (default "1GB")
--log.rotationTime duration deprecated: log rotation time always 24 hours. Env "TAOS_KEEPER_LOG_ROTATION_TIME" (default 24h0m0s)
--logLevel string log level (trace debug info warning error). Env "TAOS_KEEPER_LOG_LEVEL" (default "info")
Expand Down Expand Up @@ -119,9 +119,9 @@ incgroup = false
# path = "/var/log/taos"
level = "info"
# Number of log file rotations before deletion.
rotationCount = 30
rotationCount = 3
# The number of days to retain log files.
keepDays = 30
keepDays = 3
# The maximum size of a log file before rotation.
rotationSize = "1GB"
# If set to true, log files will be compressed.
Expand Down
33 changes: 33 additions & 0 deletions source/client/src/clientMsgHandler.c
Original file line number Diff line number Diff line change
Expand Up @@ -1358,6 +1358,37 @@ int32_t processCreateTotpSecretRsp(void* param, SDataBuf* pMsg, int32_t code) {
return code;
}

int32_t processCreateXnodeTaskRsp(void* param, SDataBuf* pMsg, int32_t code) {
SRequestObj* pRequest = param;
if (code != TSDB_CODE_SUCCESS) {
setErrno(pRequest, code);
if (code == TSDB_CODE_MND_XNODE_HTTP_CODE_ERROR) {
if (pMsg->pData != NULL && pMsg->len > 0) {
if (pMsg->len <= pRequest->msgBufLen) {
tstrncpy(pRequest->msgBuf, (char*)pMsg->pData, pRequest->msgBufLen);
} else {
taosMemoryFreeClear(pRequest->msgBuf);
pRequest->msgBuf = pMsg->pData;
pMsg->pData = NULL;
pRequest->msgBufLen = pMsg->len;
}
}
}
}

if (pMsg->pData) {
taosMemoryFree(pMsg->pData);
}
taosMemoryFree(pMsg->pEpSet);

if (pRequest->body.queryFp != NULL) {
pRequest->body.queryFp(((SSyncQueryParam*)pRequest->body.interParam)->userParam, pRequest, code);
} else if (tsem_post(&pRequest->body.rspSem) != 0) {
tscError("failed to post semaphore");
}
return code;
}


__async_send_cb_fn_t getMsgRspHandle(int32_t msgType) {
switch (msgType) {
Expand Down Expand Up @@ -1385,6 +1416,8 @@ __async_send_cb_fn_t getMsgRspHandle(int32_t msgType) {
return processCreateTokenRsp;
case TDMT_MND_CREATE_TOTP_SECRET:
return processCreateTotpSecretRsp;
case TDMT_MND_CREATE_XNODE_TASK:
return processCreateXnodeTaskRsp;

default:
return genericRspCallback;
Expand Down
2 changes: 1 addition & 1 deletion source/client/src/clientTmq.c
Original file line number Diff line number Diff line change
Expand Up @@ -2449,7 +2449,7 @@ static SMqRspObj* buildRsp(SMqPollRspWrapper* pollRspWrapper){
}

static int32_t processMqRspError(tmq_t* tmq, SMqRspWrapper* pRspWrapper){
int32_t code = 0;
int32_t code = pRspWrapper->code;
SMqPollRspWrapper* pollRspWrapper = &pRspWrapper->pollRsp;

tqErrorC("consumer:0x%" PRIx64 " msg from vgId:%d discarded, since %s", tmq->consumerId, pollRspWrapper->vgId,
Expand Down
2 changes: 1 addition & 1 deletion source/common/src/tglobal.c
Original file line number Diff line number Diff line change
Expand Up @@ -938,7 +938,7 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
// clang-format off
TAOS_CHECK_RETURN(cfgAddDir(pCfg, "dataDir", tsDataDir, CFG_SCOPE_SERVER, CFG_DYN_SERVER, CFG_CATEGORY_LOCAL, CFG_PRIV_SYSTEM));
TAOS_CHECK_RETURN(cfgAddFloat(pCfg, "minimalDataDirGB", 2.0f, 0.001f, 10000000, CFG_SCOPE_SERVER, CFG_DYN_NONE, CFG_CATEGORY_LOCAL, CFG_PRIV_SYSTEM));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "supportVnodes", tsNumOfSupportVnodes, 0, 4096, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER, CFG_CATEGORY_LOCAL, CFG_PRIV_SYSTEM));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "supportVnodes", tsNumOfSupportVnodes, 0, 1024, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER, CFG_CATEGORY_LOCAL, CFG_PRIV_SYSTEM));

TAOS_CHECK_RETURN(cfgAddString(pCfg, "encryptAlgorithm", tsEncryptAlgorithm, CFG_SCOPE_SERVER, CFG_DYN_NONE, CFG_CATEGORY_GLOBAL, CFG_PRIV_SECURITY));
TAOS_CHECK_RETURN(cfgAddString(pCfg, "encryptScope", tsEncryptScope, CFG_SCOPE_SERVER, CFG_DYN_NONE,CFG_CATEGORY_GLOBAL, CFG_PRIV_SECURITY));
Expand Down
58 changes: 44 additions & 14 deletions source/dnode/mnode/impl/src/mndXnode.c
Original file line number Diff line number Diff line change
Expand Up @@ -1620,6 +1620,25 @@ static int32_t mndValidateCreateXnodeTaskReq(SRpcMsg *pReq, SMCreateXnodeTaskReq
code = terrno;
goto _OVER;
}
SJson *errorJson = tjsonGetObjectItem(pJson, "__inner_error");
if (errorJson != NULL) {
code = TSDB_CODE_MND_XNODE_HTTP_CODE_ERROR;
char* pValueString = ((cJSON*)errorJson)->valuestring;
if (NULL == pValueString) {
mError("should not failed to get __inner_error message, task name:%s", pCreateReq->name.ptr);
goto _OVER;
}
//handle response
int32_t contLen = strlen(pValueString) + strlen(tstrerror(code)) + 32;
void *pRsp = rpcMallocCont(contLen);
if (pRsp == NULL) {
TAOS_CHECK_GOTO(terrno, NULL, _OVER);
}
pReq->info.rspLen = contLen;
pReq->info.rsp = pRsp;
snprintf(pReq->info.rsp, contLen, "%s, since: %s", tstrerror(code), pValueString);
goto _OVER;
}

// todo: only4test
// (void)mndSendReqRetJson(xnodeUrl, HTTP_TYPE_POST, 60000, pContStr, strlen(pContStr));
Expand Down Expand Up @@ -1658,7 +1677,7 @@ static int32_t mndHandleCreateXnodeTaskResult(int32_t createCode) {
static int32_t mndProcessCreateXnodeTaskReq(SRpcMsg *pReq) {
mDebug("xnode create task request received, contLen:%d\n", pReq->contLen);
SMnode *pMnode = pReq->info.node;
int32_t code = -1;
int32_t code = 0;
SMCreateXnodeTaskReq createReq = {0};

// Step 1: Validate permissions
Expand Down Expand Up @@ -3909,7 +3928,7 @@ static int32_t taosCurlPostRequest(const char *url, SCurlResp *pRsp, const char
TAOS_CHECK_GOTO(curl_easy_setopt(curl, CURLOPT_POST, 1), &lino, _OVER);
TAOS_CHECK_GOTO(curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, bufLen), &lino, _OVER);
TAOS_CHECK_GOTO(curl_easy_setopt(curl, CURLOPT_POSTFIELDS, buf), &lino, _OVER);
TAOS_CHECK_GOTO(curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L), &lino, _OVER);
TAOS_CHECK_GOTO(curl_easy_setopt(curl, CURLOPT_VERBOSE, 0L), &lino, _OVER);
TAOS_CHECK_GOTO(curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L), &lino, _OVER);

mDebug("xnode curl post request will sent, url:%s len:%d content:%s", url, bufLen, buf);
Expand Down Expand Up @@ -4001,7 +4020,7 @@ SJson *mndSendReqRetJson(const char *url, EHttpType type, int64_t timeout, const
if (!taosCheckExistFile(socketPath)) {
uError("xnode failed to send request, socket path:%s not exist", socketPath);
terrno = TSDB_CODE_MND_XNODE_URL_CANT_ACCESS;
goto _OVER;
goto _EXIT;
}
if (type == HTTP_TYPE_GET) {
if ((terrno = taosCurlGetRequest(url, &curlRsp, timeout, socketPath)) != 0) {
Expand All @@ -4018,21 +4037,32 @@ SJson *mndSendReqRetJson(const char *url, EHttpType type, int64_t timeout, const
} else {
uError("xnode invalid http type:%d", type);
terrno = TSDB_CODE_MND_XNODE_INVALID_MSG;
goto _OVER;
goto _EXIT;
}

if (curlRsp.data == NULL || curlRsp.dataLen == 0) {
_OVER:
if (terrno == TSDB_CODE_SUCCESS) {
if (curlRsp.data == NULL || curlRsp.dataLen == 0) {
pJson = tjsonCreateObject();
goto _EXIT;
}
pJson = tjsonParse(curlRsp.data);
if (pJson == NULL) {
terrno = TSDB_CODE_INVALID_JSON_FORMAT;
goto _EXIT;
}
} else if (terrno == TSDB_CODE_MND_XNODE_HTTP_CODE_ERROR) {
pJson = tjsonCreateObject();
goto _OVER;
}

pJson = tjsonParse(curlRsp.data);
if (pJson == NULL) {
terrno = TSDB_CODE_INVALID_JSON_FORMAT;
goto _OVER;
char *buf = taosMemCalloc(1, curlRsp.dataLen + 1);
Comment on lines 4055 to +4056
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

tjsonCreateObject() and taosMemCalloc() can return NULL if memory allocation fails. The returned pointers pJson and buf are used without being checked, which could lead to a crash. Please add NULL checks after these calls to handle potential memory allocation failures gracefully.

    pJson = tjsonCreateObject();
    if (pJson == NULL) {
      terrno = TSDB_CODE_NO_MEMORY;
      goto _EXIT;
    }
    char *buf = taosMemCalloc(1, curlRsp.dataLen + 1);
    if (buf == NULL) {
      tjsonDelete(pJson);
      pJson = NULL;
      terrno = TSDB_CODE_NO_MEMORY;
      goto _EXIT;
    }

(void)memcpy(buf, curlRsp.data, curlRsp.dataLen);
if (tjsonAddStringToObject(pJson, "__inner_error", buf) != TSDB_CODE_SUCCESS) {
taosMemoryFreeClear(buf);
goto _EXIT;
}
taosMemoryFreeClear(buf);
}

_OVER:
_EXIT:
if (curlRsp.data != NULL) taosMemoryFreeClear(curlRsp.data);
if (terrno != TSDB_CODE_SUCCESS) {
mError("xnode failed to send request, url: %s, since:%s", url, tstrerror(terrno));
Expand Down Expand Up @@ -5125,4 +5155,4 @@ void mndRestartXnoded(SMnode *pMnode) {
mndReleaseXnodeUserPass(pMnode, pObj);
mInfo("mndxnode xnoded restarted");
return;
}
}
1 change: 1 addition & 0 deletions source/dnode/vnode/src/vnd/vnodeStream.c
Original file line number Diff line number Diff line change
Expand Up @@ -1741,6 +1741,7 @@ static int32_t scanSubmitData(SVnode* pVnode, SStreamTriggerReaderInfo* sStreamR
WalMetaResult* pMeta = (WalMetaResult*)px;
STREAM_CHECK_RET_GOTO(buildWalMetaBlockNew(rsp->metaBlock, pMeta->id, pMeta->skey, pMeta->ekey, ver));
((SSDataBlock*)rsp->metaBlock)->info.rows++;
rsp->totalRows++;
ST_TASK_DLOG("%s process meta data:skey %" PRId64 ", ekey %" PRId64 ", id %" PRIu64
", ver:%"PRId64, __func__, pMeta->skey, pMeta->ekey, pMeta->id, ver);
px = tSimpleHashIterate(gidHash, px, &iter);
Expand Down
Loading
Loading