Dunimd Middleware Service

English | 简体中文

Help Documentation | Changelog | Security | Contributing | Code of Conduct

DMSC (Dunimd Middleware Service) — A high-performance Rust middleware framework that unifies backend infrastructure. Built for enterprise-scale with modular architecture, built-in observability, and distributed systems support.

🏗️ Core Architecture

📐 Modular Design

DMSC adopts a highly modular architecture with 18 core modules, enabling on-demand composition and seamless extension:

Module	Description
auth	Authentication & authorization (JWT, OAuth, permissions)
cache	Multi-backend cache abstraction (Memory, Redis, Hybrid)
config	Multi-source configuration management with hot reload
core	Runtime, error handling, and service context
database	Database abstraction with PostgreSQL, MySQL, SQLite support
device	Device control, discovery, and intelligent scheduling
fs	Secure file system operations and management
gateway	API gateway with load balancing, rate limiting, and circuit breaking
grpc	gRPC server and client support with Python bindings (requires `grpc` feature)
hooks	Lifecycle event hooks (Startup, Shutdown, etc.)
log	Structured logging with tracing context integration
module_rpc	Inter-module RPC communication for distributed method calls
observability	Metrics, tracing, and Grafana integration
database.orm	Type-safe ORM with repository pattern, query builder, and Python bindings
protocol	Protocol abstraction layer for multi-protocol support (requires `pyo3` feature)
queue	Distributed queue abstraction (Kafka, RabbitMQ, Redis, Memory)
service_mesh	Service discovery, health checking, and traffic management
validation	Input validation and data sanitization utilities
ws	WebSocket server support with Python bindings (requires `websocket` feature)
c	C/C++ FFI bindings for cross-language integration (requires `c` feature)
java	Java JNI bindings for Java application integration (requires `java` feature)

Note: Some modules require specific feature flags:

grpc: gRPC support (--features grpc)

websocket: WebSocket support (--features websocket)

protocol: Protocol abstraction layer (--features protocol or full)

c: C/C++ FFI bindings (--features c)

java: Java JNI bindings (--features java)

🚀 Key Features

🔍 Distributed Tracing

W3C Trace Context standard implementation
Full-chain TraceID/SpanID propagation
Baggage data transmission for business context
Multi-language compatibility (Java, Go, Python)
Automatic span creation via #[tracing::instrument] attribute

📊 Enterprise Observability

Native Prometheus metrics export
Counter, Gauge, Histogram, Summary metric types
Out-of-the-box Grafana dashboard integration
Real-time performance statistics with quantile calculation
Full-stack metrics (CPU, memory, I/O, network)

🤖 Intelligent Device Management

Auto-discovery and registration
Efficient resource pool management
Policy-based scheduling with priority support
Dynamic load balancing
Complete device lifecycle management

📝 Structured Logging

JSON and text format support
Configurable sampling rates
Intelligent log rotation
Automatic tracing context inclusion
DEBUG/INFO/WARN/ERROR log levels

⚙️ Flexible Configuration

Multi-source loading (files, environment variables, runtime)
Hot configuration updates
Modular architecture for on-demand composition
Plugin-based extension mechanism

📁 Secure File System

Unified project root directory management
Atomic file operations
Categorized directory structure
JSON data persistence
Secure path handling

🛠️ Installation & Environment

Prerequisites

Rust: 1.65+ (2021 Edition)
Cargo: 1.65+
Platforms: Linux (x64, arm64), macOS (x64, arm64), Windows (x64, arm64), Android (arm64-v8a, armeabi-v7a, x86_64)

Build Dependencies

Some features require additional system dependencies:

Dependency	Required For	Installation
protoc	etcd feature (Protocol Buffers)	Protocol Buffers
CMake + C++ compiler	kafka feature (rdkafka)	See instructions below

Installing protoc

Windows:

# Using chocolatey
choco install protoc

# Or download from GitHub releases
# https://github.com/protocolbuffers/protobuf/releases

macOS:

brew install protobuf

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install -y protobuf-compiler

Linux (CentOS/RHEL):

sudo yum install -y protobuf-compiler

Installing CMake and C++ compiler (for Kafka support)

Windows:

# CMake is usually installed with Visual Studio
# Or download from: https://cmake.org/download/

# Using chocolatey
choco install cmake

macOS:

# CMake and C++ compiler (Xcode Command Line Tools)
xcode-select --install

# Or using Homebrew
brew install cmake

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install -y cmake build-essential

Linux (CentOS/RHEL):

sudo yum install -y cmake gcc-c++ make

Quick Setup

Add DMSC to your project's Cargo.toml:

[dependencies]
dmsc = "0.1.8"

Or use cargo add:

cargo add dmsc

⚡ Quick Start

Core API Usage

use dmsc::prelude::*;

#[tokio::main]
async fn main() -> DMSCResult<()> {
    // Build service runtime
    let app = DMSCAppBuilder::new()
        .with_config("config.yaml")?
        .with_logging(DMSCLogConfig::default())
        .with_observability(DMSCObservabilityConfig::default())
        .build()?;
    
    // Run business logic
    app.run(|ctx: &DMSCServiceContext| async move {
        ctx.logger().info("service", "DMSC service started")?;
        // Your business code here
        Ok(())
    }).await
}

Observability Example

use dmsc::prelude::*;
use dmsc::observability::{DMSCTracer, DMSCSpanKind, DMSCSpanStatus};

#[tracing::instrument(name = "user_service", skip(ctx))]
async fn get_user(ctx: &DMSCServiceContext, user_id: u64) -> DMSCResult<User> {
    let user = fetch_user_from_db(user_id).await?;
    Ok(user)
}

Or using DMSCTracer directly:

use dmsc::prelude::*;
use dmsc::observability::DMSCTracer;

async fn get_user(ctx: &DMSCServiceContext, user_id: u64) -> DMSCResult<User> {
    let tracer = DMSCTracer::new(1.0);
    let _span = tracer.span("get_user")
        .with_attribute("user_id", user_id.to_string())
        .start();
    let user = fetch_user_from_db(user_id).await?;
    Ok(user)
}

🔧 Configuration

Configuration Example

# config.yaml
service:
  name: "my-service"
  version: "1.0.0"

logging:
  level: "info"
  file_format: "json"
  file_enabled: true
  console_enabled: true

observability:
  metrics_enabled: true
  tracing_enabled: true
  prometheus_port: 9090

resource:
  providers: ["cpu", "gpu", "memory"]
  scheduling_policy: "priority_based"

Configuration Sources

DMSC supports multiple configuration sources in order of priority (lowest to highest):

Configuration files (YAML, TOML, JSON)
Custom configuration via code
Environment variables (prefixed with DMSC_)

🧪 Development & Testing

Running Tests

Multi-Language Testing

DMSC provides comprehensive testing across all supported languages:

Rust: Core library tests with cargo test
Python: Python binding tests with pytest
Java: JNI binding tests with standard Java test runner
C/C++: C API tests with native compilers

Test Coverage

The tests verify:

✅ Builder pattern behavior in all languages
✅ Method chaining returns appropriate instances (language-specific)
✅ Runtime creation and lifecycle management
✅ Error handling and edge cases
✅ Cross-language API consistency

Running Rust Tests

# Run all Rust tests
cargo test

# Run specific test modules
cargo test --lib app_builder
cargo test --lib app_runtime

# Run with verbose output
cargo test -- --nocapture

# Run with all features
cargo test --all-features

Running Python Tests

# Install Python package in development mode
cd python
pip install -e .

# Run all Python tests
python -m pytest tests/Python/ -v

# Run specific test classes
python -m pytest tests/Python/test_core.py::TestDMSCAppBuilderWrapper -v
python -m pytest tests/Python/test_core.py::TestDMSCAppRuntimeWrapper -v

Running Java Tests

# Build JNI library
cargo build --release --no-default-features --features java

# Compile and run Java tests
cd java
javac -d build/classes/java/test -cp build/classes/java/main \
  src/test/java/TestAll.java src/test/java/TestAppBuilder.java

java -cp build/classes/java/test:build/classes/java/main \
  -Djava.library.path=../target/release TestAll

API Behavior Across Languages

Language	Builder Pattern	Method Chaining	Reason
Rust	Returns `Self`	Consumes original	Native builder pattern
Python	Returns `self`	Same instance	Pythonic wrapper for PyO3
Java	Returns new instance	Immutable builder	Java best practice
C	Returns new pointer	Memory management	C idioms

❓ Frequently Asked Questions

Q: How to add a new module? A: Implement the DMSCModule trait and register it via DMSCAppBuilder::with_module.

Q: How to configure logging level? A: Set logging.level in the configuration file, supporting DEBUG/INFO/WARN/ERROR levels.

Q: How to enable metrics export? A: Set observability.metrics_enabled: true and configure prometheus_port in the configuration file.

Q: How to extend configuration sources? A: Implement a custom configuration loader and register it with DMSCConfigManager.

Q: How to handle asynchronous tasks? A: Use DMSCAppBuilder::with_async_module to add async modules, the framework handles async lifecycle automatically.

🌏 Community & Citation

Welcome to submit Issues and PRs!
Github: https://github.com/mf2023/DMSC.git
Gitee: https://gitee.com/dunimd/dmsc.git
GitCode: https://gitcode.com/dunimd/dmsc.git

📄 License & Open Source Agreements

🏛️ Project License

This project uses Apache License 2.0 open source agreement, see LICENSE file.

📋 Dependency Package Open Source Agreements

Open source packages and their agreement information used by this project:

Dependencies License

📦 Package	📜 License	📦 Package	📜 License
serde	MIT/Apache-2.0	serde_json	MIT/Apache-2.0
serde_yaml	MIT/Apache-2.0	tokio	MIT
futures	MIT/Apache-2.0	futures-util	MIT/Apache-2.0
http	MIT/Apache-2.0	hyper	MIT
prometheus	MIT/Apache-2.0	redis	MIT
lapin	MIT/Apache-2.0	rdkafka	MIT
yaml-rust	MIT/Apache-2.0	toml	MIT/Apache-2.0
etcd-client	MIT	sysinfo	MIT
async-trait	MIT/Apache-2.0	dashmap	MIT
chrono	MIT	uuid	MIT/Apache-2.0
rand	MIT/Apache-2.0	notify	CC0-1.0
jsonwebtoken	MIT	reqwest	MIT/Apache-2.0
urlencoding	MIT	parking_lot	MIT/Apache-2.0
log	MIT/Apache-2.0	tracing	MIT
pyo3	MIT/Apache-2.0	jni	MIT/Apache-2.0
safer-ffi	MIT	tempfile	MIT/Apache-2.0
thiserror	MIT/Apache-2.0	hex	MIT/Apache-2.0
base64	MIT/Apache-2.0	regex	MIT/Apache-2.0
url	MIT/Apache-2.0	aes-gcm	MIT/Apache-2.0
ring	ISC	lazy_static	MIT/Apache-2.0
libloading	ISC	zeroize	MIT/Apache-2.0
zeroize_derive	MIT/Apache-2.0	secrecy	MIT
erased-serde	MIT	data-encoding	MIT
crc32fast	MIT/Apache-2.0	generic-array	MIT
bincode	MIT	typenum	MIT/Apache-2.0
html-escape	MIT	rustls	MIT/Apache-2.0
rustls-pemfile	MIT/Apache-2.0	webpki	ISC
rustls-native-certs	MIT/Apache-2.0	tokio-rustls	MIT/Apache-2.0
bytes	MIT	tonic	MIT
prost	MIT/Apache-2.0	prost-types	MIT/Apache-2.0
tokio-stream	MIT	tower	MIT
async-stream	MIT	tokio-tungstenite	MIT
tungstenite	MIT	num-bigint	MIT/Apache-2.0
oqs	MIT/Apache-2.0	sm-crypto	MIT
openssl-sys	Apache-2.0	tokio-postgres	MIT/Apache-2.0
rusqlite	MIT	sqlx	MIT/Apache-2.0
criterion	MIT/Apache-2.0

Name		Name	Last commit message	Last commit date
Latest commit History 146 Commits
.cargo		.cargo
.github/workflows		.github/workflows
assets/svg		assets/svg
benches		benches
examples		examples
java		java
python		python
src		src
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
README.zh.md		README.zh.md
SECURITY.md		SECURITY.md
cbindgen.toml		cbindgen.toml
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation

Dunimd Middleware Service

🏗️ Core Architecture

📐 Modular Design

🚀 Key Features

🔍 Distributed Tracing

📊 Enterprise Observability

🤖 Intelligent Device Management

📝 Structured Logging

⚙️ Flexible Configuration

📁 Secure File System

🛠️ Installation & Environment

Prerequisites

Build Dependencies

Installing protoc

Installing CMake and C++ compiler (for Kafka support)

Quick Setup

⚡ Quick Start

Core API Usage

Observability Example

🔧 Configuration

Configuration Example

Configuration Sources

🧪 Development & Testing

Running Tests

Multi-Language Testing

Test Coverage

Running Rust Tests

Running Python Tests

Running Java Tests

API Behavior Across Languages

❓ Frequently Asked Questions

🌏 Community & Citation

📄 License & Open Source Agreements

🏛️ Project License

📋 Dependency Package Open Source Agreements

Dependencies License

About

Topics

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 8

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages