158 lines
No EOL
3.9 KiB
Markdown
158 lines
No EOL
3.9 KiB
Markdown
# Standardized Performance Benchmarking Format
|
|
|
|
## Version 1.1.0
|
|
**Last Updated**: 2025-05-04T11:18:31-05:00
|
|
**Schema Version**: 1.1.0
|
|
|
|
## Required Sections
|
|
1. **Test Environment**
|
|
- Hardware specifications
|
|
- Software versions
|
|
- Network configuration
|
|
- Test date (ISO 8601 format)
|
|
|
|
2. **Security Requirements**
|
|
```markdown
|
|
1. Encryption: AES-256 for secrets
|
|
2. Access Control: RBAC implementation
|
|
3. Audit Logging: 90-day retention
|
|
4. Transport Security: TLS 1.3 required
|
|
5. Performance Targets:
|
|
- CLI Response ≤500ms (with security)
|
|
- Web API Response ≤800ms (with security)
|
|
- Memory ≤512MB
|
|
```
|
|
|
|
3. **Benchmark Methodology**
|
|
- Test duration
|
|
- Warmup period (minimum 5 runs)
|
|
- Measurement approach
|
|
- Iteration count (minimum 100)
|
|
- Test script reference
|
|
|
|
4. **JSON Schema Specification**
|
|
```json
|
|
{
|
|
"$schema": "http://json-schema.org/draft-07/schema#",
|
|
"type": "object",
|
|
"required": [
|
|
"version",
|
|
"timestamp",
|
|
"environment",
|
|
"cli_interface",
|
|
"web_interface",
|
|
"test_parameters"
|
|
],
|
|
"properties": {
|
|
"version": {
|
|
"type": "string",
|
|
"pattern": "^\\d{4}\\.\\d$"
|
|
},
|
|
"timestamp": {
|
|
"type": "string",
|
|
"format": "date-time"
|
|
},
|
|
"environment": {
|
|
"type": "object",
|
|
"properties": {
|
|
"hardware": {"type": "string"},
|
|
"software": {"type": "string"},
|
|
"network": {"type": "string"},
|
|
"test_date": {"type": "string", "format": "date"}
|
|
}
|
|
},
|
|
"cli_interface": {
|
|
"$ref": "#/definitions/interfaceMetrics"
|
|
},
|
|
"web_interface": {
|
|
"$ref": "#/definitions/interfaceMetrics"
|
|
},
|
|
"test_parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"iterations": {"type": "integer", "minimum": 100},
|
|
"warmup_runs": {"type": "integer", "minimum": 5},
|
|
"test_script": {"type": "string"},
|
|
"validation": {
|
|
"type": "object",
|
|
"properties": {
|
|
"schema": {"type": "string"},
|
|
"last_validated": {"type": "string", "format": "date-time"}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"definitions": {
|
|
"interfaceMetrics": {
|
|
"type": "object",
|
|
"properties": {
|
|
"baseline": {"$ref": "#/definitions/measurement"},
|
|
"security_metrics": {
|
|
"type": "object",
|
|
"properties": {
|
|
"rbac": {"$ref": "#/definitions/securityMeasurement"},
|
|
"tls": {"$ref": "#/definitions/securityMeasurement"},
|
|
"full_security": {"$ref": "#/definitions/securityMeasurement"}
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"measurement": {
|
|
"type": "object",
|
|
"properties": {
|
|
"avg_time_ms": {"type": "number"},
|
|
"throughput_rps": {"type": "number"}
|
|
}
|
|
},
|
|
"securityMeasurement": {
|
|
"allOf": [
|
|
{"$ref": "#/definitions/measurement"},
|
|
{
|
|
"type": "object",
|
|
"properties": {
|
|
"overhead_ms": {"type": "number"}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
5. **Validation Requirements**
|
|
1. JSON Schema validation
|
|
2. Timestamp format verification
|
|
3. Required field checks
|
|
4. Security metric completeness
|
|
5. Interface consistency validation
|
|
6. Test parameter validation
|
|
|
|
6. **Example CLI Benchmark**
|
|
```json
|
|
{
|
|
"cli_interface": {
|
|
"baseline": {
|
|
"avg_time_ms": 120,
|
|
"throughput_rps": 83.3
|
|
},
|
|
"security_metrics": {
|
|
"rbac": {
|
|
"avg_time_ms": 145,
|
|
"throughput_rps": 69.0,
|
|
"auth_overhead_ms": 25
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
7. **Version History**
|
|
- 1.1.0 (2025-05-04): Added CLI/web interface separation, standardized security metrics
|
|
- 1.0.0 (2025-04-15): Initial release
|
|
|
|
8. **Implementation Notes**
|
|
- Null values indicate unmeasured metrics
|
|
- Reference implementation: performance_logs.json
|
|
- Schema validation script: tests/performance/validate_schema.py
|
|
- Current implementation: performance_logs.json (v1.1.0) |