owasp-sbot MGraph-AI Type / to search

Code Issues Pull requests Actions Projects Wiki Security Insights Settings
dev MGraph-AI / docs / mgraph / mgraph-architecture-combined__jan-12.md Go to file t

DinisCruz refactored location of mgragh doc files d402bc1 · now History

Preview Code Blame 392 lines (325 loc) · 11 KB Raw

MGraph Architecture: Comprehensive Analysis and

Planning
Introduction
The MGraph-AI project has reached a critical point where architectural refinement is necessary to support its expanding
capabilities and use cases. The ability to make these significant architectural changes is enabled by our comprehensive test
coverage, achieved through AI-assisted test generation. This 100% code coverage provides confidence in refactoring while
maintaining system integrity and functionality.
At its core, MGraph-AI serves as a universal graph-based data transformation and manipulation system. It creates an internal
representation that can seamlessly convert between different data formats while preserving data fidelity. This enables powerful
data manipulation capabilities while maintaining the ability to round-trip data back to its original format or transform it into any
supported format.
Data Flow Architecture
Provider Data Flow
MGraph-AI implements a sophisticated data flow pattern where external data formats are transformed into an internal
representation and back:
1. Input Phase
External data in various formats (RSS, XML, JSON, etc.)
Provider-specific parsers and validators
Conversion to internal MGraph format
2. Processing Phase
Data manipulation using MGraph APIs
Filtering, transformation, and analysis
Graph operations and queries
3. Output Phase
Serialization to target formats
Format-specific optimizations
Data validation and integrity checks
Example workflow with RSS feed:
feed-abc.xml (RSS)
→ feed-abc.mgraph.json (Internal representation)
→ feed-abc.{json|rdf|ttl|json-ld} (Output formats)
Recent Architectural Changes
Domain Layer Clarification
Introduced Domain__ prefix for all domain classes
Creates clear separation between domain logic and provider implementations
Resolves naming conflicts and ambiguity in graph class hierarchy
Establishes clear boundaries between core domain logic and provider-specific implementations
Layer Responsibilities
1. Schema Layer (Data Models)
The Schema layer focuses on defining pure data structures without any business logic or external dependencies. It serves as the
foundation for type safety and data validation.
Core responsibilities:
Defines data structures and validation rules
Pure data containers with type definitions
No business logic or external dependencies
Focus on serialization/deserialization
Type safety enforcement
Constraint definition
Serialization support
2. Model Layer (Data Operations)
The Model layer handles all direct data manipulations, working with single model/schema pairs and maintaining data integrity within
its scope.
Core responsibilities:
Handles data manipulation within its scope
Operates on single model/schema pair
Unaware of broader graph context
Focused on data integrity and type safety
Business rule validation
Type conversion
Basic data operations
3. Domain Layer (Business Logic)
The Domain layer orchestrates operations across multiple models and maintains overall system consistency.
Core responsibilities:
Orchestrates operations across models
Maintains graph-wide consistency
Handles cross-entity relationships
Provides rich domain-specific operations
Access to complete graph context
Complex business rules
Cross-entity operations
4. Actions Layer (API Surface)
The Actions layer provides the primary interface for system users, exposing intuitive, use-case driven operations.
Core responsibilities:
Exposes intuitive, use-case driven interfaces
Simplifies complex operations
Provides clear, task-specific naming
Handles common patterns and workflows
User interaction patterns
Operation composition
Error handling
Provider Implementation Analysis
Current Challenges
1. High implementation overhead
Many required classes per provider
Significant boilerplate code
Complex inheritance chains
2. Code Duplication
Similar patterns repeated across providers
Common functionality reimplemented
Maintenance burden
3. API Consistency
Different providers may expose different interfaces
 Varying levels of functionality
Inconsistent method naming
Proposed Improvements
1. Provider Template System
Create a standardized template system for new providers:
class Provider__Template:
schema_classes = {
'node': Schema__Template__Node,
'edge': Schema__Template__Edge,
'graph': Schema__Template__Graph
}
model_classes = {
'node': Model__Template__Node,
'edge': Model__Template__Edge,
'graph': Model__Template__Graph
}
domain_classes = {
'node': Domain__Template__Node,
'edge': Domain__Template__Edge,
'graph': Domain__Template__Graph
}

2. Base Class Generation

Automated generation of required base classes
Default implementations for common patterns
Extension points for provider-specific logic
 Reduced boilerplate code
3. Standard Provider Interface
class MGraph__Provider(MGraph):
def __init__(self, config=None):
self.provider = Provider__Template()
self.graph = self.provider.create_graph(config)

@classmethod
def from_data(cls, data):
"""Create graph from serialized data"""
pass

def to_data(self, format='native'):

"""Serialize graph to specified format"""
pass

Data Format Interoperability

Core Concepts
MGraph-AI's internal representation serves as a universal intermediate format that captures all necessary information from source
formats while enabling lossless transformation to target formats. This is achieved through several key mechanisms:
1. Universal ID System
Internal GUID-based identification
Format-specific ID mapping
ID preservation for round-trip operations
 2. Metadata Preservation
Format-specific attributes
Common attribute mapping
Extended property storage
3. Relationship Mapping
Graph-based relationship model
Format-specific relationship translation
Bi-directional reference tracking
Format-Specific Considerations
Different formats have varying capabilities for representing relationships and metadata. MGraph-AI handles these differences
through:
1. JSON Format
{
"nodes": {
"n1": {
"id": "guid-1234",
"original_id": "article-1",
"format_specific": {
"rss": { "pubDate": "2024-01-12" }
}
}
}
}

2. XML/RSS Format
 <rss version="2.0">
<channel>
<item>
<mgraph:id>guid-1234</mgraph:id>
<title>Article 1</title>
<pubDate>2024-01-12</pubDate>
</item>
</channel>
</rss>

3. RDF/Turtle Format
@prefix mgraph: <http://mgraph.ai/schema#> .

<guid-1234> a mgraph:Node ;
mgraph:original_id "article-1" ;
rss:pubDate "2024-01-12" .

Core Data Structure

Define minimal representation that captures:
Node/edge relationships
Attributes and metadata
Type information
Provider-specific extensions
Format-Specific Mappings
JSON
 {
"nodes": {
"n1": {"type": "node", "attributes": {}},
"n2": {"type": "node", "attributes": {}}
},
"edges": {
"e1": {
"from": "n1",
"to": "n2",
"attributes": {}
}
}
}

XML
<graph>
<nodes>
<node id="n1">
<attributes/>
</node>
<node id="n2">
<attributes/>
</node>
</nodes>
<edges>
<edge id="e1" from="n1" to="n2">
<attributes/>
</edge>
</edges>
</graph>
RDF/JSON-LD
{
"@context": {
"mgraph": "http://mgraph.ai/schema#",
"node": "mgraph:node",
"edge": "mgraph:edge"
},
"@graph": [
{
"@type": "node",
"@id": "n1"
},
{
"@type": "edge",
"@id": "e1",
"from": "n1",
"to": "n2"
}
]
}

Transformation Pipeline
The transformation process follows these steps:
1. Import Phase
# Example RSS import
rss_provider = MGraph__RSS()
graph = rss_provider.load('feed-abc.xml')
graph.save('feed-abc.mgraph.json')
 2. Manipulation Phase
# Data manipulation
with graph.edit() as edit:
for node in edit.nodes():
if node.has_attribute('pubDate'):
# Transform date format
node.transform_date_format()

3. Export Phase
# Export to multiple formats
graph.export('feed-abc.json')
graph.export('feed-abc.ttl', format='turtle')
graph.export('feed-abc.jsonld', format='json-ld')

Implementation Priorities
1. Core Data Structure
Define minimal required fields
Create type definitions
Implement validation
Ensure format compatibility
Define extension points
2. Provider Template System
Create base templates
 Implement generation tools
Document extension points
Standardize interfaces
Automate boilerplate
3. Format Conversion Framework
Implement core serializers
Add format converters
Create validation tools
Handle data loss scenarios
Preserve metadata
4. Documentation & Examples
Provider implementation guide
Format specifications
Usage examples
Best practices
Migration guides
Next Steps
The path forward focuses on these key areas:
1. Provider Template System
Generator for boilerplate code
Standard interfaces for common operations
 Documentation and examples
Testing templates
Validation tools
2. Format Conversion Framework
Lossless conversion pipeline
Format-specific optimizations
Validation tools
Error handling
Performance optimization
3. API Enhancement
Simplified provider creation
Common manipulation patterns
Batch operation support
Error handling
Performance monitoring
4. Documentation & Examples
Complete format specifications
Transformation examples
Best practices guide
Performance guidelines
Security considerations
Conclusion
The refined architecture of MGraph-AI provides a solid foundation for handling diverse data formats while maintaining system
flexibility and extensibility. The clear separation of concerns across layers, combined with standardized provider templates and
robust format conversion capabilities, enables efficient implementation of new providers while ensuring consistent behavior across
the system.
The success of these architectural improvements will be measured by:
Reduced implementation time for new providers
Consistent behavior across formats
Maintainable and testable code
Clear and intuitive APIs
Robust error handling
Efficient performance
Regular review and refinement of these architectural decisions will ensure the system continues to meet evolving requirements
while maintaining its core principles of flexibility and reliability.