Starship 2.0

[Anmol1696]

Starship 2.0: Migrate from Helm to KubernetesJS Architecture

Epic Overview

This epic tracks the migration of Starship from a Helm-based deployment system to a modern KubernetesJS-based architecture that provides better programmatic control, improved debugging capabilities, and a more maintainable codebase.

Problem Statement

Current Architecture Limitations

1. Shell Command Dependency: The current client heavily relies on shell execution of kubectl, helm, and docker commands, creating:

   • Debugging difficulties: Errors are opaque and hard to trace
   • Platform compatibility issues: Different behavior across operating systems
   • Limited error handling: Shell command failures provide minimal context
   • Testing challenges: Mocking shell commands is complex and unreliable

2. Helm Template Complexity: The current Helm-based approach:

   • Limited programmatic control: Difficult to dynamically modify resources
   • Complex templating logic: Helm templates in starship/charts/devnet/templates/ are hard to maintain
   • No intermediate artifacts: Resources are deployed directly without visibility into generated YAML
   • Debugging overhead: Requires --dry-run --debug to see generated resources

3. Deployment Opacity:

   • No YAML inspection: Users cannot see what will be deployed before execution
   • Limited customization: Hard to modify resources programmatically before deployment
   • No staged deployment: All resources deployed at once without granular control

Current Implementation Details

Based on codebase analysis:

• Main client: clients/js/packages/client/src/client.ts (999 lines)
• Shell dependencies: Uses shelljs for executing external commands
• Helm charts: Located in starship/charts/devnet/ with complex template structure
• Dependencies: kubectl, helm, docker must be installed and configured
• CLI commands: deploy, start, stop, undeploy all rely on shell execution

// Current approach - shell-based
this.exec(['helm', 'install', '-f', this.ctx.config, this.config.name, this.ctx.chart]);
this.exec(['kubectl', 'get', 'pods', ...this.getArgs()]);

Proposed Solution: KubernetesJS Architecture

Vision

Transform Starship into a modern, programmatic Kubernetes deployment system that:

1. Generates YAML artifacts first for inspection and version control
2. Uses KubernetesJS for direct API communication
3. Provides granular control over deployment lifecycle
4. Eliminates shell dependencies for core functionality

Architecture Overview

graph TD
    A[Starship Config] --> B[KubernetesJS Templating Engine]
    B --> C[Generated YAML Files]
    C --> D[YAML Validation & Review]
    D --> E[KubernetesJS Client Deployment]
    E --> F[Direct Kubernetes API Calls]
    F --> G[Deployed Resources]
    
    H[Port Forwarding] --> I[KubernetesJS Port Forward API]
    J[Resource Monitoring] --> K[KubernetesJS Watch API]

Loading

Implementation Proposals

Proposal 1: Phased Migration (Recommended)

Phase 1: YAML Generation Layer

• Create KubernetesJS templating system parallel to Helm
• Generate YAML files to disk before deployment
• Maintain Helm deployment for backward compatibility
• Add --generate-only flag for YAML-only mode

Phase 2: KubernetesJS Deployment

• Implement KubernetesJS-based deployment alongside YAML generation
• Add --use-k8s-client flag for new deployment method
• Maintain shell-based deployment as fallback

Phase 3: Shell Command Elimination

• Replace all kubectl commands with KubernetesJS API calls
• Remove Helm dependency entirely
• Update dependency management (deps.ts) to remove helm requirement

Phase 4: API Enhancement

• Add advanced features like resource watching, streaming logs
• Implement proper error handling and retry logic
• Add deployment rollback capabilities

Proposal 2: Complete Rewrite

Advantages:

• Clean architecture from start
• No technical debt from old system
• Optimized for KubernetesJS patterns

Disadvantages:

• Higher risk due to big-bang approach
• Temporary loss of Helm-specific features
• Longer development time before usable release

Proposal 3: Hybrid Approach

Concept:

• Keep Helm for complex templating scenarios
• Use KubernetesJS for deployment and management
• Generate YAML from Helm, deploy via KubernetesJS

Trade-offs:

• Maintains Helm dependency
• More complex architecture
• Limited benefits compared to full migration

Implementation Plan

Phase 1: Foundation (Weeks 1-2)

1.1 Create KubernetesJS Templating System

// New file: clients/js/packages/client/src/kubernetes/templating.ts
export class KubernetesTemplatingEngine {
  constructor(private config: StarshipConfig) {}
  
  generateDeployments(): V1Deployment[] {
    return this.config.chains.map(chain => this.createChainDeployment(chain));
  }
  
  generateServices(): V1Service[] {
    return this.config.chains.map(chain => this.createChainService(chain));
  }
  
  generateConfigMaps(): V1ConfigMap[] {
    return this.config.chains.map(chain => this.createChainConfigMap(chain));
  }
}

1.2 Add YAML Generation Capability

// New file: clients/js/packages/client/src/kubernetes/yaml-generator.ts
export class YAMLGenerator {
  constructor(private engine: KubernetesTemplatingEngine) {}
  
  async generateAllYAML(outputDir: string): Promise<void> {
    const resources = this.engine.getAllResources();
    await this.writeYAMLFiles(resources, outputDir);
  }
}

1.3 Update Client Interface

// Modified: clients/js/packages/client/src/client.ts
export class StarshipClient {
  // Add new methods
  async generateYAML(outputDir: string): Promise<void> {
    const generator = new YAMLGenerator(this.templatingEngine);
    await generator.generateAllYAML(outputDir);
  }
  
  async deployFromYAML(yamlDir: string): Promise<void> {
    const client = new KubernetesClient({ /* config */ });
    // Deploy resources using KubernetesJS
  }
}

Phase 2: KubernetesJS Integration (Weeks 3-4)

2.1 Add KubernetesJS Client

// New file: clients/js/packages/client/src/kubernetes/client.ts
import { KubernetesClient } from 'kubernetesjs';

export class StarshipKubernetesClient {
  private client: KubernetesClient;
  
  constructor(config: StarshipClientConfig) {
    this.client = new KubernetesClient({
      restEndpoint: config.kubernetesEndpoint || 'http://127.0.0.1:8001'
    });
  }
  
  async deployChain(chain: Chain): Promise<void> {
    const deployment = this.createDeployment(chain);
    await this.client.createAppsV1NamespacedDeployment({
      path: { namespace: this.namespace },
      body: deployment
    });
  }
}

2.2 Replace Shell Commands

// Replace shell-based operations
// Before:
this.exec(['kubectl', 'get', 'pods', ...this.getArgs()]);

// After:
const pods = await this.k8sClient.listCoreV1NamespacedPod({
  path: { namespace: this.namespace }
});

2.3 Update Dependency Management

// Modified: clients/js/packages/client/src/deps.ts
export const dependencies: Dependency[] = [
  {
    name: 'kubectl',
    url: 'https://kubernetes.io/docs/tasks/tools/',
    installed: !!shell.which('kubectl'),
    required: false // Optional for API access only
  },
  {
    name: 'docker',
    url: 'https://docs.docker.com/get-docker/',
    installed: !!shell.which('docker')
  }
  // Remove helm dependency
];

Phase 3: Advanced Features (Weeks 5-6)

3.1 Resource Monitoring

// New file: clients/js/packages/client/src/kubernetes/monitoring.ts
export class ResourceMonitor {
  async waitForPodsReady(namespace: string, timeout: number): Promise<void> {
    const watch = this.client.watchCoreV1NamespacedPod({
      path: { namespace },
      query: { watch: true }
    });
    
    // Implement pod readiness watching
  }
}

3.2 Port Forwarding via API

// Replace kubectl port-forward with API calls
export class PortForwardManager {
  async forwardPort(podName: string, localPort: number, remotePort: number): Promise<void> {
    // Use KubernetesJS port forwarding API
  }
}

Migration Benefits

Immediate Benefits

1. Better Error Handling: Structured error responses from Kubernetes API
2. Improved Debugging: Direct access to resource status and events
3. Platform Independence: No shell command dependencies
4. YAML Inspection: Users can review generated resources before deployment

Long-term Benefits

1. Enhanced Testing: Mock KubernetesJS client instead of shell commands
2. Advanced Features: Real-time resource monitoring, streaming logs
3. Better Integration: Easier to integrate with CI/CD and other tools
4. Reduced Dependencies: Eliminate Helm requirement

Breaking Changes

Configuration Changes

• New optional kubernetes.endpoint configuration
• Deprecated Helm-specific settings
• New YAML output directory settings

CLI Changes

# New commands
starship generate-yaml --config config.yaml --output ./k8s-yaml
starship deploy-yaml --yaml-dir ./k8s-yaml
starship deploy --use-k8s-client --config config.yaml

# Modified existing commands (backward compatible)
starship deploy --config config.yaml  # Still works with Helm

API Changes

• New generateYAML() method on StarshipClient
• New deployFromYAML() method
• Deprecated shell-based methods (with warnings)

Testing Strategy

Unit Tests

• Mock KubernetesJS client for testing
• Test YAML generation logic
• Test resource creation logic

Integration Tests

// Example test using provided KubernetesJS patterns
describe('Starship KubernetesJS Integration', () => {
  it('should deploy PostgreSQL using KubernetesJS', async () => {
    const client = new StarshipKubernetesClient(config);
    await client.deployChain(postgresChain);
    
    const deployment = await client.getDeployment('postgres-pgvector');
    expect(deployment.status?.readyReplicas).toBe(1);
  });
});

E2E Tests

• Full deployment cycle using KubernetesJS
• Comparison tests: Helm vs KubernetesJS output
• Performance benchmarks

Documentation Updates

User Guide Updates

1. New "YAML Generation" section
2. Updated installation guide (remove Helm requirement)
3. New troubleshooting guide for KubernetesJS-specific issues
4. Migration guide from Helm to KubernetesJS

Developer Documentation

1. KubernetesJS architecture overview
2. Templating system documentation
3. Contributing guide updates
4. API reference for new methods

Dependencies

New Dependencies

{
  "kubernetesjs": "^latest",
  "js-yaml": "^4.1.0"  // Already present
}

Removed Dependencies

• Helm (clients no longer need it installed)
• Reduced kubectl dependency (optional for advanced use cases)

Success Criteria

Phase 1 Success Criteria

• YAML generation produces valid Kubernetes manifests
• Generated YAML matches current Helm output (functional equivalence)
• No breaking changes to existing CLI commands
• Comprehensive test coverage for templating engine

Phase 2 Success Criteria

• KubernetesJS deployment works end-to-end
• All shell commands replaced with API calls
• Error handling improvements measurable
• Performance parity or improvement vs Helm

Final Success Criteria

• Zero shell dependencies for core functionality
• User can inspect YAML before deployment
• Improved error messages and debugging experience
• Smooth migration path for existing users
• Documentation fully updated

Risk Mitigation

Technical Risks

1. KubernetesJS API limitations: Mitigation - Prototype critical paths early
2. Performance regression: Mitigation - Benchmark each phase
3. Feature parity gaps: Mitigation - Comprehensive feature mapping exercise

User Experience Risks

1. Breaking changes: Mitigation - Maintain backward compatibility during transition
2. Learning curve: Mitigation - Comprehensive documentation and examples
3. Migration complexity: Mitigation - Automated migration tools where possible

Related Issues

• #XXX - Add YAML generation capability
• #XXX - Implement KubernetesJS client integration
• #XXX - Replace shell commands with API calls
• #XXX - Update documentation for Starship 2.0
• #XXX - Create migration guide from Helm to KubernetesJS

Timeline

• Week 1-2: Phase 1 - YAML Generation Foundation
• Week 3-4: Phase 2 - KubernetesJS Integration
• Week 5-6: Phase 3 - Advanced Features & Polish
• Week 7: Testing, Documentation, Release Preparation
• Week 8: Release and User Migration Support

---

This epic represents a fundamental architectural improvement that will make Starship more maintainable, debuggable, and powerful while providing better user experience through YAML inspection and improved error handling.