Deployed Shared Open WebUI Instance in Cloud Foundry

Overview

In our journey to democratize AI across the organization, we successfully deployed a shared Open WebUI instance on Cloud Foundry, integrated with our enterprise AI infrastructure. This initiative has empowered multiple teams to leverage AI capabilities without the complexity of individual setups.

The Challenge

Before this deployment, teams faced several challenges:

• Fragmented AI Access: Different teams used different AI tools and interfaces
• Inconsistent Experience: Lack of standardization across the organization
• Security Concerns: Managing API keys and access control individually
• Resource Duplication: Multiple teams setting up similar infrastructure

Solution Architecture

Cloud Foundry Deployment

We chose Cloud Foundry as our platform for several key reasons:

• Scalability: Built-in auto-scaling to handle varying loads
• Multi-tenancy: Secure isolation between different team workspaces
• Enterprise Integration: Seamless connection to existing SSO and security infrastructure
• Easy Management: Push-to-deploy workflow simplified operations

Open WebUI Integration

Open WebUI provided the perfect interface for our needs:

• User-Friendly: Intuitive chat interface familiar to all users
• Model Agnostic: Support for multiple AI providers and models
• Customizable: Ability to configure prompts, models, and features per team
• Open Source: Full control and transparency

Enterprise AI Connection

The integration with our enterprise AI infrastructure included:

• Centralized API Gateway: Single point of access to multiple AI models
• Usage Tracking: Monitoring and analytics for cost management
• Access Control: Role-based permissions tied to corporate identity
• Audit Logging: Complete traceability for compliance

Implementation Highlights

1. Cloud Foundry Configuration

# manifest.yml
applications:
- name: open-webui
  memory: 2G
  instances: 2
  buildpacks:
    - nodejs_buildpack
  env:
    OLLAMA_API_BASE_URL: ${AI_GATEWAY_URL}
    ENABLE_OAUTH: true
    OAUTH_PROVIDER: enterprise-sso
  services:
    - postgres-db

2. Security Setup

• SSO Integration: Corporate single sign-on for seamless authentication
• API Key Management: Secure vault integration for credential storage
• Network Policies: Restricted access to internal AI endpoints only
• Data Encryption: End-to-end encryption for all conversations

3. Multi-Team Configuration

Each team received:

• Dedicated workspace with custom branding
• Pre-configured model access based on team needs
• Custom prompt templates for common use cases
• Usage quotas and rate limits

Impact and Results

The deployment has had significant positive impact:

Adoption Metrics

• 50+ Teams actively using the platform within first month
• 1000+ Users onboarded across the organization
• 10,000+ Conversations generated in the first quarter
• 95% User Satisfaction based on internal surveys

Business Benefits

• Reduced Setup Time: From days to minutes for new users
• Cost Optimization: Centralized billing and usage tracking
• Knowledge Sharing: Shared prompt libraries across teams
• Innovation Acceleration: Faster prototyping and experimentation

Technical Wins

• High Availability: 99.9% uptime since launch
• Auto-Scaling: Seamless handling of peak usage periods
• Fast Response Times: <2s average API response time
• Easy Updates: Zero-downtime deployments via blue-green strategy

Key Learnings

What Worked Well

1. Start Small, Scale Fast: We piloted with 3 teams before organization-wide rollout
2. User Training: Comprehensive onboarding materials and workshops were crucial
3. Feedback Loop: Regular check-ins with early adopters shaped the platform
4. Documentation: Clear guides for common use cases reduced support burden

Challenges Overcome

1. Rate Limiting: Implemented intelligent queuing to handle burst traffic
2. Model Selection: Created decision tree to help teams choose appropriate models
3. Cost Management: Built dashboards showing real-time usage and costs per team
4. Compliance: Worked closely with legal to ensure data handling met requirements

Future Roadmap

We’re continuously improving the platform with:

• Fine-tuned Models: Organization-specific models trained on internal data
• Advanced Analytics: Better insights into AI usage patterns and ROI
• Integration Hub: Connect with existing tools (Slack, Teams, Jira)
• Custom Plugins: Team-specific extensions and workflows
• Multi-Cloud: Expanding beyond Cloud Foundry to other platforms

Conclusion

Deploying Open WebUI on Cloud Foundry has been a transformative initiative for our organization. By providing a centralized, secure, and user-friendly interface to enterprise AI capabilities, we’ve empowered teams to innovate faster and work more efficiently.

The combination of Cloud Foundry’s robust platform capabilities and Open WebUI’s flexible interface has proven to be a winning formula for democratizing AI across the enterprise.

Tech Stack Used:

• Cloud Foundry
• Open WebUI
• PostgreSQL
• API Gateway
• Monitoring & Logging Tools

Want to learn more? Feel free to reach out if you’re considering a similar implementation in your organization!