Concurrent Multiverse¶

Quantum Multiverse Warp Test¶

Overview¶

This demo demonstrates concurrent multi-threaded package version testing across three different Python interpreters simultaneously. Unlike the Multiverse Healing demo that runs sequentially, this test proves OmniPkg can safely orchestrate parallel workflows where multiple threads simultaneously swap Python versions, install different package versions, and verify isolated execution—all without conflicts or race conditions.

The “Quantum Warp” name reflects the seemingly impossible feat: running Python 3.9, 3.10, and 3.11 at the same time in separate threads, each with different versions of the same package (Rich), all coordinated by a single orchestrator process.

Usage¶

For local execution, you can run the demo directly:

# Run Demo #8: Quantum Multiverse Warp
omnipkg demo 8

If you have the OmniPkg web bridge connected, you can run it live in the UI:

omnipkg demo 8

The demo requires Python 3.11 as the orchestrator and will automatically switch to it if needed.

What You’ll Learn¶

1. Concurrent Python Management: How to safely run multiple Python versions in parallel threads
2. Thread-Safe Context Switching: How OmniPkg’s locking prevents race conditions during version swaps
3. Isolated Package Bubbles: How different threads can use different package versions simultaneously
4. Performance Optimization: How parallel execution reduces total test time vs sequential approaches
5. Lock Contention Analysis: Understanding wait times and critical section performance

The Architecture¶

This test uses a sophisticated multi-threaded workflow with proper synchronization:

┌─────────────────────────────────────────────────────────────────┐
│  ORCHESTRATOR (Python 3.11)                                     │
│  └─> Spawns 3 concurrent worker threads                         │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ├──────────────────┬──────────────────┐
                              ▼                  ▼                  ▼
                    ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
                    │   THREAD 1      │ │   THREAD 2      │ │   THREAD 3      │
                    │   Python 3.9    │ │   Python 3.10   │ │   Python 3.11   │
                    │   Rich 13.4.2   │ │   Rich 13.6.0   │ │   Rich 13.7.1   │
                    └─────────────────┘ └─────────────────┘ └─────────────────┘
                            │                    │                    │
                            └────────────────────┴────────────────────┘
                                              │
                                    ┌─────────────────────┐
                                    │  Shared Lock        │
                                    │  Prevents conflicts │
                                    └─────────────────────┘

The Code Structure¶

The test script has several key components:

1. Thread-Safe Execution Helpers¶

import threading

print_lock = threading.Lock()  # Prevents garbled output
omnipkg_lock = threading.Lock()  # Prevents version swap conflicts

def safe_print(msg: str):
    """Thread-safe console output."""
    with print_lock:
        print(msg, flush=True)

2. Controlled Package Operations¶

def run_and_stream_install(python_exe: str, args: list, thread_id: int):
    """Runs omnipkg install and streams output with thread prefix."""
    prefix = f"[T{thread_id}]"
    safe_print(f"{prefix} 📦 Installing {' '.join(args[1:])}")

    cmd = [python_exe, "-m", "omnipkg.cli"] + args
    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, ...)

    # Stream output with thread identifier
    for line in iter(process.stdout.readline, ""):
        safe_print(f"  {prefix}|install | {line.strip()}")

    return process.wait()

3. Test Dimension Execution¶

Each thread runs this isolated test:

def test_dimension(config: tuple, thread_id: int) -> dict:
    """Test one Python+Rich combination."""
    py_version, rich_version = config
    prefix = f"[T{thread_id}]"

    # Get interpreter path
    python_exe = get_interpreter_path(py_version)

    safe_print(f"{prefix} ⏳ Waiting for lock...")

    # Critical section: only one thread can swap/install at a time
    with omnipkg_lock:
        safe_print(f"{prefix} 🔒 LOCK ACQUIRED")

        # Swap to target Python version
        run_omnipkg_cli(sys.executable, ["swap", "python", py_version], thread_id)

        # Install specific Rich version
        run_and_stream_install(python_exe, ["install", f"rich=={rich_version}"], thread_id)

        safe_print(f"{prefix} 🔓 LOCK RELEASED")

    # Verification happens OUTSIDE the lock (allows parallelism)
    test_script = f"""
from omnipkg.loader import omnipkgLoader
with omnipkgLoader("rich=={rich_version}"):
    import rich, importlib.metadata
    actual_version = importlib.metadata.version('rich')
    print(json.dumps({"success": True, "rich_version": actual_version}))
"""

    result = subprocess.run([python_exe, "-c", test_script], ...)
    return parse_and_verify_result(result)

4. Concurrent Orchestration¶

def main():
    """Main test orchestrator."""
    test_configs = [
        ("3.9", "13.4.2"),   # Thread 1
        ("3.10", "13.6.0"),  # Thread 2
        ("3.11", "13.7.1"),  # Thread 3
    ]

    # Phase 1: Adopt all interpreters sequentially (safer)
    for version, _ in test_configs:
        adopt_if_needed(version, 0)

    # Phase 2: Run tests concurrently
    results = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
        futures = {
            executor.submit(test_dimension, config, i + 1): config
            for i, config in enumerate(test_configs)
        }

        for future in concurrent.futures.as_completed(futures):
            result = future.result()
            if result:
                results.append(result)

    print_summary(results)

Live Execution Log¶

The test produces detailed output showing concurrent execution:

====================================================================================================
🚀 CONCURRENT RICH MULTIVERSE TEST
====================================================================================================

📥 Phase 1: Adopting interpreters (sequential for safety)...
[T0|Adopt] ✅ Python 3.9 already available.
[T0|Adopt] ✅ Python 3.10 already available.
[T0|Adopt] ✅ Python 3.11 already available.

✅ All interpreters ready. Starting concurrent tests...

[T1] 🚀 Testing Python 3.9 with Rich 13.4.2
[T2] 🚀 Testing Python 3.10 with Rich 13.6.0
[T3] 🚀 Testing Python 3.11 with Rich 13.7.1
[T3] 📍 Using: /home/minds3t/.../python3.11
[T3] ⏳ Waiting for lock...
[T3] 🔒 LOCK ACQUIRED
[T3] 🔄 Swapping to Python 3.11
[T2] 📍 Using: /home/minds3t/.../python3.10
[T2] ⏳ Waiting for lock...
[T1] 📍 Using: /home/minds3t/.../python3.9
[T1] ⏳ Waiting for lock...
[T3] ✅ swap python (384.6ms)
[T3] 📦 Installing rich==13.7.1 (Live Output Below)
  [T3]|install | ⚡ PREFLIGHT SUCCESS: All 1 package(s) already satisfied!
[T3] ✅ Install finished in 471.4ms with code 0
[T3] 🔓 LOCK RELEASED
[T3] 🧪 Testing Rich import...
[T2] 🔒 LOCK ACQUIRED
[T2] 🔄 Swapping to Python 3.10
[T3] ✅ VERIFIED:
[T3]    Python: 3.11.14
[T3]    Rich: 13.7.1 (from .../rich-13.7.1/rich/__init__.py)
[T2] ✅ swap python (492.2ms)
[T2] 📦 Installing rich==13.6.0
[T2] ✅ Install finished in 328.5ms with code 0
[T2] 🔓 LOCK RELEASED
[T2] 🧪 Testing Rich import...
[T1] 🔒 LOCK ACQUIRED
...

====================================================================================================
📊 DETAILED RESULTS
====================================================================================================
Thread   Python       Rich       Wait     Swap     Install    Test     Total     
----------------------------------------------------------------------------------------------------
T1       3.9.23       13.4.2     1.67s    491.5ms  276.2ms    170.1ms  3.04s     
T2       3.10.18      13.6.0     847.2ms  492.2ms  328.5ms    196.9ms  2.30s     
T3       3.11.14      13.7.1     1.3µs    384.6ms  471.4ms    250.5ms  1.53s     
----------------------------------------------------------------------------------------------------
⏱️  Total concurrent runtime: 4.23s
====================================================================================================

🔍 VERIFICATION - Actual Python Executables Used:
----------------------------------------------------------------------------------------------------
T1: .../cpython-3.9.23/bin/python3.9
     └─ Rich loaded from: .../rich-13.4.2/rich/__init__.py
T2: .../cpython-3.10.18/bin/python3.10
     └─ Rich loaded from: .../rich-13.6.0/rich/__init__.py
T3: .../cpython-3.11.14/bin/python3.11
     └─ Rich loaded from: .../rich-13.7.1/rich/__init__.py
----------------------------------------------------------------------------------------------------

🎉 ALL TESTS PASSED!

Key Features Demonstrated¶

1. Lock-Based Synchronization¶

The critical section (swap + install) is protected by a shared lock:

• Thread 3 acquires lock first (1.3µs wait - essentially instant)
• Thread 2 waits 847ms for Thread 3 to finish
• Thread 1 waits 1.67s for Threads 3 and 2 to finish

This serialization prevents conflicts when modifying OmniPkg’s shared state.

2. Parallel Verification¶

The verification phase runs outside the lock, allowing true parallelism:

• While Thread 1 is installing, Thread 3 is already verifying
• This optimization reduces total runtime from ~9s (sequential) to ~4s (concurrent)

3. Version Bubble Isolation¶

Each thread successfully loads a different Rich version:

• Thread 1: Rich 13.4.2 in Python 3.9 bubble
• Thread 2: Rich 13.6.0 in Python 3.10 bubble
• Thread 3: Rich 13.7.1 in Python 3.11 bubble

All three versions coexist simultaneously in memory without conflicts.

4. Thread-Safe Output¶

The safe_print function ensures output remains readable:

[T1] 🚀 Testing Python 3.9 with Rich 13.4.2
[T2] 🚀 Testing Python 3.10 with Rich 13.6.0
[T3] 🚀 Testing Python 3.11 with Rich 13.7.1

Without locking, this would produce garbled text.

How It Works¶

1. Bootstrap: Ensures orchestrator runs in Python 3.11
2. Adoption Phase: Sequentially adopts Python 3.9, 3.10, 3.11 if needed (thread-safe)
3. Thread Spawn: Creates 3 worker threads with ThreadPoolExecutor
4. Lock Acquisition: Each thread waits for exclusive access to swap/install
5. Context Switch: Thread swaps active Python version in OmniPkg config
6. Package Install: Thread installs specific Rich version for that Python
7. Lock Release: Thread releases lock, allowing next thread to proceed
8. Parallel Verify: Thread verifies package load (happens concurrently)
9. Result Collection: Orchestrator aggregates results and prints summary

Performance Analysis¶

Sequential vs Concurrent Comparison¶

Speedup: 2.1x faster due to overlapping verification phases.

Lock Contention Breakdown¶

The long wait times are expected—OmniPkg’s config must be modified atomically.

Common Pitfalls Avoided¶

Problem: Race Conditions on Config Files¶

Solution: The omnipkg_lock ensures only one thread modifies config at a time.

Problem: Garbled Console Output¶

Solution: The print_lock serializes all print statements.

Problem: Import Conflicts Between Threads¶

Solution: Each subprocess runs in isolated memory with its own bubble context.

Problem: Recursive Interpreter Adoption¶

Solution: Adoption phase uses verify_registry_contains() checks with locking.

Real-World Applications¶

This concurrent testing pattern enables several practical use cases:

1. CI/CD Matrix Testing: Test packages across multiple Python versions in parallel
2. Compatibility Validation: Verify library works with different dependency versions simultaneously
3. Performance Benchmarking: Compare execution speed across Python versions concurrently
4. Integration Testing: Run multi-service tests where each service uses different Python/packages
5. Rapid Prototyping: Experiment with multiple configurations simultaneously

Comparison to Traditional Approaches¶

OmniPkg achieves Docker-level isolation with minimal overhead, making it ideal for rapid local testing.

Technical Deep Dive¶

Why Lock During Swap+Install?¶

OmniPkg maintains shared state in: - config.json (active Python version) - pyproject.toml (dependency specifications) - Symlinks in .omnipkg/bin/

Without locking, concurrent swaps would corrupt these files.

Why NOT Lock During Verification?¶

Verification only reads from bubble directories and doesn’t modify shared state. Each subprocess: - Has its own sys.path - Loads from isolated bubble directories - Doesn’t touch OmniPkg config

This allows true parallel execution during the verification phase.

The omnipkgLoader Context Manager¶

with omnipkgLoader("rich==13.7.1"):
    import rich  # Loads from bubble, not main env

The loader temporarily modifies sys.path to prioritize the bubble, then restores it on exit. This is thread-safe because each subprocess has independent memory.

Debugging Tips¶

If the test fails, check:

1. Lock deadlocks: Increase logging in lock acquisition
2. Version conflicts: Verify bubbles exist in .omnipkg_versions/
3. Import errors: Check sys.path in verification subprocess
4. Race conditions: Add more safe_print statements to trace execution

Extending the Demo¶

You can modify this test to:

• Add more threads (test 5+ Python versions concurrently)
• Test different packages (NumPy, TensorFlow, etc.)
• Benchmark performance scaling (measure speedup vs thread count)
• Add failure injection (test error handling under contention)

Conclusion¶

The Quantum Multiverse Warp demo proves OmniPkg can safely orchestrate complex concurrent workflows across multiple Python versions. The combination of thread-safe locking, isolated bubbles, and parallel verification makes it practical to run sophisticated multi-version testing on a single machine—without Docker, VMs, or complex CI configurations.

This is the ultimate stress test for OmniPkg’s version management system, demonstrating it can handle real-world scenarios where multiple developers or CI jobs might be testing different configurations simultaneously.