Reproducible ML experiments at scale
An ML experiment that ran for six hours can vanish into irreproducibility the moment a teammate asks for the exact notebook version, data snapshot, or parameter set used. The result: another eight-hour run to maybe recreate what already exists. Shopify's Search & Discovery team—which ranks millions of products across billions of queries—hit this wall frequently enough to build a fix. That fix is now Tangle, an open-source ML experimentation platform that the company says has saved more than a year of cumulative compute time since adoption.
Tangle originated as Cloud Pipeline Studio, a tool built before Shopify, and was adapted internally to handle commerce-scale experimentation. The platform is now publicly available, with the team citing both internal utility and broader community relevance as motivation for the release. Data scientists iterate faster, engineers maintain less bespoke pipeline code, and infrastructure costs drop through shared caching—benefits the team believes extend well beyond Shopify's walls.
Six failure modes in ML development
The problems Tangle targets are familiar to any team doing serious ML work:
- Engineers manually track which custom query was used at each experiment stage, leading to mistakes and slowdowns.
- Notebooks accumulate without structure or governance.
- Hours-long data preparation steps are repeated unnecessarily.
- Old results are effectively unrecreatable.
- Deployment takes longer than training.
- Team members cannot easily share or build on each other's work.
Most platforms address a subset of these issues. Tangle was designed to handle all of them in one system.
Visual pipelines wrapping any code
Tangle is a platform-agnostic experimentation system that lets teams assemble ML and data pipelines through a visual drag-and-drop interface, then execute them in cloud environments without local development setup. Unlike tools that specialize in either data processing or model training, Tangle's pipelines can chain both—plus unconventional steps like model deployment or human-judge labeling—into a single flow.
The core architectural ideas:
Platform agnostic
Tangle works with existing code written in any language and runs on any cloud provider or local infrastructure without requiring code changes.
Robust caching
Task executions and their artifacts are tracked, letting the system automatically reuse duplicate computations across all pipeline runs from all team members. Crucially, Tangle can reuse not only fully succeeded task executions but also ones still in progress.
Visual pipeline editing
Developers build pipelines through an interactive interface that renders the full data flow as a directed acyclic graph (DAG). Structure is visible immediately, with no need to parse notebook code.
Language-neutral architecture
Components wrap existing command-line programs that read and write files. Python, Shell, Javascript, C#, C++, Rust, Java, Go, R—any language capable of CLI execution—works without rewriting code to fit framework-specific constructs.
How Tangle is structured
Tangle's model follows a layered hierarchy:
- Component: A reusable specification (like a function definition) describing metadata, interface (inputs/outputs), and implementation—either a templated command line for a containerized CLI program or a graph of tasks.
- Task: A configured component instance with specified input arguments (like a function call).
- Execution: What happens when you submit a pipeline—each task runs and produces output artifacts.
- Graph: Connected tasks where outputs flow to downstream inputs.
- Pipeline: A root graph component whose implementation is a graph of tasks.
Components are plain-text YAML files, organized into libraries, indexed, and searchable. They can be safely loaded from GitHub, the web, or cloud storage. Versioning is independent and content-addressed, so users can pin exact versions, mix different versions in one pipeline for comparison, and share specific versions without dependency conflicts.
Components follow a contract: they take inputs, do one job, produce outputs, and run isolated from everything else. They are treated as pure functions—deterministic and side-effect free—which is what makes effective caching possible.
Content-based rather than lineage-based caching
Typical ML platforms cache based on lineage: when an upstream component changes, everything downstream re-executes. Tangle takes a different approach. Downstream components check the content hashes of their outputs and reuse cached results when those outputs haven't actually changed. The payoff can be dramatic.
The team reports a real-world case where a 10-hour pipeline finished in 20 minutes after only one component changed—everything else was served from cache with identical outputs. Because the cache operates globally across all users, three data scientists submitting experiments that share a preprocessing step get that step executed exactly once, with all three pipelines reusing the artifact—even if one run is still in flight when the others start.
File-based data flow and automatic orchestration
Tangle components communicate through file paths rather than in-memory objects. A producer writes to a local path; the system uploads it to storage (GCS, S3, or similar); a consumer reads from its own local path after the system retrieves the file transparently. Placeholders are replaced with actual locations at runtime, so components use standard file I/O while storage stays abstracted.
Pipeline submission triggers a managed execution sequence:
- Queue: Every task starts queued.
- Check dependencies: Upstream tasks must complete and input artifacts be available.
- Check cache: The execution cache key is computed and searched against succeeded and running executions.
- Execute or reuse: A cache hit reuses results; otherwise a container is launched in the cloud cluster.
- Monitor: Container status, logs, and execution state are tracked.
- Finalize: Output artifact metadata (size, hash, small values) is stored and downstream tasks are notified.
All of this happens without user intervention: submit, let the system orchestrate, then inspect results.
An optional type system by design
Tangle's type system exists to support tooling, not to enforce correctness. Components declare their I/O types (e.g. String, Float, JsonObject, ApacheParquet, TensorflowModel), but those names are arbitrary conventions—the system treats all artifact data as opaque blobs or strings, and consuming components validate their own inputs.
The rationale is deliberate:
- Openness: Anyone can define and share specialized types.
- Performance: No runtime validation overhead.
- Security: No centralized parsing creates attack surface.
- Flexibility: Version compatibility is handled without rigid schemas.
Building, monitoring, and iterating
The visual editor renders pipelines as interactive DAGs. Components are dragged from a library, outputs connected to inputs, parameters configured inline, and submission is one click. During a run, users track task status, inspect artifacts and logs, identify cached steps at a glance, and spot bottlenecks in real time.
Every run is preserved with full lineage, so fast iteration never comes at the cost of experimental history.
Running Tangle today
Tangle currently runs locally (via a Docker/Podman launcher) or on HuggingFace (via the HuggingFace Job launcher). Additional installation recipes are in progress.
The quickest start is HuggingFace: head to the platform and begin building. Creating a pipeline needs no registration, but executing one requires a HuggingFace account with a Pro subscription ($9/month).
For a first pipeline, either start from the sample XGBoost training pipeline or build from scratch:
- Drag components onto the canvas.
- Connect outputs to inputs.
- Configure task arguments.
- Submit the pipeline for execution (login required).
- Monitor the run in real time.
Running Tangle on HuggingFace or locally
Tangle is available as a shared multi-tenant service on HuggingFace Spaces. That deployment leans on HuggingFace infrastructure for storage, compute, and authentication. A central tenant database tracks user IDs, access tokens, and orchestrator configs, while each tenant gets its own SQLite database in the persistent storage of the main Space. Pipelines execute through each user’s own HuggingFace Jobs, with logs and output artifacts written back to that user’s private HuggingFace Dataset repo (user/tangle_data). The UI links directly to both the artifacts and the underlying Jobs.
Teams that want full control can duplicate the Tangle Space to their own HuggingFace account and supply an HF token to get a single-tenant instance. In that setup, the database lives in your own Space’s persistent storage. Cloning Tangle to an organization yields a single-tenant, multi-user deployment: team members can see one another’s pipeline runs and share an org-wide cache.
For local development, the prerequisites are Docker and uv, with the app code pulled from the GitHub repository. After cloning, start the backend with backend/start_local.sh on Linux or macOS, or backend\start_local.cmd on Windows. Once the terminal reports “start_local: Starting the orchestrator,” open http://localhost:8000 and use the “New Pipeline” button to begin.
Who benefits from Tangle
Tangle’s design targets both organization-wide adoption and individual engineer workflows.
Organization-wide advantages
- Automatic tracking and reproducibility: Every execution records the full lineage—graph structure, logs, artifact metadata, and metrics. Intermediate data is immutable and never overwritten, which makes experiments safe to share across teams. Any pipeline run can be cloned by a colleague to investigate failures, adjust parameters, and resubmit.
- Time and compute savings: Content-based caching reuses previously executed tasks automatically, cutting runtime and cloud spend.
- Component libraries: Teams curate shared libraries of reusable components, promoting consistent patterns and faster development.
- Accessible to non-engineers: Product managers and analysts can build and run pipelines without writing code or configuring a development environment.
For individual engineers
- Zero-friction tracking: Versioning and execution history are captured automatically—no manual logging.
- No manual caching: Data moves between transformations with intelligent reuse, so custom caching logic is unnecessary.
- Non-intrusive integration: Components wrap existing code as-is and can invoke any CLI program, language, or container.
- Composable knowledge: Each component behaves like a self-contained function rather than a framework. A 50-line component can replace parsing a long tutorial, and independent components avoid dependency conflicts. Different versions can even run side-by-side for comparison.
- Language interoperability: A single pipeline can connect Python, Java, Shell, Ruby, C++, and JavaScript components without compatibility layers.
Production experience at Shopify
Shopify’s Search & Discovery team runs Tangle as production ML infrastructure, processing millions of search queries across billions of products. Use cases include product ranking models over millions of SKUs, semantic search experiments at query scale, recommendation system training, and real-time feature engineering.
The measured results from that deployment include daily deployment of new ranking models by data scientists without waiting on infrastructure changes; full recreation of any experiment from six months ago in two clicks with complete artifact provenance; global caching eliminating thousands of redundant compute hours each month; and a shared component library that accelerates development and keeps patterns consistent.
Roadmap and community
Shopify runs Tangle at production scale today and is prioritizing future work based on community input. Planned items include direct support for major clouds (GCP already works but lacks deployment documentation), an expanded component library, and artifact visualization.
The project is open source. The team points to the GitHub repository for stars and issue tracking, the documentation site for usage details, and the GitHub Discussions area for community conversation. Contributions via pull requests are welcome.
