Design an AutoML Service for Non-Technical Users

Problem statement

Design an AutoML service where a non-technical user uploads a CSV or connects a database, selects a target column, and the platform automatically profiles the data, engineers features, searches over algorithms and hyperparameters (trees, linear models, neural nets, ensembles), ranks candidate pipelines on a protected validation protocol, and delivers a trained model as a managed inference endpoint or an exportable code artifact.

The naive framing is "run many training jobs and pick the best score." The production framing is harder: the platform is a multi-tenant black-box optimization service wrapped around a statistical correctness contract and a supply chain of immutable artifacts. Three failure classes dominate real deployments. First, compute waste: naive grid or exhaustive search spends most of its budget on configurations that are obviously bad after 20% of training, which is exactly what multi-fidelity early stopping (Successive Halving, Hyperband, ASHA) and Bayesian sampling (TPE, Gaussian-process acquisition) exist to eliminate. Second, statistical malpractice: automated feature engineering that fits transforms on the full dataset leaks the target into training and produces leaderboard scores that collapse in production. Third, irreproducibility: a "best model" that cannot be re-created from a pinned dataset digest, feature-pipeline version, library bundle, and seed is an audit liability, not an asset.

The five planes

1. Ingestion plane: upload, schema inference, profiling, PII scan, immutable dataset artifact.
2. Search plane: trial suggestion (sampler), trial pruning (pruner), budget governance, fair-share scheduler.
3. Training plane: elastic worker fleet (CPU/GPU, spot), checkpointing, preemption recovery, metric streaming.
4. Selection and serving plane: protected holdout, ensemble construction, model registry, endpoint or code export, drift monitoring.
5. Governance plane: lineage, model cards, cost accounting, tenant isolation, retention and deletion.

A strong interview answer keeps the search plane and the statistical protocol separate: the scheduler optimizes compute, the validation protocol optimizes truth, and neither is allowed to overrule the other.

Public baseline versus design assumptions

Managed AutoML is operationally real: Google documents Vertex AI AutoML with explicit node-hour train budgets; AWS documents SageMaker Autopilot with configurable maximum candidates and per-job runtime; Microsoft documents Azure Automated ML with early-termination policies and automatic ensembling; H2O.ai and DataRobot sell genetic-feature-evolution and blueprint-library AutoML respectively. Open-source orchestration references (Optuna's TPE sampler and ASHA pruner, Ray Tune's distributed schedulers) prove the search-loop architecture at fleet scale. Unless a number below is tied to such a public source, it is an explicit design assumption for this answer.