Managing Development Artifacts in AI for Healthcare

In healthcare AI, difficulties often arise not from model design itself, but from a lack of traceability during development. When datasets, models, and evaluations are not consistently tracked, results become harder to reproduce, interpret, and justify.

For this reason, development artifacts should be treated as part of the system. Structured tracking from the beginning supports reproducibility, internal review, and later regulatory assessment.

1. Purpose and timing of tracking

A first question concerns the role of tracking and when it should start. In practice, its purpose is to ensure that every result can be linked to a specific configuration of data, code, and parameters.

This has several implications:

• reproducibility, by linking outcomes to exact inputs
• traceability, by documenting how models evolve
• comparability, by making experiments consistent over time
• error reduction, by avoiding ambiguity between runs

For example, if a model shows improved sensitivity, tracking allows identification of the source of change, whether it comes from data variation, parameter tuning, or evaluation differences.

From a practical perspective, this process should begin with the first experiment, not at a later stage. Delayed tracking often results in missing context and fragmented records once iterations accumulate.

2. What needs to be tracked

Once the objective is defined, the next step is to clarify what should be recorded. A useful structure is to group artifacts into three categories: data, models, and evaluation.

Data

Data documentation provides context for all downstream results. It typically includes origin, selection criteria, preprocessing steps, and known biases or limitations. In healthcare settings, this is also where representativeness and data quality considerations are captured.

Models

Model tracking focuses on the technical configuration used during training. This includes architecture, hyperparameters, training setup, and version history. These elements define how a given result was produced.

Evaluations

Evaluation records describe how performance is measured and under which conditions. This includes metrics, test populations, dataset versions, and experimental setup. Linking evaluations to specific datasets is essential for interpretation.

Together, these three elements form a complete view of the development process, from input data to final performance.

3. How tracking is implemented in practice

Once the structure is defined, tools help operationalize it across the workflow. Each layer addresses a different part of the development process, from data storage to deployment.

Dataset versioning tools such as DVC or Git LFS allow datasets to be stored and linked to specific code versions. This ensures that a given experiment can always be reproduced using the same data snapshot.

On the experimental side, platforms such as MLflow or Weights & Biases are commonly used to log parameters, metrics, and outputs. This enables systematic comparison between runs and helps identify the impact of incremental changes.

For model lifecycle management, registries such as MLflow Model Registry or Kubeflow support structured promotion of models between stages like development, staging, and production, while preserving full version history.

Documentation tools such as GitHub or GitLab complement this setup by capturing assumptions, design decisions, and known limitations alongside code. This helps maintain context that is not always reflected in metrics.

Finally, workflow orchestration tools such as Airflow or Prefect ensure that preprocessing, training, and evaluation steps remain consistent across experiments. This reduces variability caused by manual execution.

4. Practical workflow

These elements can be combined into a simple and reproducible workflow:

1. A dataset is versioned (e.g., dataset_v1.0 using DVC)
2. A model is trained and logged (model_v1.0 in MLflow)
3. Metrics and evaluation conditions are recorded
4. The dataset is updated (v1.1) and the model retrained (model_v1.1)
5. Results are compared across versions

This structure ensures that each outcome can be traced back to a specific combination of data and model configuration.

5. Common failure patterns

Without structured tracking, several issues tend to appear during development. Datasets and experiments may become mixed, models may lose linkage to training data, and metrics may be reported without sufficient context.

Another frequent limitation is reliance on informal notes instead of structured logging. This becomes particularly problematic when systems evolve or when results need to be reviewed externally.

6. Personal perspective

From a practical standpoint, tracking is not only a matter of organization. It directly affects how interpretable and maintainable a system becomes over time.

When datasets, models, and evaluations are consistently recorded, the development process becomes easier to reproduce, update, and communicate. It also reduces friction between engineering, quality, and regulatory functions.

In this sense, traceability is not an additional layer applied after development, but a structural element of how AI systems are built.