https://gabrielhumpire.github.io/tags/ml-in-production/ML in ProductionHugo Blox Builder (https://hugoblox.com)en-usMon, 15 Jan 2024 00:00:00 +0000https://gabrielhumpire.github.io/media/icon_hu_645fa481986063ef.pnghttps://gabrielhumpire.github.io/tags/ml-in-production/https://gabrielhumpire.github.io/post/project-management/Mon, 15 Jan 2024 00:00:00 +0000https://gabrielhumpire.github.io/post/project-management/<p>After 15 years working in computer vision - through a PhD, multiple industry roles, and a few startups - I&rsquo;ve shipped systems that went from whiteboard sketch to production deployment. This post is about what that journey actually looks like, and the lessons that only come from doing it end-to-end.</p> <h2 id="requirements-are-the-hardest-part">Requirements Are the Hardest Part</h2> <p>In research, the problem is given to you. In production, you have to discover it. &ldquo;Detect anomalies in CT scans&rdquo; is not a requirement. &ldquo;Flag studies with a sensitivity of at least 92% while keeping false positives below 5% per study, with results available within 3 minutes of scan acquisition, running on a single GPU workstation&rdquo; - that is a requirement.</p> <p>Every project I&rsquo;ve worked on has been redefined at least once after the first prototype. Building for changeability is more valuable than building for the original spec.</p> <h2 id="the-prototype-trap">The Prototype Trap</h2> <p>A prototype that achieves 94% accuracy on your held-out test set is not a product. The questions that matter:</p> <ul> <li>How does it perform on data from a different scanner model?</li> <li>What happens when the input is corrupted or incomplete?</li> <li>How does a clinician or operator actually interact with the output?</li> <li>What is the failure mode, and is it safe?</li> </ul> <p>Medical imaging taught me this faster than any other domain. A false negative in cancer screening is not an acceptable failure mode.</p> <h2 id="iteration-is-the-work">Iteration Is the Work</h2> <p>The drone delivery project I led at Embention is a good example. The first detection model worked well in controlled lab conditions. Outdoor lighting, vibration, variable altitude, and regulatory edge cases forced five major architecture revisions before we had something deployable. Each revision was cheaper than the last because we had built good evaluation tooling early.</p> <p>The 30% latency reduction we eventually achieved did not come from a clever algorithm - it came from profiling, identifying that the bottleneck was preprocessing, and rewriting that stage in C++ with CUDA.</p> <h2 id="deployment-is-a-feature">Deployment Is a Feature</h2> <p>Models don&rsquo;t deploy themselves. Containerisation (Docker), hardware-specific optimisation (TensorRT, ONNX), monitoring, and rollback procedures are engineering work that needs to be planned from day one - not bolted on at the end.</p> <p>The most valuable skill I have developed is being fluent in both the research side (what the model can learn) and the engineering side (what it will cost to run). That fluency is what makes end-to-end delivery possible.</p>