Viv Labs + Samsung

Senior Staff DevSecOps Engineer
Project Overview
Viv Labs was an intelligent assistant company formed by the creators of Siri. It used a dynamically evolving cognitive architecture based on natural language inputs from the user to create a plan, and leverage third party integrations to create never before seen user experiences that were truly revolutionary.

Samsung Acquired Viv Labs  on October 5th, 2016. Much of Viv's technology was integrated into the Samsung's Bixby voice assistant, which was deployed on over 400 million devices including phones, refrigerators, televisions, and more.
My Contributions
I was hired to grow the Viv technology platform from a single-cluster demo to a proper production system operating at a global scale on millions of devices, supporting dozens of languages. Viv built a collection of truly custom back-end AI services, web services, a developer tools app, a marketplace, and of course, Bixby client applications. At Viv, we believed that AI is akin to magic for our customers. To make that magic happen in a fast, secure, and delightful way no matter where in the world our customers traveled, I helped design and implement a one-of-a-kind global service mesh. I wrote custom deployment tools that went beyond what standard cluster scheduling software could provide. I solved problems of how to do zero downtime deployments without a user ever experiencing a blip in a conversation.

Viv's excellent culture afforded us a high commitment to the quality and craft of platform engineering. This meant "automating all the things" was our strategy to ensure rapid, reliable, repeatable deployments to all regions of multiple clouds. Infrastructure as code was a big part of daily life at Viv. As our technology scaled, so did our team. Ultimately we had operators in multiple countries, including South Korea, Germany, and India. It was my privilege to travel the world and train these teams on the code base and deployment strategies.
Going Global with IaC

A back-end that seamlessly serves AI content to 200 million devices in 190 countries supporting a dozen or so languages and countless third parties requires large infrastructure, with hundreds of clusters, thousands of compute instances, buckets for object storage, stateful data to manage, and secrets to protect.

I wrote a lot of Terraform. Call it 300,000 lines. This purely IaC based approach went a long way towards repeatable, reliable deployments, but it also gave me unforgettable experiences with HCL syntax changes, outdated providers, design patterns & anti-patterns, and all the ways one might get around Terraform's lack of DRY syntax (from workspaces to generators).

Terraform Logo
Service Matrix Diagram
Service Mesh with HashiCorp

To balance scalability, cost management, and management complexity, we started with homegrown service discovery and ultimately graduated to a global service mesh.

I implemented this mesh using technologies such as HashiCorp Console, Vault (mTLS), and Envoy.

I played a key role in solving the problem of facilitating seamless conversations during times when the back end may be in a degraded state.

Third Party Sandboxing & Security

Viv's third-party integrations and capsule architecture meant that we needed to make provisions for running untrusted code.

I worked with the dev team and red team leads to develop a threat model and sandbox system that used a defense-in-depth approach from layer 7 all the way down the stack.

I implemented layers of isolation at the container, cluster, and network. I built systems to automate static code analysis and other scanning tools used to evaluate code prior to deployment.

Sandboxed Application
Python Logo
Custom Dev Tool Development
I was primary contributor and tech lead for the internal DevOps tools. There were two major components:

A code library  that made up the heart of our CI/CD system and shifted complexity away from a clunky DSL  into a more universally maintainable language.

A Python CLI tool for rapid deployments, status checks, and troubleshooting with a UX in the style of awscli and git. It empowered globally distributed teams to check the correctness of deployments against service manifests, perform deployments while reducing the potential for human errors, and troubleshoot failed deployments faster than one could with industry-standard tooling.