Data integration for AI application development
Designing an intuitive, no-code application to help users integrate their data on the C3.AI platform
Background
C3.AI is an enterprise AI software company that is committed to accelerating digital transformation in various industries ranging from manufacturing, utilities and finance. C3.AI offers both SAAS (software as a service) and PAAS (Platform as a Service) applications to enable engineers to rapidly develop, deploy and operate large-scale AI applications.
However, AI development often takes a highly skilled team of developers and data scientists. At the time, C3.AI's existing PAAS legacy product called Integrated Development Studio (IDS) was riddled with UX problems and built in a piecemeal fashion. Our goal was to enable less technical users to take advantage of IDS to create AI applications.
I was the sole product designer dedicated to the data integration vertical, focused on allowing users to easily upload and use their data, working with the 1 general UX researcher, 2 product managers for 6+ months from Jan 2021 to June 2021 to redesign IDS. The product has since been launched.
Building an AI application is incredibly complex. How might we make it easier for more people to use their data on C3.AI's platform to solve problems?
Research
In my research I wanted to understand:
1. How customers currently onboarded their data into C3 and identify challenges and areas of opportunities
2. Analyze the competitive space and what other application development and data integration tools did. Where did they succeed and fail?
3. Gain clarity on the future user personas for the redesign
I conducted competitive research on 5+ applications including Microsoft Azure Data Factory and Amazon App Flow. I also interviewed 6 current users of the legacy product in various data-related roles including a forward deployment engineer, data integration engineer, and a SaaS product manager. I also looked at existing documentation on the legacy platform, training videos on C3.AI's unified, model-driven architecture and customer projects. Previous user research done by a dedicated UX researcher was used clarify our target user persona.
Key research insights
Citizen developers lack the technical skills for data integration
Both citizen developers (i.e., business oriented users) and technical users are key to the application development process. Those with technical skills prefer using code to load and map data because it's robust and flexible. However, citizen developers do not have this option and thus are reliant on their engineering team for data loading.
The data mapping process is time consuming and manual
One of the core pain points in customers transferring their data onto the C3 platform was their ability to accurately and efficiently map their own data to the unified data model C3 provided. The process took months, was error prone, required various stakeholders with business knowledge and data engineering skills, and took place over multiple meetings and workshops.
Data errors are common and cause significant delay
Data from customers often suffer data quality issues including missing or mislabeled data. A variety of validation checks are required through out the integration process to ensure that there is a complete and connected dataset.
Lack of visibility on operational tasks
Many users wanted someway to monitor their tasks such as data loading in order to get alerted if there was some delay or error or know time till completion.
"Data validation is a time sink. It can take months somethings to resolve discrepancies."
Design
Ideation & Exploration
I held workshops with the team to explore possible solutions to the challenges we identified.
Mid-fi Validation
Greyscale mid-fidelity mockups were created to test the general data integration user story and the solutions we brainstormed. This helped create buy-in and alignment from the broader organisation.
Testing & Iteration
Quick prototypes were created to get rapid feedback from users through remote user interviews. This helped refine both the data integration flows as well as different UX approaches.
Solutions
GUI based tool for app development
The graphical interface provided an easy-to- understand visualisation of all the different elements that make up the application including data sources and metrics and their connections at any time in the development cycle.
Additionally, the platform provided a way to switch from no-code to high-code, increasing collaboration between these 2 user types.
Intuitive and automated data mapping
Data fields were intelligently mapped upon upload and an easy drag-and drop mechanism was provided for low-technical users to manually map their data.
Real time validation checks and data previews
Users could preview the data at upload, mapping and when performing transformations to ensure accurate mapping and aid in checking for data quality. An additional transformation library with intellisense was provided to aid the user in writing pseudocode.
Empowering users with operational tasks
Users could monitor their data loading, errors and receive alerts to any issues.
Launch
The project completed specification in 2021 and the redesign of IDS was prioritised to be built and launched in staged releases in 2022.
Learnings
I learned that for a large project such as this that clear communication, facilitated by visual assets, data and user research, was key to creating alignment between many different stakeholders. This was especially important during the lockdown of COVID-19.