Knowledge Graphs and AI as a Service at AT&T
People & Blogs
Introduction
The intricate lifecycle of artificial intelligence (AI) consists of various stages, notably data discovery, cleaning, feature engineering, and ultimately, modeling and deployment. In the quest to enhance AI's efficiency and effectiveness, the potential of knowledge graphs in optimizing this lifecycle warrants exploration.
The Role of Knowledge Graphs
Knowledge graphs have become increasingly vital in assisting with data catalogs and also present a novel tool — feature stores. A feature store can be understood as a database with both offline and online components, designed to facilitate the discovery of relevant data features. The challenge arises from the sheer volume of features available, which can lead to analysis paralysis.
For instance, in AT&T's feature store, there are 179 feature sets encompassing around 7,000 features. For data scientists aiming to predict churn by analyzing subscriber data, finding relevant features among thousands can be daunting. Knowledge graphs can serve to streamline this process, bridging the gap between data discovery and model deployment.
Feature Store Implementation
AT&T launched its feature store in September of the previous year, quickly realizing significant improvements in model performance. In one instance, A 37% improvement in fraud detection models was achieved simply by identifying previously overlooked features. This showcases the importance of having an organized and searchable feature set.
In the world of data science, it can often be observed that multiple teams inadvertently recreate similar features. The challenge of redundant feature creation is significant, particularly in a large organization like AT&T with a workforce of around 250 data scientists. A knowledge graph acts as a collaborative hub, allowing data scientists to share and discover existing features, thus reducing redundancy.
Ontologies in Feature Discovery
The introduction of ontologies within feature stores enhances the ability to identify relevant features. By categorizing features and associating them with business concepts, data scientists can filter the large pool of features effectively. This enables them to rapidly identify unique features that could enhance their models.
For instance, a data scientist looking at features related to broadband fiber churn could use ontologies to sift through 7,000 features and isolate those that are truly relevant within minutes.
Furthermore, knowledge graphs assist in recognizing similarities among features. Without standardized terminology — especially in long-established companies — features may have varied naming conventions, making it difficult to identify duplicates. By employing knowledge graphs, data scientists can streamline comparative analysis and distinguish truly unique features.
Automation and Documentation
In addition to simplifying feature discovery, knowledge graphs also automate the labeling and documentation process. An example provided in the discussion referred to utilizing NLP tools to analyze video content from data scientists, effectively extracting semantics to improve feature descriptions. This harnesses existing content to keep feature documentation relevant and up to date.
Moreover, the graph structure allows for real-time monitoring and alerts, potentially safeguarding against data privacy issues. An example is the incident related to the anonymized New York City taxi data, which could have been mitigated by a knowledge graph that identifies intersections in data sets that might lead to de-anonymization.
Future Directions
The vision for AI at AT&T involves not just responding to present challenges but also actively enhancing the data science workflows through knowledge graphs and context-aware AI systems. Moving forward, the capability to run entire AI processes through a knowledge graph may pave the way for even more sophisticated AI applications.
Keywords
- Knowledge Graphs
- Artificial Intelligence (AI)
- Feature Store
- Data Discovery
- Ontologies
- Data Science
- Model Improvement
- Data Catalog
- NLP Tools
- Anonymization
FAQ
Q1: What is a Feature Store?
A Feature Store is a database that serves as a centralized repository of features used in machine learning models. It helps data scientists find and utilize relevant features more efficiently.
Q2: How can Knowledge Graphs improve AI workflows?
Knowledge Graphs can streamline the data discovery process, facilitate feature sharing, reduce redundancy, enhance documentation, and monitor data privacy compliance, all of which contribute to faster and better AI model development.
Q3: What challenges do large organizations face with feature management?
Large organizations often struggle with redundancy, as different teams may create similar features without awareness of each other’s work. This can lead to inefficient use of resources and delayed model development.
Q4: Are there any privacy concerns associated with AI and Knowledge Graphs?
Yes, there are privacy concerns, especially regarding data anonymization. Knowledge graphs can help monitor overlap between data sets, which may lead to unintended de-anonymization, ensuring more secure data practices.
Q5: How do ontologies function in Feature Discovery?
Ontologies organize features into categories based on business concepts, allowing data scientists to filter through large sets of features efficiently, helping them locate relevant features quickly.
