Data is changing fast, and businesses must keep up. A recent Dataversity webinar with Donna Burbank (Global Data Strategy) explored important trends in data architecture. Here are the key takeaways.
With AI growing, organizations need real-time data processing to make quick decisions. As Lucha, CTO of GridGain, said:
"With the proliferation of AI and a surge in the development and use of AI agents, the need for data processing at Del latencies is increasing—and it's increasing exponentially by the day."
How businesses can optimize real-time data processing:
Consolidate computing components into a distributed in-memory compute grid to reduce latency.
Use distributed low-latency, in-memory caching to speed up access.
Minimize data movement by integrating compute & storage layers.
A unified approach ensures faster and more efficient decision-making.
AI is powerful, but without proper data governance, it can be risky. A webinar survey respondent noted:
"AI adoption is intensifying the need for data management and governance because this becomes even more critical for ethical, transparent, and reliable AI use."
Key takeaways:
Move beyond AI hype and focus on real-world applications.
AI models need high-quality, reliable data.
Data governance should involve both business and technical teams.
Without strong governance, AI could lead to bias, compliance risks, and poor decisions.
Building a strong data architecture isn't just a technical job—it requires teamwork across IT and business teams.
As Donna Burbank said:
"You should be designing your architecture for the business, not just an academic or technical view."
Best practices for collaboration:
Use clear, business-friendly language.
Leverage visual tools like data models and architecture diagrams.
Encourage shared responsibility for data.
Successful data initiatives connect IT and business for real impact.
A well-defined data architecture leads to:
Stronger collaboration between IT and business.
Higher productivity and cost efficiency.
Better AI and analytics outcomes.
While many roles contribute to data management, data governance leads play a crucial role in ensuring accountability.
Data storage and management are evolving. Here are the key trends:
Relational databases remain dominant, but cloud-based and non-relational databases are growing.
Spreadsheets are still widely used, though not ideal for enterprise storage.
Best-of-breed solutions are becoming common instead of one-size-fits-all platforms.
Data virtualization helps unify sources but isn’t a substitute for governance.
In the next 1-2 years, the focus will be on governance, strategy, quality, and architecture—foundations for successful AI and analytics.
Data architecture is evolving rapidly. Organizations should:
Embrace real-time processing to stay competitive.
Invest in governance for ethical AI and data management.
Foster collaboration between IT and business teams.
Strengthen data foundations for long-term success.
As AI and analytics advance, solid data architecture will be key to unlocking their full potential.
Key Themes and Ideas
Real-Time Data Processing is Crucial:
The increasing use of AI agents and the need for rapid decision-making demand real-time data processing capabilities. As Lucha, CTO of GridGain, stated, "with the proliferation of AI and a surge in the development and use of AI agents, the need for data processing at Del latencies is increasing and its increasing exponentially by the day."
To make informed decisions in real-time, organizations need all relevant data, requiring efficient processing of event streams, transactions, and data from various silos.
A crucial step is to consolidate computing components into a distributed in-memory compute grid to minimize latency and enable horizontal scaling. This includes combining stream processing, feature extraction, and model execution.
A further optimization involves placing data hubs and stores into a distributed low-latency, in-memory cache. This dramatically reduces input/output (I/O) latency, speeding up data access.
The ultimate aim is to combine the compute and storage layers into a single platform (or a tightly integrated technology stack) to eliminate data movement over the network, which is a significant source of latency.
This results in a consolidated data architecture, optimizing data movement and minimizing latencies for real-time decision-making. The alternative to a single platform is, as Lucha mentions, "a couple of different components that come together or a couple of different technologies that come together...as long as these Technologies are tightly integrated."
AI Requires Strong Data Management & Governance:
While AI offers tremendous potential, the webinar highlights the critical need for data management and governance to ensure ethical, transparent, and reliable use of AI. A survey respondent noted that as organizations adopt AI, "it's really intensifying the need for data management and the need for data governance because this becomes even more critical for the ethical transparent reliable use of AI."
Concerns around the ethical implications and data quality in AI implementation are paramount. There's a need to move beyond hype to focus on the practical value of AI while mitigating risks.
This means organizations must integrate business process with data management. Data governance should involve both technical and business people to properly use and safeguard data. This integration is vital to drive meaningful outcomes from data, rather than focusing solely on advanced tools and capabilities.
The intersection of data with business process needs to be considered in order to implement AI effectively.
The Importance of Collaboration and Communication:
Data management is not solely a technical endeavor. It requires collaboration between IT and business stakeholders.
The data architecture must be designed for the business needs, with a focus on the language and understanding of business stakeholders. As Donna noted: "we often do that all the time and just that difference of the language you use and the tools you use will make your architecture sing and it will help your architecture because you should be designing your architecture for the business not just an academic or technical view."
Data models, system architecture diagrams, and data flow diagrams are vital tools for facilitating communication and collaboration.
Presenting data models from a business perspective and using business terminology ensures that all stakeholders, technical or non-technical, understand and engage with data and architecture designs.
A core tenet of the presentation is that it really does "take a village" to implement data management effectively and that there are many different types of roles that must be involved.
Data Architecture is a Foundational Element:
A defined data architecture is essential for deriving value from data initiatives. Having this has resulted in "collaboration with IT and collaboration with the business."
Data architecture helps improve collaboration, increase productivity, and reduce costs. Having a well-defined data architecture was shown to greatly benefit collaboration between technical teams and business units.
Data Architects and data governance leads, despite having different skill sets, must work closely together.
While many roles are involved in data management, the data governance lead is often the leader in driving the necessary change and accountability within an organization.
Data Platforms and Future Trends:
Relational databases remain the most widely used data platform, but cloud-based storage and non-relational databases are growing in adoption.
Spreadsheets are still a major data source, indicating a need for improved data governance and tools. Donna calls this out by stating, "...they are a terrible source for your Enterprise data storage..."
Organizations are planning to adopt a mix of data platforms, including cloud object storage and non-relational databases, showcasing the growing complexity of data ecosystems.
Architectures are becoming ecosystems of best-of-breed solutions. Organizations need to adopt a use-case-driven approach in selecting and implementing platforms.
Data virtualization can play a role in creating logical views across diverse data sources, but care must be taken not to use it as a substitute for proper data governance.
Looking ahead the focus for the next 1-2 years is expected to be on the fundamentals: data governance, strategy, quality, and architecture, all of which are designed to enable the newer, more cutting edge applications of data.