Category: Uncategorized

KQL databases in an Eventhouse are where you store and query real-time data that flows from Eventstreams and other streaming sources. Once data is loaded into tables, you can use the Kusto Query Language (KQL) or T-SQL to query your data.

Within an eventhouse, you can create:

• KQL databases: Real-time optimized data stores that host a collection of tables, stored functions, materialized views, shortcuts and data streams.
• KQL querysets: Collections of KQL queries that you can use to work with data in KQL database tables. A KQL queryset supports queries written using Kusto Query Language (KQL) or a subset of the Transact-SQL language.

Understand the power of Kusto Query Language (KQL)

To query data in a table in a KQL database, you can use the KQL. KQL is specifically designed for analyzing large volumes of structured, semi-structured, and unstructured data with exceptional performance. KQL databases are optimized for time-series data and index incoming data by ingestion time and partition it for optimal query performance. KQL is the same language used in Azure Data Explorer, Azure Monitor Log Analytics, Microsoft Sentinel, and in Microsoft Fabric.

Get familiar with KQL syntax

KQL queries are made of one or more query statements. A query statement consists of a table name followed by operators that take, filter, transform, aggregate, or join data. For example, to print any 10 rows in the stock table, execute:

kqlCopy

stock
| take 10

A more complex example might aggregate data to find average stock prices over the last 5 minutes:

kqlCopy

stock
| where ["time"] > ago(5m)
| summarize avgPrice = avg(todouble(bidPrice)) by symbol
| project symbol, avgPrice

 Tip

To learn more about KQL, see Kusto Query Language (KQL) overview.

Automate data processing with management commands

Beyond basic querying, you can automate data processing through management commands including:

• Update policies: Automatically transform incoming data and save it to different tables as it arrives.
• Materialized views: Precalculate and store summary results for faster queries.
• Stored functions: Save frequently used query logic that you can reuse across multiple queries.

 Tip

For more information about working with KQL databases, including detailed examples of update policies, materialized views, and stored functions, see Work with real-time data in a Microsoft Fabric Eventhouse.

Other query options

Using SQL

KQL databases in Eventhouses also support a subset of common T-SQL expressions for data professionals already familiar with T-SQL syntax. For example:

SQLCopy

SELECT TOP 10 * FROM stock;

Use Copilot to help with queries

Microsoft Fabric includes Copilot for Real-Time Intelligence, which can help you write queries to extract insights from your Eventhouse data. Copilot uses AI to understand what you’re looking for and can generate the required query code.

https://cosmicnext.com/hrms

Real-Time Intelligence in Microsoft Fabric provides two primary approaches for ingesting streaming data: using eventstreams or directly ingesting data into a KQL database in an Eventhouse.

Eventstreams for data ingestion and transformation

Eventstreams are a way to bring real-time events into Fabric, to transform them, and then route data to a destination.

The image shows the three main components of an Eventstream: sources where data originates, transformations optional processing applied to the data, and destinations where the processed data is sent.

Think of the Eventstream components like a water pipe system. The source is your faucet, transformations are filters along the way and you need a destination like a sink or bucket to collect and use the water.

Next, let’s review each component of an Eventstream.

Data sources for eventstreams

Once you create an eventstream in Fabric, you can connect it to a wide range of data sources. You can stream data from Microsoft sources and also ingest data from non-Microsoft platforms including:

• Microsoft sources, like Azure Event Hubs, Azure IoT Hubs, Azure Service Bus, Change Data Capture (CDC) feeds in database services, and others.
• Azure events, like Azure Blob Storage events.
• Fabric events, such as changes to items in a Fabric workspace, data changes in OneLake data stores, and events associated with Fabric jobs.
• External sources, such as Apache Kafka, Google Cloud Pub/Sub, and MQTT (Message Queuing Telemetry Transport) (in preview)

 Tip

To see all supported sources, see Supported sources.

Event transformations in eventstreams

Raw data from a source system is rarely in the exact format you need for analysis or storage. Transformations are what make your data useful and actionable. You can transform the data as it flows in an eventstream, enabling you to filter, summarize, and reshape it before storing it. Examples of available transformations include: SQL code, filter, manage fields, aggregate, group by, expand and join.

 Tip

For more information about supported transformations, see Process event data with event processor editor and Process events using SQL code editor.

Data destinations in eventstreams

Streaming data flows continuously and is temporary by nature. It requires immediate processing and storage to retain its value. The destination in an eventstream is what makes your real-time data processing actionable. It’s where your processed data becomes available for queries, reports, dashboards, alerts, actions, or integration with other systems. You can load the data from your stream into the following destinations: a KQL database in an Eventhouse, Lakehouse, a derived stream, Fabric Activator, or a custom endpoint.

 Tip

For more information about supported destinations, see Add and manage a destination in an eventstream.

Direct ingestion to a KQL database in an Eventhouse

Data can also be directly ingested into a KQL (Kusto Query Language) database in an Eventhouse. Some examples of data ingestion sources include: local files, Azure storage, Amazon S3, Azure Event Hubs, OneLake, and more. Data ingestion can be configured using connectors or through the Get data option in a KQL database as shown in this image.

 Tip

For more information about supported ingestion sources for KQL databases in Eventhouses, see Data sources and Data connectors overview.

Data transformation in a KQL database in Eventhouse with update policies

When directly ingesting data into a KQL database, data first lands in the database, then can be transformed using update policies. This is different from eventstream transformations that occur during stream processing, before routing data to a destination.

Update policies are automation mechanisms triggered when new data is written to a table. They run a query to transform ingested data and save the result to a destination table.

https://cosmicnext.com/financial

As organizations generate increasing volumes of event-driven data, the ability to process, analyze, and act on data in motion becomes essential for competitive advantage. Real-Time Intelligence provides comprehensive capabilities for working with streaming data with minimal latency.

Explore Real-Time Intelligence use cases

Unlike batch systems that process data on scheduled intervals, Real-Time Intelligence helps you respond to events as they happen, delivering near real-time insights.

Here are some common types of event data and examples of how Real-Time Intelligence can support downstream actions and business responsiveness:

• Delivery tracking: Monitor vehicle locations to alert customers when packages are delayed
• Equipment monitoring: Track machine temperature to prevent costly breakdowns
• Fraud detection: Analyze purchase patterns to block suspicious transactions immediately
• Website performance: Monitor page load times to improve user experience
• System health: Track application errors to maintain service reliability

Real-Time Intelligence components

Microsoft Fabric’s Real-Time Intelligence is an integrated set of components that work together to handle streaming data from capture through automated response.

The diagram shows how Real-Time Intelligence components work together for end-to-end processing. Each component handles a specific stage of the real-time analytics process:

Ingest and process data in motion with Eventstreams

Data ingestion and processing can happen through Eventstreams, which capture streaming data from various sources and apply real-time transformations as data flows through the system. Eventstreams can filter, enrich, and transform your data and route it to different destinations.

Store real-time data in an Eventhouse

Real-Time Intelligence stores data in KQL (Kusto Query Language) databases in Eventhouses. These databases are designed for time-series data and fast ingestion of streaming data. The storage integrates with OneLake, making your data available to other Fabric tools.

Analyze data with KQL Queryset

KQL Queryset provides a workspace for running and managing queries against KQL databases. The KQL Queryset allows you to save queries for future use, organize multiple query tabs, and share queries with others for collaboration. The KQL Queryset also supports T-SQL queries, allowing you to use familiar SQL syntax alongside KQL for data analysis.

Visualize insights with Real-Time Dashboard

Real-Time Dashboards connect directly to KQL databases and refresh automatically as new data arrives. These dashboards let you explore data interactively and monitor both current conditions and historical trends.

Act on data with Activator

Automated actions can be configured with Activator, which continuously monitors streaming data against user-defined rules and thresholds. When conditions are met, Activator can send notifications, trigger workflows in Power Automate, execute Fabric data pipelines or notebooks, creating event-driven automation that responds to real-time conditions.

Discover streaming data with the Real-Time hub

The Fabric Real-Time hub is a central location where you can discover and manage all of the data-in-motion that you have access to. It gives you a way to ingest streaming data from Azure and from external sources and it lets you subscribe to Azure and Fabric events.

Think of the Real-Time hub as your streaming data catalog where you can see what’s happening in near real-time across your organization. There are connectors you can use to ingest data into Microsoft Fabric from various sources. For example, you might connect to IoT sensor streams through Azure Event Hubs, subscribe to Azure Blob Storage events, use Change Data Capture (CDC) to stream database changes, or monitor Fabric workspace events.

Once you have configured a connection to data source or event source, these items become the foundation for event driven decision making and a wide range of real-time analytics solutions, from building dashboards and setting up alerts to triggering automated workflows and analyzing trends in your data.

To access the real-time hub, select the Real-Time icon in the main Fabric menu bar.

The real-time hub organizes data-in-motion into several main categories:

• Data sources: Browse and connect to available streaming data sources, such as Microsoft sources, database change data capture feeds, and external sources from other cloud providers
• Azure sources: Discover and configure Azure streaming data sources such as Azure IoT Hub, Azure Service Bus, Azure Data Explorer DB, and more
• Fabric events: Subscribe to system-generated events in Fabric that you can access, like job status changes, events produced by action on files or folders in OneLake, and Fabric workspace item changes
• Azure events: Subscribe to system events from Azure services that can be used to trigger automated responses such as actions on files or folders in Azure blob storage

In the Real-Time hub, you can preview and explore your streaming data by navigating directly to eventstreams or KQL databases in eventhouses for deeper analysis and querying. You can also build automated responses using Activator rules that trigger actions like notifications, workflows, or data processing when specific patterns are detected.

https://cosmicnext.com/feature

Real-time analytics is the practice of processing, analyzing, and acting on data as it’s generated, typically within seconds to minutes of when events occur. Unlike traditional analytics that works with static snapshots of historical data stored in databases, real-time analytics operates on data that’s actively flowing through your systems, enabling immediate insights and rapid responses to changing conditions. This approach is also known as near real-time analytics, since there’s always some degree of processing and network latency involved.

Understand events and streams

Events are records of things that happen in a system. They capture moments when something occurs, changes, or is completed. Examples include website clicks, stock price changes, customer purchases, patient vital sign changes, or equipment sensor readings. Think of them as digital records or log entries that document activity across your systems.

A stream is essentially a sequence of events, typically ordered by the time an event occurred. Each event in the stream represents something that happened at a specific moment. Events flow through streams continuously as they occur. For example, a stream of equipment temperature sensor readings contains temperature readings over many points of time. This continuous flow of event information allows you to detect patterns over time, identify opportunities or risks, and take action immediately after something happens, or in real-time.

Streams are the delivery mechanism that carries events from where they happen to where they need to be processed, analyzed, or acted upon.

Components of real-time analytics solutions

To build real-time analytics solutions, you need several integrated capabilities working together:

Real-time data ingestion: Collect data from multiple sources simultaneously, as information is generated. For example: database changes from change data capture, sensors, applications, system logs, and APIs.

Stream processing: Transform and analyze data while it flows from sources to destinations. This includes filtering, aggregating, joining with other data sources, and detecting patterns with minimal latency.

Low-latency storage: Use specialized databases and storage systems designed to handle high-velocity data writes and provide fast query responses.

Interactive dashboards: Create visualizations that update automatically as new data arrives, show current state and trends in real-time.

Automated decision making: Set up event-driven rules and triggers that can initiate actions, send alerts, or start workflows based on real-time conditions.

Use real-time analytics

To use real-time data effectively, information has to be ingested, processed, stored, analyzed, and presented to be actionable. Real-time analytics enables you to:

• Respond immediately to opportunities or problems as they emerge
• Optimize operations by adjusting resources and configurations based on current conditions
• Enhance customer experiences through personalized, contextual interactions
• Prevent issues by detecting anomalies before they become critical problems

Real-Time Intelligence in Microsoft Fabric brings all these capabilities together in a single platform. Through components like Eventstreams for data ingestion and transformation, Eventhouses for analytics-optimized storage, the Real-Time hub for data discovery, Real-Time Dashboards for visualization, and Activator for automated alerts and actions, Real-Time Intelligence enables you to monitor critical events, trigger automated responses, track business processes, and analyze patterns in real-time, turning what happens in your systems into actionable insights.

https://cosmicnext.com/erp

As technology advances, human and AI interaction grows more important. AI isn’t just for automation; it’s transforming industries, improving our lives, and sparking innovation. This video examines AI’s societal impact and the key considerations for its integration.

AI is changing industries by enabling data-driven decisions, automating processes, and fostering innovation. By following these guidelines, we can harness the power of AI responsibly and shape a future that aligns with our shared values and aspirations.

Data privacy: Balance the need for data with the protection of individual privacy rights.

Algorithmic bias: Detect and mitigate biases in AI systems to prevent societal biases from being reflected.

Transparency: Ensure AI decision-making processes are clear and understandable to foster trust and accountability.

Legal liability: Address responsibility for AI decisions, considering the roles of developers, users, and the AI itself.

Innovation and accountability: Striking a balance between innovation and accountability is essential for responsible AI usage.

Data sharing: Encourage data sharing while safeguarding privacy to improve AI systems without compromising individual rights.

AI research: Invest in AI research to fuel innovation and ensure the benefits of AI are accessible to all.

Digital education: Promote digital education and workforce development to equip people with the skills needed in an AI-transformed job market.

AI advisory committees: Establish AI advisory committees to provide oversight, insights, and guidance on AI development and deployment.

Government engagement: Engage with government officials to shape policies that affect AI use in communities.

These guidelines are designed to ensure the responsible and ethical use of AI, fostering a positive and fair influence on society.

https://cosmicnext.com/digital-transformation

Deepfake technology added complexity to the rapid production and distribution of digital content. AI-generated deepfakes, which can mimic real people, pose serious ethical and legal issues, especially concerning copyright and intellectual property.

This video investigates deepfakes and their impacts, highlighting strategies to protect creators’ rights and verify digital content. It addresses the challenges of deepfakes, industry’s fight against misinformation, and emerging legal frameworks.

Creating a safer digital space involves enforcing safety protocols, updating regulations, and increasing public education. Being well-informed is crucial for managing AI-created content.

https://cosmicnext.com/data-analytics

It is essential to develop AI systems based on trustworthy, fair, and privacy-conscious principles. This video outlines six key principles: accountability, inclusiveness, reliability & safety, fairness, transparency, privacy & security. Discover how these principles foster trust and help create AI systems that respect individual rights and serve society.

The journey to responsible AI begins with trust, a trust that is built on these six principles:

Accountability: Define clear roles and responsibilities for AI impacts.

Inclusiveness: Ensuring AI benefits everyone and is accessible to all.

Reliability & Safety: Extensive testing, validation, ongoing monitoring, and safety protocols with robust error handling.

Fairness: Equitable treatment of all individuals, with regular assessments to prevent bias.

Transparency: Enable users to comprehend AI decisions in order to build trust.

Privacy & Security: Safeguard user data, collect only necessary information, and implement strong security protocols.

These principles aim to develop AI systems that benefit society while upholding individual rights and values.

https://cosmicnext.com/cyber-security

As AI becomes more integrated into our lives, it’s important to understand how to use it ethically and effectively.

In this video, you explore the best practices to follow while using AI.

Responsible collaboration with AI is essential. Here are some key best practices:

Understand AI: It’s important to grasp the basics of AI and its broad capabilities. This knowledge is the foundation for using AI effectively.

Stay informed: Keep up with the latest advancements and ethical discussions in AI. This will help you leverage AI responsibly.

Recognize AI’s blind spots: AI can reflect societal biases present in the data it learns from. Actively seek unbiased information and understand how AI uses data to navigate these blind spots.

Prioritize safety and privacy: Your data is precious. Choose AI services that value user privacy and prioritize security and transparency.

Cross-verify AI-generated content: Don’t accept AI-generated content at face value. Always cross-verify information from various sources and engage your critical thinking skills.

Evaluate and refine content: Use critical thinking to evaluate and refine AI-generated content by verifying facts and sources, understanding the content’s goals and target audience while considering a range of viewpoints.

Ensure clear policies: Make sure the AI tool or service you’re using has clear policies and guidelines for secure usage.

Promote AI for good: AI should be a tool for good, aiding in areas like healthcare, education, and environmental conservation.

Join the conversation: Start discussions in your community and workplace about responsible AI use. Encourage people to think about how AI will be utilized and take steps to prevent misuse.

These methods will enable you to use AI responsibly and contribute positively to society.

https://cosmicnext.com/crm

Now let’s find inspiration for more complex prompts, and discover practical applications for AI at work.

Explore ways to use AI at work

Microsoft 365 Copilot Chat is a shared chat experience within Microsoft 365 Copilot, and is available at no additional cost with eligible Microsoft 365 and Office 365 licenses. With a Microsoft 365 Copilot license, Copilot Chat becomes more powerful by connecting to your work data across Microsoft 365 applications like Teams, Word, Outlook, PowerPoint, and Excel. This helps connect your questions with business information and apps, bringing out important details from the company’s data.

Some common examples include:

• Outlook: Summarize the content of a large email thread.
• PowerPoint: Turn a text-heavy slide into concise bullet points for greater clarity.
• Word: Rewrite a paragraph in a different tone or style.
• Teams: Summarize meetings and chat threads.

 Note

AI Agents, also known as virtual assistants, help you with different tasks based on what you need. They can handle routine tasks like scheduling meetings, managing emails, real-time data analysis, and personalized recommendations. In Microsoft 365, you can now build your own agents to fit your specific needs, making your daily tasks easier and more efficient.

To learn more, read Introducing Copilot agents and watch the short video Copilots & Agents.

The real power of Microsoft 365 Copilot comes from the flow of using it across apps. There are many use cases for Copilot in Microsoft 365, including in communications, HR, legal, IT, sales, project management, market research, finance, and more. The Copilot scenario library can help you get started.

The following table describes one sample functional scenario, Using Copilot to draft an internal communications post.

In the following short video, learn how one communications manager uses Microsoft 365 Copilot to create and enhance announcement posts.

Now that you learned about generative AI, how to interact with it, and some ways to take your prompts further, let’s practice prompting in Copilot.

https://cosmicnext.com/contact

Now it’s time to dive into various resources that can spark your creativity and enhance your prompting skills in your personal life. In this unit, find inspiration for more complex prompts, and explore new ways to use AI in your daily life.

Explore ways to use AI in your personal life

Let’s take a closer look at a couple of ways you might use AI at home.

Plan a trip

Could you use some help or inspiration to plan a trip? Plan the Perfect Trip Itinerary Using Copilot suggests some prompts to get you started so you can spend less time creating itineraries and more time enjoying them. The following table suggests some good, better, and best prompts.

 Tip

Good prompts are often simple and direct, but leave room for interpretation. Better prompts add specific elements for clarity. The best prompts are clear, actionable, and structured for a specific response.

Train for a marathon

Say you’ve decided to run a marathon, or maybe your friends have talked you into participating in your first race. Whether you’re a seasoned marathon runner or this is your first race, Train for a Marathon with Copilot guides you through how to set running goals, make a training schedule, and get ready for race day using AI.

The following table suggests some good, better, and best prompts to set your running goals and make a training schedule.

Now that you learned about using AI in your personal life, let’s discover some ways to use AI at work.

https://cosmicnext.com/application-services