SaaS Fundamentals: A Primer's Guide for Success

Overview

Have you ever encountered a problem, big or small, and thought, 'Wouldn't it be great if there was software to solve this?' You envision a tool, a solution, something that would make life easier or a process more efficient. That spark of an idea, that 'what if' moment, is where innovation begins. But what if you could do more than just imagine it? What if you could take that spark and transform it into a real, thriving application? With the SaaS approach, that's not just a dream – it's an achievable reality. This presentation will show you the fundamental steps to take your software idea from a simple thought to a successful online service.

#Innovation Series

YouTube Video

Video Agenda

• Introduction to SaaS
• Planning for SaaS Success
• Technical Approach & MVP
• Automation, Security, and Data
• Rollout and Market Presence
• Continuous Retention and Support
• How Do I Get Started?

Presentation

Introduction to Software as a Service (SaaS)

SaaS is a cloud-based software delivery model where users pay monthly or yearly to use it. It does not require the need to install anything or buy licenses.

Features:

• Cloud hosted.
• Monthly or yearly commitment.
• Reduced cost.
• Low risk investment for consumers.
• Ongoing product updates that are seamless to the users.

👉 *Key takeaway: SaaS offers a convenient and cost-effective way to access and use software.

Planning for SaaS Success

A SaaS concept should be about ideas that solve real-world problems and are possible to deliver without breaking the bank.

Concepts:

• Problem Identification: Pinpoint a real-world problem your SaaS will solve. Validate user pain points.
• Market Research: Assess market demand and analyze competitors to understand the landscape.
• Business Models: Define pricing, subscription tiers, and payment options for revenue generation.
• Financial Planning: Create a budget for development, tech, marketing, and support costs.
• Technology Budget: Estimate costs for your tech stack, tools, and third-party services.
• Marketing & Support Costs: Include marketing and customer support expenses in your financial plan.

👉 Key takeaway: Solid planning and an MVP are crucial for early validation.

Technical Approach & Minimum Viable Product (MVP)

The technical foundation of your SaaS is built during the MVP phase. Selecting the right technologies and defining the scope of your initial release are critical for efficient development and early user validation.

Concepts:

• MVP Core Features: Define the essential features that address the core problem your SaaS solves.
• Iterate with Feedback: Plan for continuous iteration based on user feedback collected during the MVP phase.
• Tech Stack: Choose the development tools and platforms that best suit your project's needs.
• Code/No-Code: Evaluate if no-code solutions can efficiently deliver the MVP's core functionality.
• Cloud Provider: Select a reliable cloud hosting provider (Microsoft Azure, Google Cloud, AWS).
• Hosting Options: Consider various hosting options based on scalability and performance requirements.

👉 Key takeaway: A focused MVP and a well-chosen tech stack accelerate early validation and development.

Operational Foundation: Automation, Security, and Data

A robust SaaS solution requires a strong operational foundation. This ensures efficient updates, seamless tenant creation, and secure cloud deployments.

Concepts:

• Build Automation: Implement automation for deployment, updates, and infrastructure management to streamline operations.
• Security: Establish a secure and scalable authentication and authorization system using federated identity management. Importance of data security and user privacy.
• Data Management: Define data storage, backup, and recovery strategies to ensure data integrity and availability.
• Ability to Scale: Design the system to handle increasing user loads and data volumes.

👉 Key takeaway: Use a third-party cloud service that provide those features

Launching Your SaaS: Marketing & Deployment

Launching your SaaS involves more than just coding; it's about building a strong market presence. A well-crafted marketing and product site, coupled with strategic deployment, will drive user adoption.

• Marketing Site:
  • Creating a compelling marketing website to attract users.
  • SEO and content marketing strategies.
  • Use a CRM to track leads.
• Product Site:
  • Creating a user-friendly product website with documentation and support resources.
• Cloud Hosting:
  • Choosing a reliable cloud hosting provider (e.g., AWS, Azure, Google Cloud).
  • Scalability and reliability considerations.
• Marketplace:
  • Exploring opportunities for listing your SaaS on marketplaces (e.g., Azure Marketplace, AWS Marketplace).
  • Benefits of marketplace distribution.

👉 Key takeaway: A strong online presence is essential for attracting and retaining users.

Building Long-Term Success: Retention & Support

The journey doesn't end after launch. Retaining users and providing exceptional support are critical for long-term SaaS success.

• Continuous Retention:
  • Implementing feedback mechanisms to gather user insights.
  • Regularly updating and improving the SaaS product.
  • Building a community around your SaaS.
• Support:
  • Providing excellent customer support through various channels (e.g., chat, email, knowledge base).
  • Proactive support and troubleshooting.
  • Creating a knowledge base.

👉 Key takeaway: Continuous improvement and excellent support are crucial for long-term success.

How Do I Get Started?

To begin your SaaS journey, it's crucial to follow the fundamental concepts we've discussed. These principles will guide you through the process of building a successful and sustainable SaaS solution.

Approach:

• Find a Viable, Real Use Case:
  • Identify a practical problem that can be solved with a SaaS solution, ensuring it's feasible and cost-effective.
• Scope the MVP and Define the Technical Approach:
  • Determine the core features for your Minimum Viable Product (MVP) and select the appropriate technology stack and cloud provider.
• Build a Robust Operational Foundation:
  • Implement automation, establish strong security measures, and create a scalable data management strategy.
• Launch with a Strong Market Presence:
  • Develop a compelling marketing site and a user-friendly product site, and explore marketplace opportunities.
• Focus on Continuous Retention and Support:
  • Prioritize user feedback, deliver ongoing updates, and provide excellent customer support to build long-term success.
• Embrace Iteration for Continuous Growth:
  • Build a culture of continuous improvement, regularly iterating on your product and processes based on user feedback and market trends.

👉 Key takeaway: By following to these fundamental concepts, you can lay a solid foundation for your SaaS journey.

Thanks for reading! 😊 If you enjoyed this post and would like to stay updated with our latest content, don’t forget to follow us. Join our community and be the first to know about new articles, exclusive insights, and more!

Leave comments on this post or contact me at:

• Twitter @ozkary
• BlueSky https://bsky.app/profile/ozkary.bsky.social

👍 Originally published by ozkary.com

Discovering Machine Language a Primer Guide

Overview

Machine Learning can seem like a complex and mysterious field. This presentation aims to discover the core concepts of Machine Learning, providing a primer guide to key ideas like supervised and unsupervised learning, along with practical examples to illustrate their real-world applications. We'll also explore a GitHub repository with code examples to help you further your understanding and experimentation.

#BuildwithAI Series

• Follow this GitHub repo during the presentation: (Please star and follow the project for updates.)

👉 https://github.com/ozkary/machine-learning-engineering

YouTube Video

Video Agenda

Agenda:

1. What is Machine Learning?

   • Definition and core concepts

2. Why is Machine Learning Important?

   • Key applications and benefits

3. Types of Machine Learning

   • Supervised Learning
   • Examples: Classification & Regression
   • Unsupervised Learning
   • Examples: Clustering & Dimensionality Reduction

4. Problem Types

   • Regression: Predicting continuous values
   • Classification: Predicting categorical outcomes

5. Model Development Process

   • Understand the Problem
   • Exploratory Data Analysis (EDA)
   • Data Preprocessing
   • Feature Engineering
   • Data Splitting
   • Model Selection
   • Training & Evaluation

Presentation

What is Machine Learning

ML is a subset of AI that focuses on enabling computers to learn and improve performance on a specific task without being explicitly programmed. In essence, it's about learning from data patterns to make predictions or decisions based on it.

Core Concepts

• Learn from data
• Improve performance with more training data
• Main goal is to make predictions and decisions on new data
• Learn the relation of data + outcome to define the model
• The new data + model predicts an outcome

Why is Machine Learning Important?

ML impacts how computers solve problems. Traditional systems rely on pre-defined rules programmed by humans. This approach struggles with complexity and doesn't adapt to new information. In contrast, ML enables computers to learn directly from data, similar to how humans learn.

• Coding Rules

def heart_disease_risk_rule_based(age, overweight, diabetic):
     """
     Assesses heart disease risk based on a set of predefined rules.

     Args:
         age: Age of the individual (int).
         overweight: True if overweight, False otherwise (bool).
         diabetic: True if diabetic, False otherwise (bool).

     Returns:
         "High Risk" or "Low Risk" (str).
     """
     if age > 50 and overweight and diabetic:
         return "High Risk"
     elif age > 60 and (overweight or diabetic):
         return "High Risk"
     elif age > 40 and overweight and not diabetic:
        return "Moderate Risk"
     else:
         return "Low Risk"

• Learning from data

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score


df = pd.DataFrame(data)
# Prepare the data
X = df[['Age', 'Overweight', 'Diabetic']]  # Features
y = df['Heart Disease']  # Target

# Split data into training and testing sets
# X has the categories/features
# y has the target value
# train data is for training
# test data is for testing
# .2 means 20% of the data is used for testing 80% for training
# 42 is the seed for random shuffling

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train a Random Forest classifier
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Make predictions on the test set
y_pred = model.predict(X_test)

# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy of the model : {accuracy}")

# 70% - 80%: Often considered a reasonable starting point for many classification problems.
# 80% - 90%: Good performance for many applications.
# 90% - 95%: Very good performance. Often challenging to achieve, but possible for well-behaved problems with good data.
# > 95%: Excellent performance, potentially approaching the limits of what's possible for the problem. Be careful of overfitting if you're achieving very high accuracy.
# 100%: Usually a sign of overfitting.

👉 Jupyter Notebook

Types of ML Models - Supervised Learning

Examples

• Regression: Predicting a continuous value (e.g., house prices, stock prices).
• Classification: Predicting a category or class label (e.g., cat/dog/bird, disease/no disease).
• Model Examples: Linear Regression, Logistic Regression, Decision Trees, Random Forest.

Types of ML Models - Unsupervised Learning

Examples

• Clustering: Grouping similar data points together (e.g., group patients by symptoms, age groups)

• Association: Discovering relationships or associations between items (e.g., symptom association)

"Patients who report 'Fever' and 'Cough' are also frequently reporting 'Headache' or 'Muscle Aches'."

• Model Examples: Clustering (k-means), association (Frequent Pattern Growth)

Supervise Learning - Common Problem Types

Regression and Classification are two main problem types to solve. With Regression, we look to predict a continuous target variable like price, cost. With Classification, we look to predict a discrete target like a group or y/n class.

Problem Types

1. Regression:

   • In regression, the target variable is continuous and represents a quantity or a number.
   • Example: Predicting house prices, temperature predictions, stock prices.

2. Classification:

   • In classification, the target variable is discrete and represents a category or a class.
   • Example: spam vs. non-spam emails, predicting heart disease Y/N.

ML Model Development Process - MLOps

Developing a new ML model involves understanding the core problem, then using a data engineering process to gather, explore, and prepare the data. We then move to the ML process to split the data, select the algorithm, train and evaluate the model.

• Development Process
  • Understand the problem
  • Exploratory Data Analysis (EDA)
  • Data Preprocessing
  • Feature Engineering
  • Data Splitting
  • Model Selection
  • Train TRaining
  • Model Evaluate & Tuning
  • Deployment

MLOps is the operation process to manage the training, evaluation and deployment of your models

👉 Vehicle MSRP Regression

Machine Learning Summary

Machine learning (ML) enables computers to learn patterns from data and make predictions or decisions without explicit programming, unlike rule-based systems. ML models improve as they process more data.

• Supervised Learning:

  • Learns from labeled data (input-output pairs)
  • Regression: Predicts continuous values (e.g., house prices)
  • Classification: Predicts categories (e.g., heart disease y/n)

• Unsupervised Learning:

  • Learns from unlabeled data (only inputs) and discovers patterns and structures.
  • Clustering: Grouping similar data points (e.g., group patients by symptoms, age groups)
  • Association: Discovering relationships between items (e.g., symptoms association)

While we've explored foundational areas, numerous other exciting topics exist, such as neural networks, natural language processing, computer vision, large language models (LLMs). Visit the repository for more exploration.

Thanks for reading! 😊 If you enjoyed this post and would like to stay updated with our latest content, don’t forget to follow us. Join our community and be the first to know about new articles, exclusive insights, and more!

Leave comments on this post or contact me at:

• Twitter @ozkary
• BlueSky https://bsky.app/profile/ozkary.bsky.social

👍 Originally published by ozkary.com

Smart Charts: Powered by AI to enhance Data Understanding

Overview

This presentation explores how Generative AI, particularly Large Language Models (LLMs), can empower engineers with deeper data understanding. We'll delve into creating complex charts using Python and demonstrate how LLMs can analyze these visualizations, identify trends, and suggest actionable insights. Learn how to effectively utilize LLMs through prompt engineering with natural language and discover how this technology can save you valuable time and effort.

#BuildwithAI Series

• Follow this GitHub repo during the presentation: (Give it a star and follow the project)

👉 https://github.com/ozkary/ai-engineering

YouTube Video

Video Agenda

Agenda:

1. Introduction to LLMs and their Role in Data Analysis and Training

   • What are LLMs, and how do they work?
   • LLMs in the context of data analysis and visualization.

2. Prompt Engineering - Guiding the LLM

   • Crafting effective prompts for chart analysis.
   • Providing context within the prompt (chart type, data).

3. Tokens - The Building Blocks

   • Understanding the concept of tokens in LLMs.
   • How token limits impact prompt design and model performance.

4. Let AI Help with Data Insights - Real Use Case

   • Creating complex charts using Python libraries.
   • Write Prompts for Chart Analysis
   • Utilizing an LLM to analyze the generated charts.
   • Demonstrating how LLMs can identify trends, anomalies, and potential areas for improvement.

5. Live Demo - Create complex charts using python and ask AI to help you with the analysis

   • Live coding demonstration of creating a complex chart and using an LLM to analyze it.

Why Attend?

• Discover how to leverage LLMs to gain deeper insights from your data visualizations.
• Learn practical techniques for crafting effective prompts to guide LLM analysis.
• Enhance your data analysis skills with the power of AI.

Presentation

What are LLM Models - Not Skynet

Large Language Model (LLM) refers to a class of Generative AI models that are designed to understand prompts and questions and generate human-like text based on large amounts of training data. LLMs are built upon Foundation Models which have a focus on language understanding.

Common Tasks

• Text and Code Generation: LLMs can generate code and data analysis text based on specific prompts

• Natural Language Processing (NLP): Understand and generate human language, sentiment analysis, translation

• Text Summarization: LLMs can condense lengthy pieces of text into concise summaries

• Question Answering: LLMs can access and process information from various sources to answer questions, making a great fit for chatbots

Training LLM Models - Secret Sauce

Models are trained using a combination of machine learning and deep learning. Massive datasets of text are collected, cleaned, and fed into complex neural networks with multiple layers. These networks iteratively learn by analyzing patterns in the data, allowing them to map inputs like chart data to desired outputs, such as chart analysis.

Training Process:

• Data Collection: Sources from books, articles, code repositories, and online conversations

• Preprocessing: Data cleaning and formatting for the ML algorithms to understand it effectively

• Model Training: The neural network architecture is trained on the data. The network adjusts its internal parameters to learn how to map input data (user stories) to desired outputs (code snippets)

• Fine-tuning: Fine-tune models for specific tasks like code generation, by training the model on relevant data (e.g., specific programming languages, coding conventions).

Transformer Architecture - Not Autobots

Transformer is a neural network architecture that excels at processing long sequences of text by analyzing relationships between words, no matter how far apart they are. This allows LLMs to understand complex language patterns and generate human-like text.

Components

• Encoder: Process the input (use story) by using multiple encoder layers with self-attention Mechanism to analyze the relationship between words

• Decoder: Uses the encoded information and its own attention mechanism to generate the output text (like code), ensuring it aligns with the text.

• Attention Mechanism: Enables the model to effectively focus on the most important information for the task at hand, leading to improved NLP and generation capabilities.

👉 Read: Attention is all you need by Google, 2017

Fine-Tuning for Specific Domain

Fine-tuning LLMs is a process to specialize a pre-trained model into a specific domain like data analysis with your process information.

Process:

• Use the knowledge and parameters gained from a large pretrained dataset, source model
• Enhance the model performance by retraining the source model with a domain specific and smaller dataset
• Use the target model for the final integration

Tokens - The Building Blocks of Language Models

Large language models work by dissecting text into a sequence of tokens. These tokens act as the building blocks, allowing it to grasp the essence, structure, and connections within the text.

Details

• Tokens can be individual words, punctuation marks, or even smaller sub-word units, depending on the specific LLM architecture.
• The length of a word can influence the number of tokens it generates.
• Similar to how Lego bricks come in various shapes and sizes, tokens can vary depending on the model's design.
• Measure cost.

👉 Think of tokens as Lego blocks

Prompt Engineering - What is it?

Prompt engineering is the process of designing and optimizing prompts to better utilize LLMs. Well described prompts can help the AI models better understand the context and generate more accurate responses.

Features

• Clarity and Specificity: Effective prompts are clear, concise, and specific about the task or desired response

• Task Framing: Provide background information, specifying the desired output format (e.g., code, email, poem), or outlining specific requirements

• Examples and Counter-Examples: Including relevant examples and counterexamples within the prompt can further guide the LLM

• Instructional Language: Use clear and concise instructions to improve the LLM's understanding of what information to generate

Chart Analysis with AI

By combining existing charts with AI-driven analysis, we can unlock deeper insights, automate interpretation, and empower users to make more informed decisions.

Data Analysis Flow:

• Chart Data: Identify the key data points from the chart (e.g., x-axis values, y-axis values, data labels, limits).

• Chart Prompt: Format this information in a concise and human-readable format, such as:

"We are looking at a control chart measuring Curvature data points…"

• Analysis Prompt: Provide details about what would you like to learn from the data:

"Interpret this chart, data series, limits and action items to take?"

👉 LLM generated analysis is not perfect, if the prompt is not detail enough, hallucinations may occurred

Smart Charts Powered by AI - Summary

LLMs empower developers, data engineers/analysts, and scientists by enhancing data understanding through AI-driven chart analysis. To ensure accurate and insightful analysis, crafting detailed prompts is crucial.

• Provide Chart Context:

  • Chart Type: (e.g., line chart, bar chart, pie chart, scatter plot)
  • Chart Title
  • Data Series
  • Data Ranges/Limits: (e.g., Time period, Upper/Lower Limits)

• Provide Guiding Questions:

  • What is the overall trend of the data?
  • Are there any significant peaks or dips?
  • Are there any outliers or anomalies?
  • What are the key takeaways from this chart?
  • What actions, if any, should be considered?

👉 By framing the prompt with contextual and guiding questions, you effectively "train" the model to analyze the chart in a more human-like and insightful manner.

Thanks for reading! 😊 If you enjoyed this post and would like to stay updated with our latest content, don’t forget to follow us. Join our community and be the first to know about new articles, exclusive insights, and more!

Leave comments on this post or contact me at:

• Twitter @ozkary
• BlueSky https://bsky.app/profile/ozkary.bsky.social

👍 Originally published by ozkary.com

Cosmos DB for MongoDB: Tapping into Change Streams for Real-Time Integration

Overview

Azure Functions triggers for Cosmos DB enable developers to write event-driven applications that respond to changes in a collection/container. While this integration works seamlessly with the core SQL API, it doesn't directly support the MongoDB API. To achieve similar functionality with the MongoDB API, you can leverage change streams, a powerful feature that provides real-time monitoring of data modifications. This article will guide you through setting up and utilizing change streams in Cosmos DB's MongoDB API within an Azure Function.

Cosmos DB is a data base service which supports various database systems: NoSQL, MongoDB, PostgreSQL, Apache Cassandra, Apache Gremlin, and Table.

Understanding Change Streams

Change streams offer a continuous, ordered stream of changes occurring in a MongoDB collection (or a Cosmos DB container using the MongoDB API). They track inserts, updates, replaces, and deletes, providing your applications with real-time visibility into data modifications. This is invaluable for scenarios like:

• Real-time Analytics and Reporting: Update dashboards and analytics systems as data changes.
• Data Synchronization: Keep different data stores in sync by reacting to changes in real time.
• Event-Driven Architectures: Trigger downstream processes and workflows based on data modifications.
• Auditing and Logging: Capture a detailed history of data changes for audit trails and compliance.

Implementing Change Streams in Azure Functions

Here's how to set up a change stream within an Azure Function:

• Prerequisites:

  • An active Azure subscription.
  • A Cosmos DB account configured with the MongoDB API.
  • An Azure Function App.
  • Install the MongoDB Driver: Use npm to install the necessary driver:

npm install mongodb

Implement the Azure Function

For this integration, we can use a Timer Trigger function. Since the MongoDB API doesn't offer a direct change feed trigger like the SQL API, the Timer Trigger provides a workaround. The function will execute at specified intervals (e.g., every 5 minutes or less) and establish a connection to MongoDB. Upon connection, it can then retrieve change stream events. This approach maintains the serverless nature of Azure Functions, as the function isn't continuously running but activates periodically to process changes.

An alternative to an Azure Function is to build a Node.js or .NET Core application and run it as a service on a VM. This provides a constantly running process for change stream monitoring, but requires managing the VM and application lifecycle.

• Configure the Timer Trigger:

In your Azure Function's function.json, configure the timerTrigger to define the execution schedule. The schedule expression follows the NCrontab format. For example, to trigger every 5 minutes, use */5 * * * *.

{
  "bindings": [
    {
      "name": "myTimer",
      "type": "timerTrigger",
      "direction": "in",
      "schedule": "*/5 * * * *"
    }
  ]
}

The name property specifies the name of the timer object that will be passed to your function. The schedule expression determines the frequency of execution. Adjust the schedule value as needed to control the polling interval for change stream events. More frequent polling captures changes more rapidly, but consumes more resources. Less frequent polling conserves resources, but may introduce latency in processing changes.

• Configure your app settings

  Use the local.settings.json for local development and your function settings on Azure to store the following configurations values:

{
  "IsEncrypted": false,
  "Values": {
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "AzureWebJobsStorage": "<your-storage-connection-string>",
    "CosmosDBConnectionString": "mongodb://<your-cosmosdb-connection-string-from azure>",
    "CosmosDBDatabaseName": "<db-name>",
    "CosmosDBCollectionName": "<collection-name>"
  }
}

Ensure you define the AzureWebJobsStorage setting with a valid Azure Storage connection string. This is essential for the Azure Functions runtime. Furthermore, we'll use this storage account to persist the last processed change stream resume token. Each change stream record includes a token, enabling the resumption of processing from a specific point. By saving the token after each function execution, we can restart the function and continue processing new changes without duplicates. Upon restarting, the function will retrieve the stored token and resume the change stream from that point.

• Implement the Function Code:

  Use the MongoDB Node.js driver to connect to Cosmos DB and process the change stream:

import { InvocationContext, CosmosDBTrigger } from "@azure/functions";
import { MongoClient, ChangeStreamDocument, Document } from "mongodb";


const getResumeToken = async function() : Promise<Binary | null> {

    // document = {token, updatedDt, _id}
    const lastProcessedDoc = await getLastProcessedToken() || null; // retrieve the last processed token
    let lastProcessedToken = lastProcessedDoc?.token ? Binary.createFromBase64(lastProcessedDoc.token) : null;        

    return lastProcessedToken;
}

const factoryTrigger = async function (context: Context, documents: ChangeStreamDocument<Document>[]): Promise<void> {

    // read the database env settings from local.settings.json or function configuration (Azure)
    const connectionString = process.env["CosmosDBConnection"]; 
    const databaseName = process.env["CosmosDBDatabaseName"]; 
    const collectionName = process.env["CosmosDBCollectionName"]; 
    const client = new MongoClient(connectionString);

    let currentToken = null;    

    try {
        await client.connect();

        const database = client.db(databaseName);
        const collection = database.collection(collectionName);                          
        const lastProcessedToken = await getResumeToken();

        // define the pipeline with the events and properties on the document
        const pipeline = [ 
        { $match: { "operationType": { $in: ["insert", "update", "replace"] } } }, 
        { $project: { "_id": 1, "fullDocument": 1, "ns": 1, "documentKey": 1} } ];

        // use resumeAfter when the token is found
        const changeStream = lastProcessedToken ? 
            collection.watch(pipeline, { resumeAfter: { _data: lastProcessedToken}, fullDocument: 'updateLookup'}) : collection.watch(pipeline,  { fullDocument: 'updateLookup' });                             

        // Set up event handlers for the change stream the doc is the full document with the current changes
        changeStream.on('change', async (doc: ChangeStreamDocWithToken<Document>) => {
            console.log('Data change detected', doc);

            // get the resume token from the document
            const binToken = doc._id._data;
            const token = binToken.toString('base64');

            // Save the last processed token the next run
            currentToken = { token, updated: doc.fullDocument.updatedDt, id: doc.fullDocument._id };            

            // Add your auditing logic here
            console.log(`Send to storage id: ${doc.fullDocument._id}`);

            await sendToLog(logCollection, doc.fullDocument);

            const dateString = new Date().toISOString().slice(0, 10); // "YYYY-MM-DD"
            const blobName = `log-${dateString}.json`;  // Example filename: log-2024-07-25.json
            const changeData = JSON.stringify(doc.fullDocument) + '\n'; // Newline-delimited JSON
            console.log(changeData);

            // Append the data to blob or another mongodb collection depending on your requirements            

        });

        changeStream.on('error', async (error: any) => { 
          // or more specific error type if known
            context.log("Change stream error:", error);                          
        });

        changeStream.on('close', () => {  
            // Handle the 'close' event, usually optional
            context.log('Change Stream Closed');           
        });

        changeStream.on('connect', () => {  
            console.log('Change Stream Connected');           
        });

        // max timer to allow function to stop
        const timeout = setTimeout(async () => { 
                console.log(`Closing the stream after timeout ${maxDuration}`);
                clearTimeout(timeout);            
                await changeStream.close();   
                await client.close();  
                if (currentToken)       
                    updateLastProcessedToken(currentToken);
            }, maxDuration);

        context.log('Watching for changes...');
    } catch (err) {
        context.log('Error setting up change stream:', err);

        if (client)
            await client.close();
    } 
};

export default cosmosDBTrigger;

The factoryTrigger function uses a timer to periodically poll a Cosmos DB change stream (using the MongoDB API) for data modifications. It retrieves the last processed change stream token from blob storage to resume processing from where it left off. The function then watches the specified Cosmos DB collection for inserts, updates, and replaces, processing each change by sending the full document to a log collection and appending it to a newline-delimited JSON file in blob storage.

A timeout is implemented to limit the function's execution time and maintain its serverless nature. The timer ensures the function doesn't run continuously, conserving resources, while still periodically checking for and processing changes. The timeout further enforces this resource constraint by closing the change stream and exiting the function after a specified duration. This prevents runaway execution and associated costs while allowing the function to pick up where it left off in the next timer-triggered execution.

• Implement the Blob Storage API

import { fileRead, fileWrite } from "./blobStorageUtils";

export interface DocumentToken {
    token: string;
    updated: string;
    id: string;
}

const tokenKey = 'resume-token';

const MISSING_CONFIG = 'Missing configuration partition/row key for last processed token.'
const NOT_FOUND = 'No existing entity found'
const FAILED_UPDATE = 'Failed to update record';

/**
 * Read the last process token from a blob storage
 * @returns the value 
 */
export async function getLastProcessedToken(): Promise<DocumentToken | null> {

    if (!tokenKey) {
        throw new Error(MISSING_CONFIG);
        return;
    }

    try {
        const blobName = `${tokenKey}.json`;        
        const value = await fileRead(blobName);
        return JSON.parse(value) as DocumentToken;
    } catch (error) {
        console.log(NOT_FOUND);
        return null;
    }
}

export async function updateLastProcessedToken(value: DocumentToken): Promise<void> {

    if (!tokenKey || !rowKey) {
        throw new Error(MISSING_CONFIG);
        return;
    }

    try {        
        const blobName = `${tokenKey}.json`;        
        await fileWrite(blobName, JSON.stringify(value));

    } catch (error) {
        console.log(FAILED_UPDATE, error);        
    }
}

This is a simple implementation wrapper which simplifies interaction with Azure Blob Storage, abstracting away the complexities of the @azure/storage-blob package (which requires import { BlobServiceClient, ContainerClient } from "@azure/storage-blob";). It offers convenient functions for reading and writing JSON data to blobs, streamlining the process of managing the change stream resume token. Consult the Azure documentation on the @azure/storage-blob package for more detailed information and advanced usage scenarios.

The getLastProcessedToken function reads the resume token from a file named resume-token.json. The updateLastProcessedToken function then overwrites this file with the latest resume token. This mechanism allows the change stream to be restarted from a specific point, ensuring that changes are processed sequentially without gaps or duplicates.

Conclusion:

Change streams provide a powerful mechanism for reacting to data modifications in Cosmos DB's MongoDB API. While the MongoDB API doesn't directly support a change feed trigger within Azure Functions, the timer-based approach outlined here offers a near real-time solution. By periodically polling the change stream, applications can effectively capture and process data changes with minimal latency. This approach balances the need for real-time responsiveness with the efficiency and cost-effectiveness of serverless functions. Leveraging change streams in this way opens up opportunities to build dynamic, data-driven applications that react swiftly to evolving information, combining the scalability and flexibility of Cosmos DB with the familiar MongoDB development experience.

Thanks for reading.

Send question or comment at Twitter @ozkary 👍 Originally published by ozkary.com

Introduction to Data Lakes and Data Warehouses - Data Engineering Process Fundamentals -

Overview

In this technical presentation, we will delve into the fundamental concepts of Data Engineering, focusing on two pivotal components of modern data architecture - Data Lakes and Data Warehouses. We will explore their roles, differences, and how they collectively empower organizations to harness the true potential of their data.

• Follow this GitHub repo during the presentation: (Star the project to follow and get updates)

👉 GitHub Repo

• Data engineering Series:

👉 Blog Series

YouTube Video

Video Agenda

Agenda:

1. Introduction to Data Engineering:

2. Brief overview of the data engineering landscape and its critical role in modern data-driven organizations.

3. Operational Data

4. Understanding Data Lakes:

5. Explanation of what a data lake is and its purpose in storing vast amounts of raw and unstructured data.

6. Exploring Data Warehouses:

7. Definition of data warehouses and their role in storing structured, processed, and business-ready data.

8. Comparing Data Lakes and Data Warehouses:

9. Comparative analysis of data lakes and data warehouses, highlighting their strengths and weaknesses.

10. Discussing when to use each based on specific use cases and business needs.

11. Integration and Data Pipelines:

12. Insight into the seamless integration of data lakes and data warehouses within a data engineering pipeline.

13. Code walkthrough showcasing data movement and transformation between these two crucial components.

14. Real-world Use Cases:

15. Presentation of real-world use cases where effective use of data lakes and data warehouses led to actionable insights and business success.

16. Hands-on demonstration using Python, Jupyter Notebook and SQL to solidify the concepts discussed, providing attendees with practical insights and skills.

17. Q&A and Hands-on Session:

18. An interactive Q&A session to address any queries.

Conclusion:

This session aims to equip attendees with a strong foundation in data engineering, focusing on the pivotal role of data lakes and data warehouses. By the end of this presentation, participants will grasp how to effectively utilize these tools, enabling them to design efficient data solutions and drive informed business decisions.

Presentation

Data Engineering Overview

A Data Engineering Process involves executing steps to understand the problem, scope, design, and architecture for creating a solution. This enables ongoing big data analysis using analytical and visualization tools.

Topics

• Data Lake and Data Warehouse
• Discovery and Data Analysis
• Design and Infrastructure Planning
• Data Lake - Pipeline and Orchestration
• Data Warehouse - Design and Implementation
• Analysis and Visualization

Follow this project: Give a star

👉 Data Engineering Process Fundamentals

Operational Data

Operational data is often generated by applications, and it is stored in transactional relational databases like SQL Server, Oracle and NoSQL (JSON) databases like MongoDB, Firebase. This is the data that is created after an application saves a user transaction like contact information, a purchase or other activities that are available from the application.

Features:

• Application support and transactions
• Relational data structure and SQL or document structure NoSQL
• Small queries for case analysis

Not Best For:

• Reporting system
• Large queries
• Centralized Big Data system

Data Lake - Analytical Data Staging

A Data Lake is an optimized storage system for Big Data scenarios. The primary function is to store the data in its raw format without any transformation. Analytical data is the transaction data that has been extracted from a source system via a data pipeline as part of the staging data process.

Features:

• Store the data in its raw format without any transformation
• This can include structure data like CSV files, unstructured data like JSON and XML documents, or column-base data like parquet files
• Low Cost for massive storage power
• Not Designed for querying or data analysis
• It is used as external tables by most systems

Data Warehouse - Analytical Data

A Data Warehouse is a centralized storage system that stores integrated data from multiple sources. The system is designed to host and serve Big Data scenarios with lower operational cost than transaction databases, but higher costs than a Data Lake. This system host the Analytical Data that has been processed and is ready for analytical purposes.

Data Warehouse Features:

• Stores historical data in relational tables with an optimized schema, which enables the data analysis process
• Provides SQL support to query the data
• It can integrate external resources like CSV and parquet files that are stored on Data Lakes as external tables
• The system is designed to host and serve Big Data scenarios. It is not meant to be used as a transactional system
• Storage is more expensive
• Offloads archived data to Data Lakes

Discovery - Data Analysis

During the discovery phase of a Data Engineering Process, we look to identify and clearly document a problem statement, which helps us have an understanding of what we are trying to solve. We also look at our analytical approach to make observations about at the data, its structure and source. This leads us into defining the requirements for the project, so we can define the scope, design and architecture of the solution.

• Download sample data files
• Run experiments to make observations
• Write Python scripts using VS Code or Jupyter Notebooks
• Transform the data with Pandas
• Make charts with Plotly
• Document the requirements

Design and Planning

The design and planning phase of a data engineering project is crucial for laying out the foundation of a successful system. It involves defining the system architecture, designing data pipelines, implementing source control practices, ensuring continuous integration and deployment (CI/CD), and leveraging tools like Docker and Terraform for infrastructure automation.

• Use GitHub for code repo and for CI/CD actions
• Use Terraform is an Infrastructure as Code (IaC) tool that enables us to manage cloud resources across multiple cloud providers
• Use Docker containers to run the code and manage its dependencies

Data Lake - Pipeline and Orchestration

A data pipeline is basically a workflow of tasks that can be executed in Docker containers. The execution, scheduling, managing and monitoring of the pipeline is referred to as orchestration. In order to support the operations of the pipeline and its orchestration, we need to provision a VM and data lake, and monitor cloud resources.

• This can be code-centric, leveraging languages like Python
• Or a low-code approach, utilizing tools such as Azure Data Factory, which provides a turn-key solution
• Monitor services enable us to track telemetry data
• Docker Hub, GitHub can be used for the CI/CD process

Data Warehouse - Design and Implementation

In the design phase, we lay the groundwork by defining the database system, schema model, and technology stack required to support the data warehouse’s implementation and operations. In the implementation phase, we focus on converting conceptual data models into a functional system. By creating concrete structures like dimension and fact tables and performing data transformation tasks, including data cleansing, integration, and scheduled batch loading, we ensure that raw data is processed and unified for analysis. Create a repeatable and extendable process.

Data Warehouse - Data Analysis

Data analysis is the practice of exploring data and understanding its meaning. It involves activities that can help us achieve a specific goal, such as identifying data dimensions and measures, as well as data analysis to identify outliers, trends, and distributions.

• We can accomplish these activities by writing code using Python and Pandas, SQL, Visual Studio Code or Jupyter Notebooks.
• What's more, we can use libraries, such as Plotly, to generate some visuals to further analyze data and create prototypes.

Data Analysis and Visualization

Data visualization is a powerful tool that takes the insights derived from data analysis and presents them in a visual format. While tables with numbers on a report provide raw information, visualizations allow us to grasp complex relationships and trends at a glance.

• Dashboards, in particular, bring together various visual components like charts, graphs, and scorecards into a unified interface that can help us tell a story
• Use tools like PowerBI, Looker, Tableau to model the data and create enterprise level visualizations

Conclusion

Both data lakes and data warehouses are essential components of a data engineering project. The primary function of a data lake is to store large amounts of operational data in its raw format, serving as a staging area for analytical processes. In contrast, a data warehouse acts as a centralized repository for information, enabling engineers to transform, process, and store extensive data. This allows the analytical team to utilize coding languages like Python and tools such as Jupyter Notebooks, as well as low-code platforms like Looker Studio and Power BI, to create enterprise-quality dashboards for the organization.

We've covered a lot today, but this is just the beginning!

If you're interested in learning more about building cloud data pipelines, I encourage you to check out my book, 'Data Engineering Process Fundamentals,' part of the Data Engineering Process Fundamentals series. It provides in-depth explanations, code samples, and practical exercises to help in your learning.

Amazon

Thanks for reading.

Send question or comment at Twitter @ozkary 👍 Originally published by ozkary.com