Tuesday, December 31, 2024

High Level Design Document - what all to include?

 A high-level design (HLD) document should provide a clear overview of the system's architecture, components, and interactions without delving into excessive implementation details. Here's a breakdown of the level of detail to include for both the current and proposed designs:

Current Design:

  • Overall Architecture:
    • Diagrams illustrating the main components and their relationships.
    • Brief descriptions of each component's functionality.
    • Key technologies and frameworks used.
  • Key Data Flows:
    • Illustrate how data moves through the system.
    • Identify major data sources and destinations.
    • Highlight any critical transformations or processing steps.
  • Limitations and Challenges:
    • Outline existing bottlenecks, performance issues, or scalability concerns.
    • Mention any known security vulnerabilities or compliance gaps.
    • Summarize maintainability or technical debt issues.

Proposed Design:

  • Architecture Diagram:
    • Show the new or modified components and their interactions.
    • Explain how the proposed design addresses the limitations of the current system.
    • Highlight any new technologies or architectural patterns being introduced.
  • Component Details:
    • Describe the functionality of each major component.
    • Outline key interfaces and APIs.
    • Mention any relevant design patterns or best practices being followed.
  • Data Flow Diagrams:
    • Illustrate how data will flow in the new system.
    • Highlight any changes in data sources, destinations, or processing.
    • Explain how data integrity and consistency will be maintained.
  • Non-Functional Requirements:
    • Address performance, scalability, security, and reliability considerations.
    • Define metrics and targets for key non-functional requirements.
    • Outline any trade-offs or compromises made.

General Guidelines:

  • Focus on the "what" not the "how": Avoid implementation-specific details like code snippets or specific algorithms.
  • Use diagrams and visuals: Clearly illustrate the system's architecture and data flows.
  • Keep it concise and clear: Use simple language and avoid jargon.
  • Target audience: Consider the technical background of the stakeholders who will be reviewing the document.

By following these guidelines, you can create a high-level design document that effectively communicates the essential aspects of your system's design without getting bogged down in unnecessary details.

Sunday, October 20, 2024

Mastering rsync: A Comprehensive Guide

 

Introduction

rsync, a versatile utility, is a powerful tool for efficient data synchronization and backup. It's particularly useful for mirroring directories, transferring files, and maintaining consistency between systems. In this blog post, we'll delve into the fundamentals of rsync, explore its key features, and provide step-by-step examples to help you get started.

Understanding rsync

rsync operates on the principle of delta transfer, meaning it only transfers the differences between the source and destination files. This significantly reduces transfer time, especially for large files or directories.

Key features of rsync:

  • Efficiency: Delta transfer minimizes data transfer.
  • Versatility: Can sync files, directories, and devices.
  • Reliability: Includes checksum verification for data integrity.
  • Flexibility: Supports various protocols (SSH, rsync, direct TCP).

Basic Usage

The basic syntax for rsync is:

rsync [options] source destination

To synchronize the contents of the source_dir directory to the destination_dir directory, use:

rsync -av source_dir/ destination_dir/
  • -a: Archives mode, preserving permissions, ownership, times, and symlinks.
  • -v: Verbose output, providing detailed information about the transfer process.

Example 2: Excluding files

To exclude files with the .swp extension from the synchronization, use:

rsync -av --exclude='*.swp' source_dir/ destination_dir/

Example 3: Using SSH

To synchronize files to a remote server using SSH, use:

Bash
rsync -av user@remote_server:/path/to/destination source_dir/

Advanced Usage

  • Incremental backups: Use the --incremental option to create incremental backups, only transferring changes since the last backup.
  • Dry run: Use the --dry-run option to simulate the transfer without actually copying any files.
  • Compression: Use the -z option to compress data during transfer.
  • Hard links: Use the -H option to preserve hard links.

Conclusion

rsync is an invaluable tool for efficient data synchronization and backup. By understanding its key features and following the examples provided, you can effectively leverage rsync to streamline your data management tasks. Experiment with different options and configurations to tailor rsync to your specific needs.

Friday, October 18, 2024

File Systems - A Primer!

 A file system is a method of organizing and storing data on a computer's storage device. It's like a filing cabinet for digital information, providing a structure for storing, retrieving, and managing files.

Key components of a file system:

  • Files: Individual units of data, such as documents, images, or programs.
  • Directories: Folders that contain files or other directories, creating a hierarchical structure.
  • Metadata: Information about a file, such as its name, size, creation date, and permissions.
  • Data blocks: The smallest units of data that are stored on the physical storage device.

Traditional File Systems:

  • EXT4: A widely used file system for Linux systems, offering good performance and reliability.
  • NTFS: The default file system for Windows systems, known for its features like compression, encryption, and quota management.
  • HFS+: The native file system for macOS, designed for Apple's operating system.
  • FAT32: An older file system, still used for compatibility with older devices and systems.

Network File Systems (NFS):

  • NFS: A network protocol that allows clients to access files on a remote server as if they were local.

Distributed File Systems:

  • GlusterFS: A scalable, open-source distributed file system that can be used to create large-scale storage clusters.
  • Ceph: A distributed object storage system that can be used to create scalable and reliable storage solutions.

Choosing the right file system for your specific needs involves considering several factors. Here's a breakdown of key considerations:

Operating System Compatibility:

  • Linux: EXT4, XFS, Btrfs, ZFS
  • Windows: NTFS, FAT32, exFAT
  • macOS: APFS (Apple File System), HFS+

Performance Requirements:

  • Sequential access: EXT4, XFS, Btrfs generally excel.
  • Random access: ZFS can be efficient for certain workloads.
  • Large files: ZFS and Btrfs can handle large files effectively.

Scalability:

  • Storage capacity: ZFS and Btrfs can scale to large storage pools.
  • Number of clients: NFS and GlusterFS can handle multiple clients.

Features:

  • Data integrity: ZFS and Btrfs offer strong data integrity features.
  • Compression: ZFS and Btrfs support compression to save storage space.
  • Snapshotting: ZFS and Btrfs provide efficient snapshotting capabilities.
  • Deduplication: ZFS can reduce storage requirements by identifying and eliminating duplicate data.

Here's a table summarizing the key characteristics of some popular file systems:

File SystemOperating SystemPerformanceScalabilityFeatures
EXT4LinuxGood for sequential accessModerateData integrity, journaling
NTFSWindowsGood for general useModerateData integrity, compression, encryption
APFSmacOSExcellent performanceGoodData integrity, compression, encryption, space sharing
ZFSLinux, Solaris, FreeBSDExcellent for large datasetsHighData integrity, compression, deduplication, snapshots
BtrfsLinuxGood for large files and snapshotsHighData integrity, compression, snapshots, copy-on-write

Monday, September 16, 2024

The Linux File System structure explained

 

Linux File System Structure

The Linux file system is a hierarchical structure, meaning files and directories are organized in a tree-like fashion. This structure allows for efficient organization and management of files.

Root Directory (/)

  • The topmost directory in the file system hierarchy.
  • All other directories and files are organized beneath it.

Directories

  • Containers for files and other directories.
  • Common directories include:
    • /bin: Contains executable binaries.
    • /etc: Configuration files for the system.
    • /home: Home directories for users.
    • /lib: Dynamic libraries used by programs.
    • /mnt: Mount point for external file systems.
    • /opt: Additional software packages.
    • /tmp: Temporary files.
    • /usr: User-related files, including programs, libraries, and documentation.
    • /var: Variable data, such as logs, mail, and temporary files.

Files

  • Basic units of data storage.
  • Can contain text, images, programs, or any other type of data.

Absolute and Relative Paths

  • Absolute path: Starts with a slash (/) and specifies the exact location of a file or directory from the root directory.
    • Example: /home/user/documents/report.txt
  • Relative path: Specifies the location of a file or directory relative to the current working directory.
    • Example: documents/report.txt (if the current working directory is /home/user)

Inodes

  • Unique identifiers for files and directories.
  • Store metadata about the file, such as its size, permissions, and ownership.

File Permissions

  • Control who can access a file or directory.
  • Typically divided into three categories:
    • Owner: The user who created the file.
    • Group: The group that the file belongs to.
    • Other: All other users.
  • Permissions can be set for reading, writing, and executing.

Example of a Linux file system structure:

/
├── bin
├── etc
├── home
│   └── user1
│       ├── Documents
│       │   └── report.txt
│       └── Downloads
│           └── image.jpg
├── lib
├── mnt
├── opt
├── tmp
├── usr
├── var
└── .ssh

This structure provides a clear and organized way to manage files and directories on a Linux system.

Tuesday, August 27, 2024

Options for choosing a great Linux Distro in 2024?

 When choosing a Linux distribution (distro) that is popular, stable, widely supported, and offers a good desktop environment, several options stand out. These distros are well-suited for both beginners and advanced users who value reliability, community support, and a polished user experience.

1. Ubuntu

  • Popularity: One of the most popular Linux distros, with a large user base and extensive community support.
  • Stability: Ubuntu is known for its stability, especially with its Long-Term Support (LTS) releases, which are supported for five years.
  • Support: Widely supported by both community and commercial entities. Many software vendors offer official Ubuntu packages.
  • Desktop Environment: Ubuntu uses the GNOME desktop environment by default, which is user-friendly, modern, and highly customizable. The Ubuntu community also offers other official flavors with different desktop environments (e.g., Kubuntu with KDE Plasma, Xubuntu with XFCE, and Ubuntu MATE with MATE).

Ideal For: Users looking for a balance between ease of use, stability, and support.

2. Linux Mint

  • Popularity: Very popular, particularly among users who prefer a traditional desktop experience.
  • Stability: Built on Ubuntu LTS releases, which gives it the same level of stability and support.
  • Support: Benefits from Ubuntu's vast repository and community support, with additional support from the Linux Mint community.
  • Desktop Environment: Offers three main desktop environments: Cinnamon (the flagship, known for its traditional look and modern features), MATE (lightweight and stable), and XFCE (even lighter and faster).

Ideal For: Users who prefer a traditional desktop interface with high stability and ease of use.

3. Fedora

  • Popularity: Popular among developers, sysadmins, and those who want cutting-edge software.
  • Stability: Fedora is known for being more up-to-date than Ubuntu or Mint, but it is still quite stable. It has a fast release cycle with new versions every six months.
  • Support: Supported by Red Hat and has a strong community. It's also a base for Red Hat Enterprise Linux (RHEL), so it receives a lot of attention in the enterprise space.
  • Desktop Environment: GNOME is the default desktop environment, with a strong emphasis on providing a vanilla GNOME experience. Fedora also offers spins with KDE Plasma, XFCE, LXQT, and other desktop environments.

Ideal For: Users who want to be on the cutting edge of Linux technology while maintaining a solid desktop experience.

4. Debian

  • Popularity: Debian is the base for many other distributions, including Ubuntu. It is highly respected in the Linux community.
  • Stability: Renowned for its stability, especially in its "Stable" branch. Debian prioritizes the reliability of its software packages.
  • Support: Extensive community support, with a large repository of packages. It's also widely used in servers and embedded systems.
  • Desktop Environment: Debian offers a choice of desktop environments during installation, including GNOME, KDE Plasma, XFCE, and LXDE. The choice allows users to pick the environment that best suits their needs.

Ideal For: Users who prioritize stability and have some experience with Linux.

5. openSUSE

  • Popularity: Well-known in the Linux community, particularly among developers and sysadmins.
  • Stability: openSUSE offers two main versions: openSUSE Leap, which is highly stable and suitable for production environments, and openSUSE Tumbleweed, which is a rolling release for those who want the latest software.
  • Support: Backed by SUSE, with a strong community. Leap benefits from the same enterprise-grade code as SUSE Linux Enterprise.
  • Desktop Environment: KDE Plasma is the default for openSUSE, offering a polished and feature-rich desktop experience. GNOME and other environments are also available.

Ideal For: Users who want a solid, stable system with enterprise-level features and good support for KDE Plasma.

6. Pop!_OS

  • Popularity: Gaining popularity, especially among developers, gamers, and those who use Linux for creative work.
  • Stability: Built on Ubuntu, so it inherits the stability and support of Ubuntu's LTS releases.
  • Support: Developed and supported by System76, with a strong focus on user experience and performance.
  • Desktop Environment: Pop!_OS uses a customized version of GNOME, optimized for workflow efficiency and ease of use. It also has tiling window management built-in, which is a plus for power users.

Ideal For: Users who want an Ubuntu-based system with enhancements for development, gaming, and creative tasks.

7. Manjaro

  • Popularity: Very popular among those who want an Arch Linux-based system with easier setup and maintenance.
  • Stability: While Arch Linux is known for being cutting-edge, Manjaro adds extra testing to ensure stability, making it more reliable for everyday use.
  • Support: Strong community support, with a wealth of documentation and forums. Manjaro is also part of the Arch User Repository (AUR), giving access to a vast array of software.
  • Desktop Environment: Manjaro offers multiple official editions with different desktop environments, including XFCE (default), KDE Plasma, and GNOME. Each is carefully tuned for the best possible user experience.

Ideal For: Users who want a rolling release with a balance between cutting-edge software and stability.

Summary:

  • Ubuntu: Best for general users, including beginners, with strong stability and support.
  • Linux Mint: Ideal for those who prefer a traditional desktop experience.
  • Fedora: Suited for developers and those who want cutting-edge software.
  • Debian: Best for users who prioritize stability and reliability.
  • openSUSE: Great for users who prefer KDE Plasma and want enterprise-level features.
  • Pop!_OS: Designed for developers and creative users who want an optimized, Ubuntu-based experience.
  • Manjaro: Best for users who want an Arch-based system with easier setup and good stability.

Each of these distros offers a good desktop environment, with varying degrees of stability, support, and community engagement, making them strong choices depending on your specific needs and experience level.

Which tool to use for network debugging?

Traceroute, MTR, and similar tools are used for network diagnostics, particularly for identifying issues in network paths and understanding the performance of connections between different nodes. Here’s when and why you would use each of these tools:

1. Traceroute

  • When to Use:

    • To identify the path that packets take from a source machine to a destination.
    • To diagnose routing problems or identify points of failure in a network path.
    • To see where traffic is being delayed or where packets are being dropped.
  • Why Use It:

    • Traceroute shows each hop (intermediate router) along the path, including the IP address and sometimes the DNS name of each hop.
    • It helps identify if a network issue is local, within your ISP, or somewhere further along the path to the destination.
  • Example Use Case:

    • If a user is experiencing slow website access or cannot reach a particular server, you might use traceroute to see where the delay or blockage is occurring.
  • Basic Command:

    • traceroute example.com (on Unix/Linux/macOS)
    • tracert example.com (on Windows)

2. MTR (My Traceroute or Matt's Traceroute)

  • When to Use:

    • To continuously monitor the path between a source and destination over time, combining the functionalities of traceroute and ping.
    • To diagnose intermittent network issues, such as packet loss or latency spikes, over time.
    • To get a more detailed view of the network performance, including the variability of latency and packet loss at each hop.
  • Why Use It:

    • MTR provides a real-time, continuous view of the route, showing not just the hops but also how each hop performs over time.
    • It helps to identify transient issues that might not appear in a single traceroute snapshot.
    • MTR displays statistics such as average latency, best/worst latency, and packet loss for each hop, making it easier to pinpoint problematic nodes.
  • Example Use Case:

    • If users report that their connection quality fluctuates or degrades at certain times, MTR can help you observe these variations and identify where they are happening.
  • Basic Command:

    • mtr example.com (on Unix/Linux/macOS)
    • winmtr example.com (on Windows)

3. Ping

  • When to Use:

    • To check if a specific host is reachable and how long it takes for packets to travel to the host and back.
    • To measure the round-trip time (latency) and check for packet loss.
    • To quickly verify network connectivity to a specific IP address or hostname.
  • Why Use It:

    • Ping is a simple and quick tool to check the status of a network connection.
    • It helps diagnose basic connectivity issues or to verify that a network path is working as expected.
    • Ping can help determine whether a network problem is due to latency or packet loss.
  • Example Use Case:

    • If you cannot access a server, ping the server's IP address or hostname to see if it is reachable and to measure latency.
  • Basic Command:

    • ping example.com (on all operating systems)

4. Pathping (Windows Only)

  • When to Use:

    • To combine the functions of ping and tracert with a focus on packet loss over each hop along the network path.
    • To identify where packet loss is occurring along the route from the source to the destination.
  • Why Use It:

    • Pathping provides detailed information about packet loss and latency over a network path, similar to a combination of ping and traceroute.
    • It is useful for detecting network congestion and identifying problematic routers or links.
  • Example Use Case:

    • If you suspect network congestion or packet loss but need more detailed statistics than what ping or traceroute provide.
  • Basic Command:

    • pathping example.com (on Windows)

5. Netstat

  • When to Use:

    • To display network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.
    • To monitor active connections and listening ports on a host.
  • Why Use It:

    • Netstat is useful for network troubleshooting and for monitoring network traffic.
    • It helps identify which processes are using which network ports and whether there are any abnormal connections.
  • Example Use Case:

    • If you want to verify which services are listening on which ports or identify unexpected or suspicious network connections on your machine.
  • Basic Command:

    • netstat -an (on Unix/Linux/macOS/Windows)

6. TCPdump/Wireshark

  • When to Use:

    • To capture and analyze network traffic at a very granular level.
    • To diagnose and troubleshoot complex network issues, particularly those involving specific protocols or packet contents.
  • Why Use It:

    • These tools provide deep insight into the actual packets traveling across the network, allowing detailed analysis of protocol behavior, anomalies, or malicious traffic.
    • They are essential for diagnosing issues that involve packet-level details, such as TCP handshake problems or SSL certificate errors.
  • Example Use Case:

    • If you need to troubleshoot an issue where a specific protocol isn’t behaving correctly or to investigate potential security incidents.
  • Basic Command:

    • tcpdump -i eth0 (on Unix/Linux/macOS)
    • Wireshark is typically used with a graphical interface.

Summary:

  • Traceroute: Best for one-time diagnostics of the route between source and destination, identifying where delays or drops occur.
  • MTR: Best for continuous monitoring of the network path, especially to detect intermittent issues.
  • Ping: Best for quick connectivity checks and basic latency measurement.
  • Pathping: Best for Windows users needing combined insights into packet loss and route.
  • Netstat: Best for viewing active connections and listening ports, useful for security and troubleshooting.
  • TCPdump/Wireshark: Best for deep packet analysis and troubleshooting complex network issues at the protocol level.

Each of these tools serves different purposes and can be used in combination to provide a comprehensive view of network performance and health.

Monday, June 15, 2020

Graphs

Introduction

Graphs represent relationship between objects - consists of Nodes (a.k.a Vertices) and Edges.

Edges could be:
- weighted or unweighted.
- directed or undirected.

Graphs could be:
- cyclic
- acyclic
- a tree

How to represent Graphs?
- adjacency matrix: build a 2D matrix (N * N) and insert weight of matrix if node i & node j have a edge

Look up time: O(1)
Space: O(N*N)

- adjacency list: build list of size N. ith node will contain all its edges as a list element

Look up time: O(N)
Space: O(N+E)


Graph Traversals:
- bfs : can be done with a queue data structure
- dfs: can be done with recursion or using stack data structure

Selected Graph Algorithms:


Minimum Spanning Tree: min(sum of edges) such that all the nodes are visited.

- edges are undirected & all nodes are connected
- MST is not unique: there can be many MST for a particular Graph

a. Kruskal's Algorithm for finding MST: greedily add the lowest edge weight to the spanning tree


Algorithm:
- sort edge weights and add one by one to MST
- only add edge if there is no cycle (edges that connect unvisited nodes)

How to check if there is no cycle in graph? Optimized ans: Union Find.

Implementation:

#include<iostream>
#include<vector>
#include<algorithm>

using namespace std;

// find common ancestor which represents a set
int find (int x, vector<int> &parent) {
    return parent[x] == x ? x : find(parent[x], parent);
}

int kruskal (vector<vector<int>> &G, int n) {
    int edges = G.size();
    int x, y, cost, mincost = 0;

    // Step1: sort edge according to weights/cost to add one by one to MST
    sort(G.begin(), G.end(), [](vector<int> a, vector<int> b) { return a[2] < b[2];});
    
    //initialize parent vector for union find
    vector<int> parent (n,0);
    for(int i = 0; i < n; i++) {
        parent[i] = i;
    }

    //iterate through the edge list from lowest to highest cost
    for(int i = 0; i < edges; i++) {
        x = G[i][0];
        y = G[i][1];
        cost = G[i][2];

        int a = find(x-1, parent);
        int b = find(y-1, parent);

        //check if a cycle is formed using union find. If yes, don't add.
        if(a != b) {
            mincost += cost;
            parent[a] = b;
        }
    }

    return mincost;
}

int main() {
    // initialize the graph for MST
    // {vertexi, vertexj, cost/weight}
    vector<vector<int>> MST = {{1,2,7},{1,4,6},{4,2,9},{4,3,8},{2,3,6}};
    int MST_nodes = 4;

    cout << "Kruskal MST size = " << kruskal(MST, MST_nodes);
}

Time complexity: sorting the elements is O(NlogN) or overall complexity of Union find which is O(ElogV)

b. Prim's Algorithm for finding MST: same greedy approach as Kruskal, but instead of edge add vertex.


Algorithm:
- keep 2 sets of vertices - one part of growing MST, other that doesn't belong
- select vertices from the lowest edge first and add it to MST, then find the connecting next cheapest to visit vertex and add it to the MST, ensuring there are no cycles. (priority queue sorted by edge weight)

Implementation:

#include<iostream>
#include<vector>
#include<queue>
#include<algorithm>

using namespace std;

vector<vector<pair<int,int>>> convertEdgeListToAdjacencyList (vector<vector<int>> &G, int n) {
    // n is vertices
    
    vector<vector<pair<int, int>>> adj (n);
    int edges = G.size();
    for(int i = 0; i < edges; i++) {
        adj[G[i][0]-1].push_back({G[i][2], G[i][1]-1});
        adj[G[i][1]-1].push_back({G[i][2], G[i][0]-1});
    }

    return adj;
}

int prim (vector<vector<int>> &G, int n) {
    int edges = G.size();
    
    /*
    // Step1: find smallest edge - no need to sort
    int minedge = INT_MAX; int x = 0, y = 0;
    for(int i = 0; i < edges; i++) {
        if(G[i][2] < minedge) {
            minedge = G[i][2];
            x = G[i][0];
            y = G[i][1];
        }
    }
    */

    vector<vector<pair<int, int>>> adj = convertEdgeListToAdjacencyList (G, n);

    priority_queue<pair<int,int>, vector<pair<int,int>>, greater<pair<int,int>>> q;

    int x = 0, y = 0, mincost = 0;
    vector<bool> visited (n, 0);

    q.push({0,1}); // start from a random vertex and go from there, also assign it 0 weight

    while(!q.empty()) {
        
        pair<int,int> p = q.top(); //select the item with lowest weight
        q.pop();
        x = p.second;
        
        // check for cycle by checking visited
        if(visited[x]) continue;

        mincost += p.first;
        visited[x] = 1;
        for(int i = 0; i < adj[x].size(); i++) {
            int cost = adj[x][i].first; 
            y = adj[x][i].second;
            if(!visited[y]) q.push({cost,y});
        }
    }

    return mincost;
}

int main() {
    // initialize the graph for MST
    // {vertexi, vertexj, cost/weight}
    vector<vector<int>> MST = {{1,2,7},{1,4,6},{4,2,9},{4,3,8},{2,3,6}};
    int MST_nodes = 4;

    cout << "Prim's MST size = " << prim(MST, MST_nodes) << endl;
}


Time complexity: priority queue insertion is logN time and iterating thru all edges and vertices. Total: O((V+E) logV).


High Level Design Document - what all to include?

 A high-level design (HLD) document should provide a clear overview of the system's architecture, components, and interactions without d...