DEV Community: Ashwin Hariharan

Organizing AI Applications: Lessons from traditional software architecture

Ashwin Hariharan — Mon, 05 Jan 2026 13:30:00 +0000

When I started learning AI and diving into frameworks like LangGraph, n8n, and the OpenAI APIs, I found plenty of great tutorials. They taught me how to build a simple chatbot, how to make my first LLM call, how to chain a few prompts together. Useful stuff for getting started.

Great for for learning. Less great for shipping.

After the first couple of weeks, I wanted to build an actual production-ready application which goes beyond standard POCs. Something which uses AI but involving dozens of routes, multiple features and services, database operations, caching layers. But those beginner tutorials weren't enough. Where do the embeddings live? How do I structure my agent workflows? Should my API routes call AI directly, or is there supposed to be a layer in between?.

The documentation showed me how to use the framework abstractions and APIs. It didn't show me how to organize them.

If you're like me, coming from a JavaScript/TypeScript background, you know the language gives you a lot of freedom - no enforced folder structure, no prescribed architecture. You can organize your code however you want. But that freedom comes at a price.

Without clear patterns to guide where things should go, you might end up with working code in all the wrong places. Calls to OpenAI API scattered everywhere, business logic tangled with your routes, and you just know this is going to be painful to maintain later.

The cost of 'just making it work'

Here's the thing: software architecture trends come and go. In 2015, everyone said microservices were the future. In 2018, serverless. In 2021, JAMstack. In 2023, everyone quietly went back to monoliths.

But you know what remained constant through all these trends? The fundamental principle of separating concerns. These same software development principles apply to AI and agentic AI applications. Whether you're building traditional web apps or AI-powered systems, the need for clear architecture remains constant.

What Makes AI projects maintainable

Let's establish what properties a good AI project should have:

Clear Ownership Boundaries

Clear ownership boundaries define which part of your system is responsible for what. Good boundary means when something breaks or needs extension, you immediately know which component to check.

Each distinct concern in your application should be handled by a single, well-defined module or component. This way, when something goes wrong or when you need to add a feature, you'll immediately know which part of your codebase is responsible.

Clear boundaries mean each concern lives in an identifiable place. When something goes wrong, you immediately know which module to check. When you need to add a feature, you know which component to extend.

Reusability across entry points

Reusability means writing logic once and calling it from anywhere.

Your core business logic should work the same way regardless of how it's triggered. Whether called from a web API, an AI agent, a scheduled job, a command-line tool, a message queue, or a test suite, the same functionality should be available without rewriting it.

Why it matters: Today it's a chat API. Tomorrow you may want a Slack bot. Next week, batch processing. And who knows? Maybe you'll discover that your users actually prefer the regular search over your fancy AI chatbot anyway. If your AI code is tied to say, your controllers, you might have to rewrite it each time.

Testability

Testability is the degree to which a piece of software can be tested easily and effectively. It describes how simple it is to check that the software works as intended. High testability means tests can be written quickly, run reliably, and give clear results, while low testability leads to tests that are difficult, slow, or unclear.

AI applications have many moving parts. Vector search slow? Cache not hitting? Agent hallucinating? Embedding generation failing? When you're debugging, you shouldn't have to hunt through your entire codebase to find the problem.

Provider Independence

Swapping from GPT-4 to Claude to Gemini shouldn't require changing business logic.

Why it matters: AI models evolve weekly. Today's best model is next month's deprecated one. Providers change pricing. Features get sunset. Your architecture should make provider switching very straightforward.

If you're a software engineer who has started building AI applications and you want to move beyond simple code snippets and demos, let me show you what worked for me. In the following sections, I'll walk you through patterns I extracted from a real project - patterns that apply whether you're building e-commerce, SaaS tools, content platforms, or any AI-powered application.

For illustration, I'll use an AI-powered restaurant discovery application. Think of a platform where users search for restaurants, browse by cuisine or location, and make reservations - the typical flow you'd see in apps like Zomato or Yelp. Now imagine you decide to add a chat feature to make restaurant discovery more conversational - a dining assistant in a corner chat window that enables finding the perfect restaurant through natural conversation.

View the implementation on GitHub

So now, instead of just filtering by "Italian cuisine", diners can now also ask questions like "I need a romantic spot with live music for an anniversary dinner".

What this really comes down to:

How do you add these AI capabilities without major refactoring of your existing system? And without over-engineering a solution that's way more complex than it needs to be?

Throughout the code examples, I will use Node.js / JavaScript as its syntax is widely familiar to anyone building apps for the web and generalizes well to other languages. But these architectural patterns apply equally to Python, Java, or any other language you're working with.

Architectural patterns for AI apps

After some research and quite a few experiments and trying out different approaches, I settled on few patterns that actually work. Before we dive in, know that these aren't necessarily AI-specific. They're the same principles that make any large application maintainable. But when you build with these patterns from the start, adding and testing AI features later becomes straightforward.

Let's look at what they look like:

Pattern 1: Structure by business components or modules

Rather than organizing code by technical function (all controllers together, all models together), organize by business components. Each module represents a bounded context with its own API, logic, and data access.

Why This Matters for AI:

AI applications typically have multiple distinct domains:

Conversation management (chat history, session state)
AI workflow orchestration (agents, tools, prompts)
Business entities (restaurants, users, reservations)
Data processing (embeddings, vector search)

Mixing these concerns creates cognitive overload and makes it hard to reason about the system and maintain it.

// ❌ Bad: Organized by technical layers

src/
├── controllers/
│   ├── chatController.ts
│   ├── restaurantController.ts
│   └── reservationController.ts
├── services/
│   ├── chatService.ts
│   ├── restaurantService.ts
│   └── reservationService.ts
└── repositories/
    ├── chatRepository.ts
    ├── restaurantRepository.ts
    └── reservationRepository.ts

Problem: To understand the "chat" feature, you jump between three different directories. Adding a new feature touches files across the entire codebase.

Colocation: For a feature, put related code close together. Code that changes for the same feature should be neighbors and a short navigation away.

This is also popularly known as "Domain-driven design"

When you add a new feature to the chat system, you typically need to modify the API endpoint, update the business logic, and adjust the data access layer. With domain-driven design, all these files are in the same chat/ directory-you never leave that folder. Without it, you're jumping between controllers/, services/, and repositories/ directories, trying to remember which pieces connect.

Each domain is self-contained with its own API, Domain, and Data layers

Benefit: Everything related to "chat" lives in one place. Each business component is self-contained.

// ✅ Good: Organized by business components

modules/
├── chat/
│   ├── api/chatController.ts
│   ├── service/chatService.ts
│   └── data/chatRepository.ts
├── restaurants/
│   ├── api/restaurantController.ts
│   ├── service/restaurantService.ts
│   └── data/restaurantRepository.ts
└── reservations/
    ├── api/reservationController.ts
    ├── service/reservationService.ts
    └── data/reservationRepository.ts

When you need to modify how restaurants are searched, you go directly to modules/restaurants/. When you need to add a new AI tool, it goes in modules/ai/agentic/tools. There's no guessing, no hunting through dozens of files.

This also means you could extract any of these services into a separate microservice later without major refactoring - the boundaries are already defined.

Pattern 2: Layer your feature modules with 3-tier architecture

While Pattern 1 is about grouping by business domain, Pattern 2 applies the same concept of colocation within each domain. The API, domain, and data layers for a feature stay together in the same feature module folder, not scattered across the codebase.

Three-tier architecture

Within each module, maintain clear separation between three concerns:

Entry Points (API routes, message queue consumers, scheduled jobs)
Domain Logic (business rules, workflows, services)
Data Access (database queries, external API calls) - this is also called repository-pattern.

The Critical Rule:

Never pass framework-specific objects (Express Request/Response, HTTP headers, etc.) into your domain layer. Domain logic should be pure and reusable across different entry points.

// ❌ Don't do this: Domain logic coupled to Express

import { Request, Response } from 'express';

export async function handleChatMessage(req: Request, res: Response) {
  const sessionId = req.cookies.sessionId;
  const message = req.body.message;

  // AI workflow logic mixed with HTTP handling
  const result = await aiAgent.run(message);

  res.json({ response: result });
}

The problem with the above code is that this function would only work with Express. But AI workflows often need to be triggered from multiple sources:

HTTP API requests (user chat)
Scheduled jobs (batch processing)
Message queues (async workflows)
Tests (validation)

If your AI logic is tightly coupled to your HTTP layer, you can't reuse it elsewhere, and you won't be able to call it from a scheduled job, test, or CLI tool.

Your AI workflow orchestration should be completely separate from the HTTP layer. Here's an example:

// ✅ Good: Clean separation

import { Request, Response } from 'express';
import { getResponseFromAgent } from '@modules/chat';

export async function handleChatMessage(req: Request, res: Response) {
  const sessionId = req.cookies.sessionId as string;
  const message = req.body.message as string;

  const result = await getResponseFromAgent(message, sessionId);

  res.json({ response: result });
}

// ✅ business logic - no HTTP dependencies
export async function getResponseFromAgent(message: string, sessionId: string) {

  const result = await aiAgent.run(message, sessionId);
  return {
    response: result.text,
    toolsUsed: result.tools
  };
}

Now, getResponseFromAgent() can be called from anywhere - HTTP endpoints, scheduled jobs, tests, or CLI scripts.

The API layer now focuses only on handling HTTP concerns - receiving the request, extracting the session ID and message, and returning a response - while delegating all business logic to the domain layer.

Similarly, use the Repository pattern to prevent database details from leaking into business logic:

// ❌ Don't put this in your services / domain logic
async function searchRestaurants(query: string) {
  const redis = await redisClient.connect();
  const results = await redis.ft.search('idx:restaurants', query);
  // transform query response...
}

// ✅ Good
async function searchRestaurants(query: string) {
  const queryEmbeddings = await generateEmbeddings([query]);
  return restaurantRepository.vectorSearchRestaurants(queryEmbeddings);
}

// ✅ restaurant-repository.ts
export class RestaurantRepository {
  async vectorSearchRestaurants(searchVector: number[]) {
    // Redis-specific implementation hidden
    const results = await redis.ft.search(/* ... */);
    return this.transformToRestaurants(results);
  }
}

The domain layer remains pure and independent of HTTP and database implementation, returning structured results. Now if you need to switch databases based on performance or cost, your domain services don't change - only the repository implementation does.

Pattern 3: Tools and prompts should call domain logic, not implement it

When you're building AI agents with LangChain, LangGraph, or similar frameworks, you define "tools" - functions the AI can call to perform actions. Need to search products? Create a tool for that. Add items to cart? Tool. Get user preferences? Tool.

There are two common anti-patterns where business logic ends up in the wrong place:

Anti-pattern 1: Business logic in prompts

const prompt = `
You are a restaurant discovery assistant. Follow these rules:
1. Only show budget-friendly restaurants (under ₹500 per person) for budget users
2. Apply member discounts on reservations for gold members
3. Suggest fine dining establishments to premium members
...
`

Problem: Your business rules now live in natural language. They're non-deterministic, untestable, and invisible to code review.

Anti-pattern 2: Business logic in tools

It feels natural to write business logic directly in tool functions. During a conversation in the chat interface, the user wants to search products, so you write the search logic right there in the tool. You need database access, so you import the database client. You need to validate the search query, calculate relevance scores, apply business rules - all of it goes into the tool.

// ❌ Bad: Business logic inside AI tool

import { tool } from '@langchain/core/tools';
import { z } from 'zod';
import { createClient } from 'redis';

export const searchRestaurantsTool = tool(
  async ({ query }) => {
    // Database access directly in tool
    const client = await createClient.connect();
    const embedding = await openai.embeddings.create({ input: query });
    const results = await redis.ft.search('idx:restaurants', ...);

    // Business logic in tool
    const filtered = results.filter(r => r.priceFor2 < 2000);
    const sorted = filtered.sort((a, b) => b.rating - a.rating);

    return sorted.slice(0, 5);
  },
  {
    name: "search_restaurants",
    description: "Search for restaurants",
    schema: z.object({ query: z.string() })
  }
);

The problem with the above code is that the core logic is locked in the AI tool. Few months later, you might realize you need that same search logic in a REST API endpoint, or in a scheduled job, or in a different agent. But it's tightly coupled to the AI framework you used.

Tools should be thin wrappers that translate AI intent into domain service calls.

That's it. The tool's job is simple:

Receive parameters from the AI
Validate/transform them if needed
Call the appropriate domain service
Return the result in a format the AI understands

The actual business logic? That lives in domain services, completely independent of any AI framework.

AI tools as thin wrappers calling domain services

Here's how you can write it:

// ✅ Good: logic lives separately

export async function searchRestaurants(query, limit) {
  const queryEmbeddings = await generateEmbeddings([query]);
  const searchVector = queryEmbeddings[0];

  const restaurants = await restaurantRepository.vectorSearchRestaurants(
    searchVector,
    limit
  );

  return restaurants.sort((a, b) => b.rating - a.rating);
}

// ✅ Tool simply calls domain service
import { searchRestaurants } from '@modules/restaurants';

import { tool } from '@langchain/core/tools';
import { z } from 'zod';

export const searchRestaurantsTool = tool(
  async ({ query }) => {
    // Just a thin wrapper
    const restaurants = await searchRestaurants({ query, limit: 5 });
    return restaurants;
  },
  {
    name: "search_restaurants",
    description: "Search for restaurants",
    schema: z.object({ query: z.string() })
  }
);

Benefit: Your business logic is now independent, just plain functions that take input and return output - you can reuse it everywhere. searchRestaurants() can now be called from AI tools, HTTP endpoints, CLI scripts, or tests.

Ask yourself: "If I needed this logic in a REST API endpoint tomorrow, would I have to copy-paste code or could I just import a function?"

If the answer is copy-paste, your logic is in the wrong place.

When your tools are thin adapters to domain services, you can test the core behavior locally without calling the LLM at all. This makes tests fast, reliable, and deterministic. Your code becomes:

Reusable across HTTP, CLI, scheduled jobs, tests
Testable without AI framework mocking
Maintainable because business logic lives in one place
Flexible because you can change AI frameworks without rewriting business logic

Pattern 4: Dependency Inversion

High-level policy (your business logic) should not depend on low-level details (specific frameworks, providers, or databases). This follows the Dependency Inversion Principle from Robert C. Martin's SOLID principles: your code should depend on abstractions, not on concrete implementations.

In simpler terms: when you use external services-whether it's OpenAI, AWS Bedrock, LangChain, or Redis-hide them behind interfaces that match your domain's needs, not theirs.

Why this matters for AI:

AI applications introduce volatile dependencies that change frequently:

AI Providers switch constantly: Today's OpenAI becomes tomorrow's Anthropic or AWS Bedrock
Models evolve rapidly: GPT-4 becomes GPT-4 Turbo becomes GPT-4o
Frameworks shift: LangChain might become LangGraph, or you might switch to a different agentic framework entirely.
Vector databases compete: Pinecone, Weaviate, Redis Vector Search - requirements change.

Without DIP, switching any of these requires changes across your entire codebase. With DIP, you change one file.

Real-world example: switching AI providers

Consider an embeddings service. Without DIP, you'd scatter OpenAI SDK calls throughout your codebase:

// ❌ Bad: Tightly coupled to OpenAI everywhere
import OpenAI from 'openai';

async function searchRestaurants(query: string) {
  const openai = new OpenAI({ apiKey: process.env.OPENAI_KEY });
  const embedding = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: query
  });
  // search logic using embedding...
}

Now imagine your company decides to switch to AWS Bedrock for cost savings. You'd need to find and modify every place that calls OpenAI's embedding API.

With DIP, high level policy does not depend on low level details:

// ✅ Good: embeddings.ts

import OpenAI from 'openai';
export async function generateEmbeddings(texts: string[]) {
  // Implementation hidden behind interface
}

async function searchRestaurants(query) {
    // Generate embedding for the search query
    const textEmbeddings = await generateEmbeddings([query]);

    // Use repository for pure data operations
    return restaurantRepository.vectorSearchRestaurants(textEmbeddings);
}

The implementation file changes based on your provider, but the interface stays the same:

import { BedrockEmbeddings } from '@langchain/aws';

export async function generateEmbeddings(texts: string[]) {
  const embeddings = new BedrockEmbeddings({
    model: process.env.model,
    region: process.env.awsRegion
  });
  return await Promise.all(texts.map(text => embeddings.embedQuery(text)));
}

// openai-version/embeddings.ts
import OpenAI from 'openai';

export async function generateEmbeddings(texts: string[]) {
  const openai = new OpenAI({ apiKey: CONFIG.openAiApiKey });
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: texts
  });
  return response.data.map(item => item.embedding);
}

Your domain services import from the abstraction:

import { generateEmbeddings } from '@modules/ai/helpers';

export async function findRestaurantsBySemanticSearch(query: string) {
  const queryEmbeddings = await generateEmbeddings([query]);
  const searchVector = queryEmbeddings[0];
  return await restaurantRepository.vectorSearchRestaurants(searchVector);
}

Switching providers now means changing one file, not hunting through your entire codebase. The LLM becomes just one dependency behind a clear interface, which we can replace with mocks or fixtures during testing.

Example: Abstracting agent frameworks

The same principle applies to agentic AI frameworks. Your business logic shouldn't know whether you're using LangGraph, CrewAI, or AutoGen.

Instead of spreading LangGraph-specific code everywhere, isolate your agentic AI workflow implementation in its own module:

// ✅ Good
import { HumanMessage } from '@langchain/core/messages';
import { restaurantReservationsWorkflow } from '@modules/ai/workflows';

export async function processUserQuery(query: string, sessionId: string) {
  const result = await restaurantReservationsWorkflow.invoke({
    messages: [new HumanMessage(query)],
    sessionId
  });

  return {
    response: result.result,
    cacheStatus: result.cacheStatus,
    toolsUsed: result.toolsUsed
  };
}

With Dependency Inversion, you can experiment with different models without touching business logic and move between frameworks gradually as your needs evolve. It does introduce extra abstraction, which can sometimes feel like over-engineering, but it's most valuable when dependencies are volatile, you’re evaluating multiple options, or vendor lock-in is a real risk.

Pattern 5: Use environment-aware, secure, and hierarchical config

AI applications have complex configuration needs:

Multiple AI provider credentials (OpenAI, Anthropic, AWS), model identifiers and versions
Rate limits and timeouts
Feature flags for different AI capabilities
Vector database connections
Caching configuration
Guardrail settings

So your configuration should be:

Environment-aware: Different values for dev/staging/production
Secure: Secrets never committed to version control
Validated: Fail fast on startup if config is invalid
Hierarchical: Organized for easy discovery
Type-safe (optional): Preferably with TypeScript or runtime validation

// ❌ Bad: Secrets hardcoded, scattered config
const openai = new OpenAI({
  apiKey: "sk-abc123...",  // Hardcoded secret!
});

const bedrockModel = "anthropic.claude-v2";  // Magic string
const redisHost = "localhost";  // Where's production config?

The problem with the configuration above is that the configuration is scattered everywhere, and there’s no clean way to switch between models and environments.

// ✅ Good: Centralized, validated, environment-aware
// config.js
import dotenv from 'dotenv';
dotenv.config();

const requiredEnvVars = [
  'OPENAI_API_KEY',
  'AWS_ACCESS_KEY_ID',
  'REDIS_HOST'
];

// Fail fast on startup if config is missing
requiredEnvVars.forEach(varName => {
  if (!process.env[varName]) {
    throw new Error(`Missing required environment variable: ${varName}`);
  }
});

export default {
  // AI Providers
  openAi: {
    apiKey: process.env.OPENAI_API_KEY,
    model: process.env.OPENAI_MODEL || 'gpt-4o',
    timeout: parseInt(process.env.OPENAI_TIMEOUT || '30000')
  },

  aws: {
    region: process.env.AWS_REGION || 'us-east-1',
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    bedrockModelId: process.env.BEDROCK_MODEL_ID || 'anthropic.claude-v2'
  },

  // Databases
  redis: {
    host: process.env.REDIS_HOST,
    port: parseInt(process.env.REDIS_PORT || '6379'),
    password: process.env.REDIS_PASSWORD
  }
};

# .env (never committed to git)
OPENAI_API_KEY=sk-abc123...
AWS_ACCESS_KEY_ID=AKIA...
REDIS_HOST=localhost

# .env.production (deployed separately)
OPENAI_API_KEY=sk-prod123...
AWS_ACCESS_KEY_ID=AKIA...
REDIS_HOST=redis.production.com

The benefit now is that all configuration lives in one place, it's automatically validated on startup, switching between environments is easy, and secrets are no longer hardcoded in the codebase.

Pattern 6: Separate persistent data from agent memory

AI agents need to remember things during conversations, but not everything should live in your main database. User preferences? Database. The fact that someone just asked about "cozy sweaters" 30 seconds ago? Agent memory.

Why this matters for AI:

Mixing persistent and ephemeral data leads to bloated databases and slow AI responses. Different types of data have different storage requirements:

Database: User profiles, restaurant catalog, reservation history - things that need to persist indefinitely
Agent Memory: Conversation context, temporary preferences, session state - things that can expire
Vector Store: Restaurant embeddings, semantic search indexes - specialized AI data structures

The Implementation:

services/
├── chat/
│   ├── services/
│   │   ├── workflow.js         # LangGraph orchestration
│   │   └── memory.js           # Session state, conversation context
│   └── data/
│       └── session-store.js    # Fast, ephemeral storage
├── restaurants/
│   └── data/
│       ├── restaurant-repository.js    # Persistent restaurant vector store
└── users/
    └── data/
        └── user-store.js       # User profiles, preferences

Benefits:

Faster AI responses (no database queries for temporary data)
Cleaner main database (only persistent data)
Automatic cleanup (session data expires)
Better performance (right storage for each data type)

Getting the storage layer right is crucial for AI performance.

summary

Check out Redis for AI learning path for hands-on experience implementing these memory patterns and vector search capabilities.

Your final project architecture could look something like this:

modules/
│
├── restaurants/             # Restaurants Domain
│   ├── api/                   # HTTP layer
│   ├── service/                # Business logic
│   └── data/                  # Data access
│
├── reservations/            # Reservations Domain
│   ├── api/
│   ├── service/
│   └── data/
│
├── chat/                    # Conversation Domain
│   ├── api/
│   ├── service/
│   └── data/
├── ai/                      # AI Domain
│   ├── agentic-restaurant-workflow/
│   │   ├── index.js          # Workflow orchestration
│   │   ├── nodes.js          # Agent definitions
│   │   ├── tools.js          # AI tools
│   │   └── state.js          # State management
│   └── helpers/
│       ├── embeddings.js     # Embedding generation
│       └── caching.js        # Cache logic

Complete architecture showing all six patterns working together

Check out the example implementation on Github!

redis-developer / restaurant-discovery-ai-agent-demo

An Agentic AI restaurant discovery platform that combines Redis's speed with LangGraph's intelligent workflow orchestration. Get personalized restaurant recommendations, make reservations, and get lightning-fast responses through semantic caching.

🍽️ Restaurant Discovery AI Agent

Redis-powered restaurant discovery with intelligent dining assistance.

An AI-powered restaurant discovery platform that combines Redis's speed with LangGraph's intelligent workflow orchestration. Get personalized restaurant recommendations, smart dining suggestions, and lightning-fast responses through semantic caching.

App screenshots

Tech Stack

Node.js (v24+) + Express - Backend runtime and API framework
Redis - Restaurant store, agentic AI memory, conversational history, and semantic caching
Redis LangCache API - Semantic caching for LLM responses
LangGraph - AI workflow orchestration
OpenAI API - GPT-4 for intelligent responses and embeddings for vector search
HTML + CSS + Vanilla JS - Frontend UI

Product Features

Smart Restaurant Discovery: AI-powered assistant helps you find restaurants, discover cuisines, and manage your reservations. Both text and vector-based search across restaurants
Dining Intelligence: Get restaurant recommendations with detailed information for any cuisine or occasion using RAG (Retrieval Augmented Generation)
Demo Reservation System: Reservation management…

View on GitHub

Next steps

Applied AI is quite fluid today - patterns, frameworks, and libraries are changing constantly. Teams are expected to deliver features fast. In this environment, your architecture needs flexibility. You can't afford to lock your code into rigid structures that make change expensive.

I have been working with the patterns we discussed for quite some time and they really helped. Remember, these are not rigid rules. Learn what works for your project, and adapt. Think of your project not as a monolithic whole, but as independent, composable features with clear interfaces:

If your code is scattered across technical layers, start grouping domains together grouping domains together.
If your AI tools contain database queries, extract them into domain services.
If you're tightly coupled to OpenAI, add an abstraction layer.

If you're working in the Node.js ecosystem, frameworks like NestJS can help you implement many of these patterns out of the box - modules, dependency injection, layered architecture. But here's the thing: these patterns aren't tied to any specific framework or even any specific language.

Choose the tools that work for your team, and these patterns will help you organize things in such a way that refactoring is something that is performed casually on a daily basis. It will also help pave the way to full-blown microservices in the future once your app grows. You'll likely refactor anyway after you've built something real.

Thanks to Raphael De Lio and Bhavana Giri for the detailed technical review and great suggestions on the article's structure.

How to use Elasticsearch to Create an Exceptional User Experience in Retail

Ashwin Hariharan — Fri, 12 May 2023 07:59:24 +0000

What is Elasticsearch
Search in retail
How is search used in online shopping
What you will learn in this tutorial (watch the demo!)
Data exploration
- Data types and field mappings
- Kibana developer console
- Search ranking and score
Searching for information
- Text search for a product
- Text search for a product of a specific brand
- Search for a particular category of products
- Search for products within a certain category
- Search and filter for products by ratings
- Filter by pricing parameters
- Sorting search results
Improving search relevance
Enhancing search experiences on mobile devices
A day in the life of an Elasticsearch Architect
Resources

You just went to the Google home page. What do you see?

You saw a search box, inviting you to type something. Well I’m a programmer, so I’ll just go ahead and type in the word recursion.

Haha. A recursion joke. You’re funny, Google. But jokes aside, let’s look at the top 5 results:

It probably took less than a second for Google to return me documents from the web that are about recursion. All ordered neatly in order of relevance.

If you're reading this on your phone or laptop right now, you've likely heard of the term search engine. A search engine is a software component that allows you to find information in a computer. It usually consists of an interface that lets you enter a search query - where you specify criteria about an item of interest. This could be in the form of words relevant to the specific topic (such as COVID-19, or The Matrix), and the search engine uses these words to find the relevant information from a database. The search results are usually presented in a list and are commonly called hits, which are displayed to the user.

Often referred to as a Web Search engine, Google took "Seek, and you shall find" to a whole new level. But that's not the only place where they're used. Any domain or industry that relies on the power of computing can benefit immensely from a search engine. In particular, the ability to find documents or information within a system is critical to creating a great user experience for users. Hence, search engines are used in many customer-facing applications across web, desktop and mobile devices.

For instance, in e-commerce platforms such as Amazon, search is used to find products across multiple categories such as clothing, electronics and furniture. Search is also used in blogs and news publication platforms to find articles or sources of information for research. And in entertainment platforms like Netflix, search allows you to find content that you like from a massive catalogue of movies, TV shows and music.

Let's look at a popular open-source software that powers the search feature for some of the world's most widely used enterprise software.

Elasticsearch

Elasticsearch is a free and open-source search engine allowing you to search for nearly any kind of information - including textual, numerical and geospatial data. It emerged from an earlier open source project called Lucene, primarily to provide the following features:

Handle terabytes of data.
Do large scale parallel searching really fast.
Use inexpensive commodity hardware to scale and increase search speed.

In order to achieve the above, what started out as a search box within applications has become a NoSQL database in its own right. Elasticsearch is popularly used along with the ELK Stack, or simply the Elastic Stack. It is a collection of products offered by the Elastic company, and consists of the following components:

Elasticsearch, which powers the other components as a distributed JSON-based search engine.
Logstash: As Elasticsearch started to see widespread usage, users wanted a way to search for logs and compile data for analysis. Logstash is an open server-side data processing pipeline engine which helps with a variety of data analysis and transformation related tasks.
Kibana: A visual user-interface that provides an admin dashboard and data visualization tool for managing and debugging data within Elasticsearch. Users can use Kibana to create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

Some of the most popular use-cases built using the Elastic Stack are:

Powering complicated search feature within applications which involve sophisticated querying and filtering.
Telemetry, logging and metrics.
Business Analytics.
Forecasting and anomaly detection.

Here's an excellent visual story on how search can be leveraged in a myraid of ways.

Let's examine how the Elastic Stack can be used in more detail by considering a use-case.

Search in Retail

In retail platforms, search is a powerful and an integral part of the shopping experience. Here are some interesting statistics:

Search Statistics

A recent report from Forrester indicates that over 43% of users on retail websites go directly to the search bar to shop for products.
Screen Pages analyzed search usage across 21 e-commerce websites and found that the average revenue from visitors using search were significantly higher than those by non-search users.
More than half of all internet traffic comes from native mobile applications, meaning the search experience plays a crucial role in helping users find what they're looking for without exiting the application. This is especially the case when mobile users spend nearly 20x more time shopping than website users!

Elasticstack in Retail

If you're reading this, you're probably wondering whether your app or business can benefit from Elasticsearch. The Elastic Stack is used by some of the world's top e-commerce and retail business in-order to enhance the shopping experience for customers. Here are a few examples:

Ebay leverages Elasticsearch to help users find relevant products over 800 million records.

But this isn't necessarily limited to only shopping. Shopify uses Elasticsearch to add a powerful search feature for the documentation part of their website!

You can find several more usecases on retail here.

At Egen, we work with some of the world's largest retail brands in-order to modernize their IT architecture and help scale. We've put together a short tutorial to demonstrate and help you understand how you can use the ELK stack for your own retail business or application.

Let's first explore what a search experience looks like in online shopping.

Search Experience in online shopping

Here's how a typical search looks like on Amazon or any other online shopping platform. There are generally 3 main categories of search:

Text Search

The home page of Amazon presents me with a search input box. If I want to check out smartphones, I could simply enter "smartphone" in the search box.

This search functionality allows a search across multiple departments or categories.

Search within departments or categories

Alternatively, users can also search for products within a certain department.

Filtering

Users also have the option to apply a variety of filters to include and exclude products based on multiple criteria.

Here are some common filter types:

Yes/no or boolean type filters. For example, fetch products which have single day deliveries.
Multi select filters which include products belonging to certain brands.
Range filters which include searching for products within a certain price range, or above a certain customer rating score.

Sort search results

Users can also decide how they'd like the search results to be sorted, based on factors such as pricing or average reviews:

Let's look at how Elasticsearch can help you provide similar such search experience for your retail application.

What you will learn

After much tinkering and exploration, I managed to build this sweet e-commerce website, with search and filtering powered by Elasticsearch:

booleanhunter-tech-blog / e3-retail

An E-commerce retail website demonstrating Elasticsearch

E³Retail

An E-commerce products-list website that demonstrates the application of Elasticsearch in retail & online shopping.

App Screenshots

Home Page

Products filter page

Features

Responsive on web and mobile.
Built using Progressive enhancement philosophy.

Search features

Search for a product by typing in the search input.
Search within a category.
Search across multiple categories.
Filter by brand, pricing and ratings.
Sort by relevance, ratings or price.
Paginate through search results.
Search input autocomplete suggestions (requires JavaScript to be enabled).

Requirements

A laptop or a desktop computer.
Chrome browser.
Internet connectivity.

Instructions to install

Requires AWS OpenSearch
Initialize environment variables. If you're running the app locally, create a .env file:

    NODE_ENV=development
    ELASTICSEARCH_HOST=your-elasticsearch-domain
    ELASTICSEARCH_PORT=443
    ELASTICSEARCH_USER=your-elasticsearch-master-user
    ELASTICSEARCH_PASSWORD=your-elasticsearch-master-password
    ELASTICSEARCH_INDEX=index-to-search-on

Install Node.js
Clone or fork this repo
Run npm install in your project root
Run npm start to start the app server

Common commands

Creating an index

npm run create-index your-index-name

Bulk uploading Data

Syntax…

View on GitHub

Wanna know how it works? Watch the demo:

You’ll learn the features of Elasticsearch used to create the above website and similar such search experiences!

For the purposes of this tutorial, I will be using a dataset from Kaggle, that contains pre-crawled information of products from Flipkart, a leading Indian eCommerce platform similar to Amazon. You'll learn how to leverage Elasticsearch's powerful APIs in online shopping to search for relevant products that you can purchase from an e-commerce platform.

In-order to use the Elastic Stack, you have the following options:

Install Elasticsearch, LogStash and Kibana on your own hardware or cloud service provider.
Use the managed Elasticsearch cloud offering from the company Elastic.

Once you've installed, open up the Kibana console, and import your dataset:

Data Exploration

When you begin the process of uploading your data, Kibana scans through the first 1000 columns or documents and returns a summary of the file contents in the visual data explorer.

The explorer shows you the fields in your dataset, the number of distinct values in them, along with the distribution of those values.

As you proceed with importing your data, Kibana will ask you to create an index.

If you go with the simple import, kibana automatically assigns a default data-type to the individual fields in your dataset.

But I personally find the advanced import to be more useful, as it allows me to view and choose the most appropriate data-type for the fields in my data.

Data types and field mappings

Elasticsearch provides common data-types like text, boolean, keyword and numbers, and also some advanced data types like object, range, date and geo-spatial. Here's the complete list.

In the advanced import section, you'll be able to review the data-type mappings for the fields/columns in your dataset and modify them if necessary.

Here are the types Kibana assigned to my dataset fields by default:

To understand what data-type fits best, let's study our fields in more detail:

product_name

The name of the product. This is of the text type.

description

As the name implies, this contains the description of a product. This too, is of the text type.

category

The department or category that a product belongs to. When I uploaded my dataset into Kibana first and checked the mappings, I found that Kibana had assigned it a text type. But it is better to assign it as a keyword type instead.

Why is this important?

Elasticsearch optimizes certain fields to perform well for certain kinds of queries.

In our dataset, the category field is useful when its made into a keyword rather than ordinary text. The reason being - shoppers often look for products within a certain category, and these categories always contain the same value.

Keyword types are also useful in aggregation queries - for instance, how many products are in each product category?

brand

The brand name of the product. By default, this was assigned a keyword type.

In e-commerce platforms, shoppers often use filters while searching for a product across different brands - for example, Adidas. In which case, a keyword type fits well.

On the other hand, shoppers also often search for specific brands by directly entering the brand name on the search bar, like "Adidas shoes". In which case, a text field fits well.

And so for such use-cases, Elasticsearch also allows you to assign multiple types to a field. Here's how you do it in the Kibana console.

"brand": {
    "type": "keyword",
    "fields": {
        "raw": {
            "type": "text"      
        }
    }
}

has_next_day_delivery

This is a boolean-type field that tells whether the product can be delivered within the next day.

Elasticsearch optimizes numeric fields, such as integer or float, for range queries.

retail_price

Kibana erroneously assigned this a keyword type, which I changed to float. This will be useful when a user wants to search for products within a certain price range.

product_rating

A number between 0 to 5, where 5 indicates the highest rating on average given by customers, and 1 being the lowest. 0 indicates that the product hasn't been rated yet. I changed the type of this field to half_float.

Once you've imported your dataset, you can look at the mappings by going to the Index Management section in Kibana.

Here's the JSON of the mappings:

{
  "mappings": {
    "_doc": {
      "_meta": {
        "created_by": "file-data-visualizer"
      },
      "properties": {
        "brand": {
          "type": "keyword",
          "fields": {
            "raw": {
              "type": "text"
            }
          }
        },
        "category": {
          "type": "keyword"
        },
        "description": {
          "type": "text"
        },
        "discounted_price": {
          "type": "float"
        },
        "has_next_day_delivery": {
          "type": "boolean"
        },
        "id": {
          "type": "long"
        },
        "image": {
          "type": "keyword"
        },
        "overall_rating": {
          "type": "half_float"
        },
        "pid": {
          "type": "keyword"
        },
        "product_name": {
          "type": "text"
        },
        "product_rating": {
          "type": "half_float"
        },
        "product_url": {
          "type": "keyword"
        },
        "retail_price": {
          "type": "float"
        },
        "uniq_id": {
          "type": "keyword"
        }
      }
    }
  }
}

Kibana Developer Console

The Kibana Devtools has a console that allows you to write and test different kinds of queries on your dataset.

Let's try out one.

Get information about documents in an index

Type GET products/_search on the console, and hit the run icon on the right to execute the query. This tells Elasticsearch to fetch some sample documents from the products index.

As you can see, Elasticsearch returns a JSON response with a sample list of documents from that index, along with plenty of useful metadata as well. Here's a quick breakdown of what each of those fields mean:

hits contains some metadata along with the actual hits found in your dataset.

total: Indicates how many number of actual documents were identified as hits. By default, Elasticsearch returns 10000 in the response. So if your dataset contains more than 10000 documents, this field would still indicate 10000.
relation: This indicates whether the actual number of hits are equal to, less than, or greater than the total hits returned in the response.

By default, this query returns an array of 10 hits. Within each hit, you'll find some additional metadata:

_score: This is the value assigned by Elasticsearch's ranking algorithm to determine how relevant the document is. Since this API returns only a sample, all documents contain the same value of 1.
_index: Indicates the field that the document was retrieved from.

This an example of a basic search query using Kibana. Let's try another one.

Find the number of products in each category

On the console, type this:

GET products/_search
{
  "aggs": {
    "products": {
      "terms": {
        "field": "category",
        "size": 1000
      }
    }
  }
}

It'll return the following response:

Common terminologies

If you're just getting acquainted with search engines, you will frequently come across terms such as ranking and score. These terms are often inter-related to one another. It's useful to understand what they mean and stand for before you begin to dive into querying in Elasticsearch.

Ranking

Ranking determines the order in which search results are returned, by assigning each hit a relevance score. Results are then ordered in decreasing order of their scores - with the highest scores at the top of the list, and the least scored items at the bottom of the list.

Score

Score is a value that implies how relevant the document is to the user. Higher the score, more relevant is the item, and vice-versa. A variety of factors are involved in determining the score of a document, such as:

Term frequency: How many times the search term, or a subset of the search term, appears in the document.

Inverse Term frequency: Often in the query, there might be words or phrases that occur too often in several documents, such as "How to", or "the". When such terms are encountered within a document, inverse term frequency lowers the weight for that document, and increases the weight for terms that occur rarely.

Searching for information

Elasticsearch exposes a set of REST APIs to allow users to query for information. A search query is given in the form of a GET request, after which Elasticsearch processes it and returns the results as a JSON response.

There are two main ways to search in Elasticsearch:

Queries retrieve documents that match the specified criteria.
Aggregations present the summary of your data as metrics, statistics, and other analytics.

Simple Text Queries

Query #1: Text search for a product

This is done using a match query. Here's the syntax:

GET name-of-index/_search
{
  "query": {
    "match": {
      "Column/field within which to search": {
        "query": "Search keywords"
      }
    }
  }
}

Here's a query that searches for shoes in the product_name field and returns upto a 100 products:

GET products/_search
{
  "size": 100,
  "query": {
    "match": {
      "product_name": {
        "query": "shoes"
      }
    }
  }
}

Here's the response:

Here's a sample structure of the JSON response:


"hits" : [
  {
    "_score" : 7.9427953,
    "_source" : {
      "product_name" : "WorldWearFootwear Running Shoes, Walking Shoes",
      "pid" : "SHOEG7FXJFJDFPBD",
      "retail_price" : "999.0",
      "discounted_price" : "380.0",
      "image" : [
        "http://img5a.flixcart.com/image/shoe/s/3/p/white-ast-212-worldwearfootwear-8-original-imaeg7fsg67hxgss.jpeg",
      ],
      "has_next_day_delivery" : "false",
      "description" : "Key Features of WorldWearFootwear Running Shoes, Walking Shoes Material: Mesh Occasion: Sports Color: Multicolor Heel Height: 0,Specifications of WorldWearFootwear Running Shoes, Walking Shoes General Ideal For Men, Boys Occasion Sports Shoe Details Weight 200 g (per single Shoe) - Weight of the product may vary depending on size. Heel Height 0 inch Outer Material Mesh Color WHITE",
      "product_rating" : "0",
      "overall_rating" : "0",
      "brand" : "WorldWearFootwear",
      "category" : "Footwear"
    }
  },
  {
    "_index" : "products",
    "_type" : "_doc",
    "_id" : "xg5Fw30BkBYPif_fIlZO",
    "_score" : 7.9427953,
    "_source" : {
      "id" : "529",
      "uniq_id" : "b168c7b936c3ad9d9a02f1bfafa4926d",
      "product_url" : "http://www.flipkart.com/steppings-running-shoes-casuals/p/itmebg6znr8nbwth?pid=SHOEBG6ZUGTSDGBU",
      "product_name" : "Steppings Running Shoes Casuals Shoes",
      "pid" : "SHOEBG6ZUGTSDGBU",
      "retail_price" : "3699.0",
      "discounted_price" : "2589.0",
      "image" : [
        "http://img6a.flixcart.com/image/shoe/k/t/6/black-8120-2-steppings-39-1000x1000-imaebg2fcsnjff8h.jpeg",
      ],
      "has_next_day_delivery" : "false",
      "description" : "Adidas Running Shoes Casuals Shoes. Price: Rs. 2,589.    .   Feel on the top of the world with this pair of Shoes by Adidas. Let the sun go down for uncomfortable sandals, opt for this pair of Flats that is crafted using a comfortable sole. Team this pair with a midi skirt and a floral crop top for an ultimate look..Feel on the top of the world with this pair of Shoes by Adidas. Let the sun go down for uncomfortable sandals, opt for this pair of Flats that is crafted using a comfortable sole. Team this pair with a midi skirt and a floral crop top for an ultimate look.",
      "product_rating" : "0",
      "overall_rating" : "0",
      "brand" : "Adidas",
      "category" : "Footwear"
    }
  },

But sometimes, the word shoes may not always appear in the product's name. Often, sellers may name them sneakers or something in the name, but will put in the word shoes in the product's description.

In such a case, you'll need to search across multiple columns/fields. That's where the multi-match query can help.

Here's the syntax:

GET products/_search
{
  "query": {
    "multi_match": {
      "query": "Search keywords",
      "fields": ["field1", "field2"]
    }
  }
}

So here's how you can retrieve products containing the word shoes in both the name and the description:

GET products/_search
{
  "size": 100,
  "query": {
    "multi_match": {
      "query": "shoes",
      "fields": ["product_name", "description"]
    }
  }
}

Query #2: Text search for a product with a brand name or a certain phrase

Sometimes, a product name contains 2 or more words, in which case the order of the words that you type become important. Here are a few examples:

Microsoft Surface
United Colors of Benetton

When a search phrase containing 2 or more words need to appear in the same order, and they must appear next to each other, a Match phrase query is useful.

GET name-of-index/_search
{
  "query": {
    "match_phrase": {
      "Column/field within which to search": {
        "query": "Search phrase"
      }
    }
  }
}

Here's a query that searches for Kindle Paperwhite:

GET products/_search
{
  "size": 100,
  "query": {
    "match_phrase": {
      "product_name": {
        "query": "Kindle Paperwhite"
      }
    }
  }
}

But this need not be limited to just the product_name field. Here's the syntax for searching for a phrase across multiple fields:

GET name-of-index/_search
{
  "query": {
    "multi_match": {
      "query": "Search phrase",
      "fields": [
        "field_1",
        "field_2",
        "field_3"
      ],
      "type": "phrase"
    }
  }
}

Here's an example:

GET products/_search
{
  "query": {
    "multi_match": {
      "query": "Kindle Paperwhite",
      "fields": [
        "product_name",
        "description",
      ],
      "type": "phrase"
    }
  }
}

Query #3: Search for a particular category of products

Often, shoppers will click on a particular category on the home page to search for products and browse through them, as opposed to typing a particular search phrase. This can be done using a term query. Here's the syntax:

GET name-of-index/_search
{
  "query": {
    "term": {
      "name-of-field": "keyword",
    }
  }
}

In our dataset, the category field is of the keyword type, which makes it possible for us to use a term query.

So in the above picture, when a user clicks on the Grocery category, the following query can be used to fetch and display grocery items:

GET products/_search
{
  "query": {
    "term" : { "category" : "Grocery" } 
  }
}

Compound Queries

Compound queries allow you to combine 2 or more queries into more sophisticated ones. For instance:

Use a text search to look up products, but only within a certain category or categories.
Look up a product within a certain price range, or above a certain rating.
Or all of the above!

For this, a bool query comes in handy. It allows you to combine multiple queries into a single one wih the help of boolean clauses:

must: All queries in the must clause must be satisfied for a document to be returned as a hit.
should: The should allows you to specify nice-to-haves. In other words, it will still return documents that don't match those nice-to-have or desirable conditions, but if they do, they will be ranked above than the ones that don't match.
must_not: You can tell elasticsearch to not include certain products if they match some conditions.
filter: This behaves exactly like the must query, except that the score of the matching documents isn't computed. When the score doesn't matter and all you care about is whether documents match the filter criteria, it is recommended to use this query. Elasticsearch automatically caches frequently used filter queries to give them a performance boost.

Let's look at how we can use compound queries to map out a user's search journey.

Here's a user searching for the smartphone "Redmi Note 10" in the Smartphones category on Amazon:

Query #4: Text search for products within a certain category

In-order to make search results relevant, shoppers often search for products within a certain department.

As an example, the following query can be used to search for "Redmi Note 10" in the "Smartphones" category:

GET products/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match_phrase": {
            "product_name": "Redmi Note 10" 
          }
        },
        {
          "term": {
            "category": "Smartphones" 
          }
        }
      ]
    }
  }
}

Here's the general syntax:

GET name-of-index/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match/match_phrase/term": {
            "name-of-field": "value" 
          }
        },
        {
          "match/match_phrase/term": {
            "name-of-field": "value" 
          }
        }
      ]
    }
  }
}

Query #5: Search and filter for products by ratings

For this, we can use the range query. Here's the general syntax:

GET name-of-index/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match/match_phrase/term": {
            "name": "Enter the value you are looking for" 
          }
        },
        {
          "range":{
            "field-name": {
              "gte": "Enter lowest value of the range here",
              "lte": "Enter highest value of the range here"
            }
          }
        }
      ]
    }
  }
}

Here's how we apply it:

GET products/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match_phrase": {
            "product_name": "Redmi Note 10" 
          }
        },
        {
          "term": {
            "category": "Smartphones" 
          }
        },
        {
          "range":{
            "product_rating": {
              "gte": "4",
            }
          }
        }
      ]
    }
  }
}

Query #6: Filter by pricing parameters

Similarly, applying a price range works the same way:

GET products/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match_phrase": {
            "product_name": "Redmi Note 10" 
          }
        },
        {
          "term": {
            "category": "Smartphones" 
          }
        },
        {
          "range": {
            "product_rating": {
              "gte": "4",
            }
          }
        },
        {
          "range": {
            "retail_price": {
              "gte": 300,
              "lte": 450
            }
          }
        }
      ]
    }
  }
}

Sorting search results

By default, Elasticsearch returns a list of documents sorted by most relevant to least relevant. In online shopping however, shoppers like to view search results sorted by pricing, or customer ratings as well.

Here's the general syntax:

GET products/_search
{
  "sort" : [
    { "field-name" : {"order": "specify-a-sort-order"}}
  ],
  "query": {
    ...
  }
}

Here's how we can apply it in the previous query:

GET products/_search
{
  "sort" : [
    {
      "retail_price" : {
        "order": "asc"
      }
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match_phrase": {
            "product_name": "Redmi Note 10" 
          }
        },
        {
          "term": {
            "category": "Smartphones" 
          }
        },
        {
          "range": {
            "product_rating": {
              "gte": "4",
            }
          }
        },
        {
          "range": {
            "retail_price": {
              "gte": 300,
              "lte": 450
            }
          }
        }
      ]
    }
  }
}

Improving search relevance

When you search for something, you want to see the most relevant search results on the top. It can be frustrating when the search results are not exactly what you were looking for.

Search engines like the one that you see on Google's home page determine the relevance of your query using a variety of criteria - such as the text input, geolocation, accuracy of your spelling, and even the intent by analyzing your browsing patterns using machine learning techniques. In-fact, they can be so sophisticated that they fall in a category of their own - recommendation engines.

Search relevance is the measure of accuracy of the relationship between the search query and the search results. Which raises the question -

How do we measure the relevance of the results returned from a search query?

To understand this better, let's look at the characteristics of the data that may or may not be returned by the search engine.

In the above diagram, the dots represent the entire dataset stored in our database. When we search for something using a search engine, the dots within the circle indicate the document that is selected by the search engine as being relevant, also called hits. A ranking algorithm assigns each of these hits a score, after which they are returned back to the user.

True positives: These are data that are truly relevant to the user, and are also identified by the search engine as relevant and returned to the user as the response.

False positives: These are data that are not relevant to the search query, but are still marked by the search engine as being relevant and returned back.

True negatives: These are documents that are not relevant, and are correctly omitted by the search engine amongst the returned results.

False negatives: These are documents that are in-fact relevant to the search query, but were not identified by the search engine. Therefore they were not returned back in the search results, when they should have been returned.

Broadly speaking, there are two factors - precision and recall - that are used to measure the relevancy of search results.

Precision

Precision consists of the dots within the circle in the above diagram. It tells us what proportion of results that were identified as relevant by the search engine truly are relevant. It is a ratio of true positives (documents identified as relevant, and which are actually relevant) to all positives (all documents identified as relevant, irrespective of whether that was correct). In other words, it is the ratio of True Positives/(True Positives + False Positives).

Recall

Recall or sensitivity, is another metric that's about the dots within the left part of the above diagram. It tells us what proportion of documents that actually are relevant to our search query, were identified by the search engine as being relevant. It is a ratio of true positives (documents identified by the search engine as relevant, and which are actually relevant) to all the data that are relevant (irrespective of whether the search engine identified them correctly). The formula for this metric is given by - True Positives/(True Positives + False Negatives).

Precision helps you understand what portion of data returned, are actually relevant. Whereas on the other hand, recall helps you understand what portion of relevant data are *actually being returned. Together, these two factors determine the relevancy and quality of your search results.

In Elasticsearch, you can fine-tune your queries to be more precise, or have more recall. Since these two metrics are at odds with each other, fine-tuning for precision would make the search engine err on the side of returning less documents and focus more on quality, whereas whereas fine-tuning for relevance would make it focus more on quantity.

Let's look at how we can improve the relevance of our search:

Boost the relevance of individual fields

If a search term occurs in specific fields of a document, you can tell Elasticsearch to boost their relevance score higher. Let's see how this can be used within the context of grocery shopping.

I'm in the mood to have some strawberries. Let's see if I can find any fresh ones on Walmart! 😋

Notice how the search results show the actual fruits at the top, and shows products like mixed fruits, dried fruit etc towards the bottom?

When I click on the product titled Mixed fruit, here's what the description reads:

It seems like when you search for something like strawberries on Walmart, it searches across both the product name as well as the description fields. But its also clever enough to rank the products that contain the word strawberries in their titles higher, than the ones that have this word in their descriptions.

Why? Because products mentioning the word "strawberries" in their titles are more likely to be related to my search, than the products that mention "strawberries" in their description.

Elasticsearch allows you to boost individual fields in-order to improve the precision of your search results. So if we were to apply boosting on our dataset, here's what it would look like:

GET products/_search
{
  "query": {
    "multi_match": {
      "query": "strawberries",
      "fields": [
        "product_name^2",
        "description",
      ]
    }
  }
}

This basically tells Elasticsearch to increase the weight of the product_name by a factor of 2. Here's the general syntax:

GET name-of-index/_search
{
  "query" : {
    "multi_match": {
      "query": "search term",
      "fields": [
        "field1^2",
        "field2",
        "field3"
      ]
    }
  }
}

Decrease the relevance of documents for certain search terms

There might be instances when you'd like to decrease the relevance of certain documents if they contain some search terms, without excluding them from the search results. Here's an example:

The top three products are indeed laptops, but the 4th one isn't. Notice how the search lowered the relevance of a related product that contains the word laptop, but isn't actually one? But Amazon very aptly thinks that this is something that I might be interested to buy, so it didn't exclude it from the search results!

Elasticsearch allows you to apply negative boosting to demote certain documents without excluding them from the search results, with the help of the boosting query. Here's how its done:

GET products/_search
{
  "query": {
    "boosting": {
      "positive": {
        "term": {
          "text": "laptop"
        }
      },
      "negative": {
        "term": {
          "text": "usb"
        }
      },
      "negative_boost": 2
    }
  }
}

Here's the general syntax:

GET name-of-index/_search
{
  "query" : {
    "boosting": {
      "positive": {
        "term": {
          "text": "search term"
        }
      },
      "negative": {
        "term": {
          "text": "term to demote"
        }
      },
      "negative_boost": 2
    }
  }
}

Tip: Use the should clause if you want to boost documents that contain certain additional terms.

Enhancing e-shopping experience even further

Most shoppers prefer using mobile devices. But typing on the search box isn't nearly so easy as it is on a desktop! It take a lot more effort, and you're more prone to making typos. A study from Grammarly shows that mobile users are nearly 5 times more likely to make mistakes while typing on a phone than on a regular PC!

Here are some cool features of Elasticsearch that you can leverage to further improve search on mobile:

Suggesters: This adds an autocomplete suggestion feature, providing relevant search results as the user keeps typing on their search box!

Fuzzy queries: Useful for handling typos and to provide "Did you mean this instead" feature. Guessing what the user actually intended to search for rather than returning no results at all is a great way to further enhance the shopping experience.

Summing up

If you have a retail platform, providing a means to search for products quickly and easily can immensely enhance the shopping experience for your users. This translates to more frequent purchases, better customer reviews and increased profits.

At Egen, we use Elasticsearch along with several other technologies to create captivating customer experiences for our retail customers. Here's a sneak peak into the life of an Elasticsearch Architect in our organization:

Looking to create an amazing user experience for your retail application? Or have any questions related to this post? Do leave your thoughts on the comment section!

Resources

What the heck is a Service Mesh, anyway?

Ashwin Hariharan — Thu, 23 Mar 2023 10:38:38 +0000

Software applications can be thought as an ever-evolving organism. As the number of active users for your apps increase, at some point you're going to eventually consider the problem of scale.

One of the solutions that gradually evolved over the years to manage the challenges of scale were microservices. The design principle of a micro-service based architecture is to treat the application as a collection of individual, loosely-coupled services. This offers several advantages over monoliths-faster development, better deployment, fault tolerance, security and more to name a few.

But designing a microservice-based architecture comes with its own set up challenges. For instance:

How do you manage and visualize traffic flows between microservices?
How do you keep the data flow between microservices secure?
Observability - how do measure the internal states of a particular service?
Monitoring and tracing of network requests.

For this, you need something called Service Mesh. But first, let's first look at how a typical cluster is implemented using Kubernetes.

A conceptual container orchestration setup

Here is a diagram of a typical cluster in Kubernetes:

Here, microservice A could for instance, use the services of microservice B, which in-turn could rely on another microservice C. A typical architecture built using Kubernetes would involve the Pod A making a network request to Pod B, and in-turn would make a request to Pod C.

A container orchestration tool like Kubernetes helps you manage and handle the pods, but it has little control over the actual data or network flow between pods. What's going on at the network connections when each individual pod makes a request to another pod?

In a cluster with dozens or even hundreds of pods, you will need visibility on the interconnections between the pods themselves. Here is where a service mesh comes into play.

What is a Service Mesh?

A Service Mesh is an extra layer of abstraction that you add to your Cluster of services, in-order to efficiently manage and monitor them.

For instance, if you're using Kubernetes, you can think of a service mesh as this layer of software that resides underneath your kubernetes pods. This software takes over control over every network request made between pods in the entire cluster.

Wait, but why though?

The service mesh software intercepts each incoming network request, and re-routes it to the desired container. In-doing so, it can implement some function or some mesh logic, that uses the network request and do some pretty interesting stuff. For instance:

Measure the time taken for the network request to complete.
Collate network errors that occur in any part of the cluster.
Implement security - for instance, all network calls must be encrypted.
Re-route requests based on a given criteria or when certain conditions are met.
Visualize and manage network traffic

Without a service mesh, it would be pretty difficult to implement the above.

Indeed, the most useful features of a service mesh involves telemetry. Telemetry involves monitoring the health and security of microservices, measuring their performance, and collecting useful statistics such as the memory overhead, CPU utilization and more.

Another feature of a service mesh is tracing. This would allow you to visualize the network flow and trace the chain of requests within the context of a particular feature or service.The service mesh could gather such useful information that would enable for faster debugging and bug resolution, or even run a time series analysis to predict future outages of a particular service.

Istio

Istio is a Kubernetes native solution, used by several large scale corporations like Google, IBM and Microsoft. It has support for K8s and also virtual machines.

There are many different ways by which a service mesh could be implemented. Here's how Istio implements it:

For each of the pods in the cluster, Istio injects its own container, also known as a proxy. Each individual proxy takes over and handles all incoming and outgoing network requests for that pod. This proxy contains the mesh logic to further process the network request for tracing, telemetry, and re-direct it to the target container proxy.

Istio is designed to be loosely coupled. The individual containers have no knowledge about the proxy injected by Istio, and neither do they need to contain any code for traffic management, tracing or other features provided by Istio. The individual pods continue to run and make network calls as though they were directly communicating to other pods.

This design of service mesh implemented by Istio is also known as the Sidecar pattern.

Other implementations

Similar to Istio, there are other popular implementations as well:

Linkerd: Claims to be the lightest and simpler than any other service mesh implementation. It's K8s only. Has its own proxy written in Rust.
Consul: A full-feature service management system. Runs as a Daemon on every K8s node and works with Envoy.

Here's an article that compares these implementations.

Istio under the hood

Istio actually assembles together several third party components in-order to construct a service-mesh for your cluster.

Data plane

Istio uses Envoy, an open-source cloud-native service proxy under the hood for the Proxy containers within your pods. These proxies constitute the data plane layer of Istio.

Control plane

The control plane consists of majorly 3 components:

Galley: If you're using Kubernetes, Galley reads your Kubernetes YAML file and transforms it into a format that Istio understands. It can also work with other orchestration frameworks such as Mesos and Consul.

Pilot: This component takes the configurations and then converts them into a format that Envoy understands, in-order to deploy the proxies.

Citadel: Responsible for certificate management and enabling TLS / SSL across your entire cluster.

Since Istio version 1.5, the above components have been combined into a single binary called istiod.

At Egen, we've put together a video in-order to understand fully well how Istio works, along with use-cases showing what you can do with it. We also show you a real-world example of a service not working as intended, and how Istio can help you narrow down the problem and apply an appropriate fix.

Thanks for reading

We host several such meetups discussing scalability, distributed systems, application engineering and more. If you're interested in learning these topics, do give us a follow on collective.ac to get notified for future events.

Resources

DevRel job roles - Content Creator or Content Developer?

Ashwin Hariharan — Sat, 12 Mar 2022 13:37:22 +0000

Developer Relations is a niche field in software engineering. It is an umbrella term for a lot of activities that are geared towards building relationships with developers.

I've been in this field for almost 2 years, and one of the most significant activities as part of my work is to create content, mostly in the form of technical deep dives. But to say that I'm a content writer or technical writer seems like an understatement, and I feel like I might be under-selling myself.

There are several more activities that I (or anyone else in this field) could be involved in. Here are some examples (in no particular order):

Before even beginning to create content, one would need to do a lot of research, and planning what they create and how they'll execute it in the first place.
Sometimes the content might be talking about tech, but isn't necessarily a coding tutorial. For example - if my organization is into Retail, I would need to write articles like "What challenges do online retailers face when trying to scale up?" That again involves research and acquiring industry knowledge.
For the actual content, it's not always text-based. Often someone who's doing the writing also might be involved in creating infographics, and even explore other formats like videos.
Working with Director of Talent for coming up with creative job ads for hiring developers.
Working with the Head of Marketing to decide on factors like SEO, and the right platforms to publish the content on.

So if a person working in DevRel does the above activities and need to describe themselves in short, what should they call themselves? Would Content Creator make more sense, or Content Developer? Or are there are other titles that would work better?

What do you think? Let's discuss in the comments below - I'd love to know your thoughts! 😄

How to build your own Social Audio Chat Application - Part 2

Ashwin Hariharan — Wed, 23 Feb 2022 11:28:15 +0000

What you will learn and build
Components for enabling real-time communication using audio or video
How to implement a Signaling Server
What are web sockets, and how do they work
Feature #1: Enter a room to participate in ongoing conversations
Feature #2: Leave a room
Next steps
- Some features that are worth replicating
- Architecture, Scaling and costs
References and resources

Welcome to Part 2 of this series on how to build a social-networking app that enables you to have conversations in real-time using audio. In this tutorial, you'll continue to build a social audio application similar to Clubhouse and Spotify Greenrooms.

To recap, here's the list of features:

Feature #1: Discover new rooms in the hallway. You already built this in Part 1.

Feature #2: Users can enter a room to talk, and have conversations with people with similar passions and interests.

Feature #3: A user can opt to leave a room anytime they feel like on the click of a button

Make sure to read part 1 thoroughly before you proceed!

How to build your own Social Audio chat application

Ashwin Hariharan for Egen ・ Feb 23 '22

#tutorial #webdev #node #postgres

What you will learn and Build

In this tutorial, you'll learn how to mimic and build the feature that enables users to join a room and interact with each other via audio in real time! Here's a side-by-side comparison of this room feature across different apps — Clubhouse, Spotify Greenrooms and Twitter Spaces:

Components for enabling real-time communication using audio or video

For enabling real-time communication, you need a few key components. Here are the most basic components for a peer-to-peer network architecture ¹:

A Media Input and Output Device

If you're reading this on a computer or a smartphone, your gadget very likely comes along with a microphone, speakers and a webcam. Most modern browsers also can request the user for permission to access one of these input devices using JavaScript.

WebRTC

A technology that is built in open web standards, and is part of most modern browsers. Using WebRTC, you can transmit audio, video and generic data directly between different clients.

WebRTC can be a bit complicated to use. So for this tutorial, you'll make use of PeerJS, a library that abstracts away all the underlying implementation complexity so that you can focus on building the functionality.

A signaling server

For WebRTC to work, each client (a web browser, or an Android or an iOS application) will need the peer information ² of every other client in the room or network in-order to establish a webrtc peer connection. A signaling server's job is to let peers in the room or network find each other by providing information about each peer in the room, so that they can establish a WebRTC connection.

The WebRTC specification does not provide APIs to implement this signaling mechanism. So it is left up-to us developers to figure out a way to provide information of the peers to every other peer in the network, so that they can communicate with each other. But fret not, there are a lot of techniques you could use!

How to implement a Signaling Server

Technique #1: Using a database that supports real time updates

The Codelab in the official WebRTC website describes how to implement the signaling server using this very technique, with the help of the Cloud Firestore database. Here's a brief description of how it works:

On the client-side — when you join a room, a new entry is made in the database with your user information such as your username, name, and most importantly — your peer information.
Other users (clients) within the room are connected to this database, and are listening for changes and updates. When they detect that a new participant (a.k.a you) has been added to the database, they receive your information along with your peer information.
Now using this peer information, these clients can establish a peer connection with you using WebRTC protocols.

Any database that allows subscribing or listening to database changes or updates can be used as a signalling server to transmit the peer information between clients.

Technique #2: Using regular HTTP requests

Here's how this technique could work:

Upon joining a room, the client makes a POST request to create a new entry in the database with the new particpant's details and peer information.
Other clients in the room would either:

Use a regular polling technique, where they keep making network requests to an HTTP API endpoint at every given time-interval, to check if any new participant has joined or not. OR
Use a long-polling technique, where the server keeps the connection alive for as long as it can until it can send some new data to the client.

As you can very well imagine, regular HTTP requests have several drawbacks:

Drawbacks

Long-polling can be highly resource intensive especially when there are multiple clients connected with the server.
Regular polling technique isn't really “real time”. And again, the shorter the intervals, the more resource intensive it can be!

Traditional HTTP requests like GET and POST initiate a new connection request and later close the connection after the server sends back the response. If you were to attempt building a real time app using HTTP, the client would have to initiate requests at regular intervals to check for new information (which may or may not be available). This is because of the fact that the server itself is unable to push information on its own.

And this is highly inefficient — the client would waste resources in constantly interrupting the server and saying “Hi, I'm XYZ — let's shake hands. Do you have something new for me?”, and the server will be like — “Hi (shaking hands). No I don't. Good-bye!” over and over again, which means even the server is wasting resources!

So moving on to another technique —

Technique #3: Using Web sockets

This is perhaps the most common way to implement a signalling server.

What are Web Sockets, and how do they work?

Using Sockets, you can send and receive information via events, or in other words signals. There can be different types of such signals, and if the parties involved know what kind of signal to ‘listen' to, then there can be an exchange of information.

Unlike HTTP requests, web-sockets create a persistent connection between a client and the server. So this way the client need not keep asking the server, the server can push information when it needs to. This method is much more efficient for building real time applications.

For implementing the signalling server, you'll make use of web-sockets with the help of socket.io. Here's how this technique would work:

Upon joining a room, the client emits a web-socket event with its peer information and other details. Let's call this event user-joined-room .
The web-socket enabled server listens for any user-joined-room events. When it receives the event, it makes a new entry in the database, and adds the new participant's socket to the room or channel. Next, it broadcasts a web-socket event and sends the new participant's info and peer details to all connected clients within that channel.
Any clients within the room are also listening for a user-joined-event. As soon as the server broadcasts the new participant's information from the previous step, they receive the peer details and then can use it to initiate a WebRTC connection.

Feature #1: Enter a room to participate in ongoing conversations

In the last tutorial, you replicated the hallways feature, where you see a list of rooms from the clubs that you follow. Now when a user clicks on a room, they should be able to enter the room, listen to other speakers within that room, and be able to participate in the conversation.

Step #1: Adding details of a new participant in the database, and sharing their peer information with all clients in the room

Backend

First, we'll write a function that adds takes in information of the new participant and creates a new entry in the participants table.participant to the Add the following code inside /models/participants.js:

async function addParticipant(participant) {
    try {
        const text = `
            INSERT INTO participants (room_id, user_id, role, peer_id)
            VALUES ($1, $2, $3, $4)
            ON CONFLICT (room_id, user_id)
            DO
                UPDATE
                    SET join_date = NOW(), peer_id = $4
                WHERE participants.room_id = $1 AND participants.user_id = $2
            RETURNING *
        `;

        const values = [participant.room_id, participant.user_id, participant.role, participant.peer_id];

        const res = await db.query(text, values);
        return res.rows[0];  

    } catch (err) {
        console.log(err.stack);
    }
}

If the participant already exists, we just update their join-date field to the current timestamp.

Next, you'll configure our web server to listen for any user-joined-room events. When it receives an event:

You add/subscribe the new participant's socket to the room that they wish to join.
Next, you'll use the above function to add the participant information in the database.
Then, you emit a user-joined-room event to the rest of the clients within that room. That way, any clients that are subscribed to this room and listening to this event will receive the participant's details.
Finally, you'll update the participant that they've joined the room by using a callback.

Open socket-events.js and write the following logic:

socket.on('user-joined-room', async (eventInfo, callback) => {
    socket.join(eventInfo.roomId);

    const participant = await db.addParticipant({
        room_id: eventInfo.roomId,
        user_id: eventInfo.user.id,
        role: 'SPEAKER',
        peer_id: eventInfo.peer_id,
    });

    participant.name = eventInfo.user.name;

    socket.to(participant.room_id).emit('user-joined-room', participant);

    callback(participant);
});

For keeping things simple, we'll add a new participant as a speaker by default. In the real Clubhouse app however, when a new user joins a room, they are part of the audience and need to request the room moderators in-order to be added to the speaker panel.

Front-end

First, you'll use the peerjs constructor to issue the current client a unique Peer ID. This ID will be necessary for establishing a webrtc connection with another peer.

Inside modules/webrtc.js, add the following code:

export const peer = new Peer({
    host: location.hostname,
    port: location.port || (location.protocol === 'https:' ? 443 : 80),
    path: '/peerjs'
});

peer.on('open', (id) => {
    console.log('My peer ID is: ' + id);
    updatePeerId(id); //save this ID
});

Our NodeJS web-server is configured to issue a new Peer ID upon calling new Peer
When a connection to the Peer-server is established, it generates a unique random identifier string. We'll save this ID in a variable PEER_ID, which we can use later.

Next, you'll write a function joinEvent that you'll call when a room is selected. Here's how it works:

Emit a socket event user-joined-room, and send the room-id, the user information, and the PEER_ID .
The signalling logic that you wrote earlier will receive this event, and create a new entry in the participants table, and will notify the client once its done.
Once the client has been notified, you'll make a network request to fetch the room info, and then display it in the UI. The user is now officially in the room!

Add the following code inside /modules/index.js:

function joinRoom(roomId) {
    socket.emit('user-joined-room', {
        roomId,
        user: USER_INFO,
        peer_id: PEER_ID,
    }, async (response) => { //callback
        updateParticipationInfo(response);
        console.log('PARTICIPATION_INFO', PARTICIPATION_INFO);

        fetchRoomDetails(roomId).then(result => { // fetch room info
            renderRoomDetailsSection(ROOM_INFO); // display room
        });
    });
}

Now, you'll just call the above method when a room card is clicked:

export async function onRoomSelect(e) {
    if (USER_INFO.id && PEER_ID) {
        selectRoomNode(e.currentTarget); // Highlights the selected room
        const roomId = e.currentTarget.getAttribute('data-id');
        joinRoom(roomId);
    } 
}

Similarly, other clients connected to this room would also need to know if a new participant has joined. Add the following code:

socket.on('user-joined-room', handleNewParticipantJoin);

async function handleNewParticipantJoin(participantInfo) {
    console.log('user-joined-room', participantInfo);

    if (participantInfo.role === 'AUDIENCE') {
        ROOM_INFO.audience.push(participantInfo);
        const audienceList = document.getElementById('audience');
        displayParticipant(audienceList, participantInfo);
    } else {
        ROOM_INFO.panel.push(participantInfo);
        const panelMembersList = document.getElementById('panel-members');
        displayParticipant(panelMembersList, participantInfo);
    } 
}

Try testing this out!

Open two new browser tabs on Chrome: http://localhost:3000?username=rick and http://localhost:3000?username=elon.
Scroll through the list of rooms, and select one that's common between the two to join.
You should be able to see the names of the participants appear one-by-one, in the order by which they join.

Now that you've successfully managed to exchange participants information with each other in a room! Next, you can use this peer information to stream information back-and-forth:

Step #2: Connect with the peers in the room, and stream their audio

In the previous step, we merely iterated through the list of speakers in the panel and displayed their names in the UI. Along with displaying their name as a text, we'll now also do the following:

Basically, any new participant that joins the room will also be the one to initiate the peer connection. So upon joining the room, you'll iterate through the list of participants. If they're not the current client/user:

Initiate a call using the participant's peer-id. If the current user is part of the panel-list, then stream their audio as well
If the participant is also part of the panel, then add an audio element in the UI, and attach their audio stream to this element so that the current user can hear them speaking.

Add the following code inside modules/helpers.js:

async function displayAndConnectWithRoomParticipant(
    containerElement,
    participantInfo
) {
    if (participantInfo.peer_id) {
        const participant = displayParticipant(
            containerElement,
            participantInfo
        );

        // establish peer connection
        if (participantInfo.user_id.toString() !== USER_INFO.id.toString()) {
            let mediaStream;

            if (PARTICIPATION_INFO.role !== 'AUDIENCE') {
                mediaStream = await openUserMedia();
            } else {
                mediaStream = new MediaStream();
            }

            if (participantInfo.role !== 'AUDIENCE') {
                const participantAudio = createParticipantAudio(participantInfo.user_id);
                participant.appendChild(participantAudio);

                const call = peer.call(
                    participantInfo.peer_id,
                    mediaStream,
                    {
                        metadata: {
                            participantInfo: PARTICIPATION_INFO
                        }
                    }
                );

                call.on(
                    'stream', attachAudioStream(participantInfo.user_id, participantAudio)
                );

            } else {
                const call = peer.call(
                    participantInfo.peer_id,
                    mediaStream, {
                        metadata: {
                            participantInfo: PARTICIPATION_INFO
                        }
                    }
                );
            }      
        }
    }
}

When the other participants receive the peer connection request, they'll need to accept the call as well.

Add the following inside modules/webrtc.js:

peer.on('call', async function(call) {
    const participantUserId = call.metadata.participantInfo.user_id;
    console.info(`call request from ${participantUserId}`); // if call requester part of speaker panel,stream their audio

    if (call.metadata.participantInfo.role !== 'AUDIENCE') {
        const participant = document.getElementById(`participant-${participantUserId}`);
        const participantAudio = createParticipantAudio(participantUserId);
        participant.appendChild(participantAudio);

        call.on(
            'stream',
            attachAudioStream(participantUserId, participantAudio)
        );
    }

    // if part of speaker panel, send own audio stream
    if (PARTICIPATION_INFO.role !== 'AUDIENCE') {
        const localStream = await openUserMedia();
        console.log('Answer the call');
        call.answer(localStream);
    } else {
        call.answer();
    }
});

This way, the other peer responds to a connection request and similarly exchanges their own audio information with the other peer. And we have a 2 way communication / audio flow established!

Try testing this out!

This time, you'll need the help of a friend to test the feature. You can use localtunnel to generate a unique URL that exposes your localhost server temporarily to the internet.
Just like earlier, Open a browser tab localtunnelurl?username=rick on Chrome, and share another URL localtunnelurl?username=elon with your friend so that they may open it on their browser.
When you both join the same room, you should be able to talk to each other!

Feature #2: Leave a room

Here's the breakdown of the application flow:

Upon leaving a room, the client emits a web-socket event user-left-room , with its peer information and other details. It'll also close all the peer connections between the rest of the participants in the room.
The web-socket enabled server listens for any user-left-room events. When it receives the event, it'll unsubscribe the user's socket from the room, and remove the participant from the database. Next, it broadcasts a web-socket event to notify the other participants in the room that the client has left.
Any clients within the room are also listening for a user-left-room event. As soon as they receive the notification from the server, they simply remove the user from the UI.

Front-end

On the current client, upon leaving a room you'll emit a user-left-room event, close the existing peer connections, and update the UI and reset the store:

export function leaveRoom() {
    if (PARTICIPATION_INFO.user_id) {
        socket.emit('user-left-room', PARTICIPATION_INFO);
        closeConnections();
        updateParticipationInfo(null);
        resetRoomDetails();
        unselectRoomNodes();
    }  
}

document.getElementById('leave-room').addEventListener('click', leaveRoom);

The other clients would also need to be notified when a participant leaves, and update their UI as well:

socket.on('user-left-room', handleParticipantLeave);

async function handleParticipantLeave(participantInfo) {
    console.log('user-left-room', participantInfo);
    if (participantInfo.role === 'AUDIENCE') {
        ROOM_INFO.audience = ROOM_INFO.audience.filter(obj => obj.user_id !== participantInfo.user_id);
    } else {
        ROOM_INFO.panel = ROOM_INFO.panel.filter(obj => obj.user_id !== participantInfo.user_id);
    } removeParticipantNode(participantInfo);
}

Backend

You'll need two functions in models/participants.js:

A function removeParticipant that deletes a participant from the database
A function updateParticipant that updates a participant's peer information in the database and sets it to null. We'll use this method if the user is the room's host.

async function removeParticipant(roomId, userId) {
    try {
        const text = `
            DELETE FROM participants
            WHERE room_id = $1 AND user_id = $2
            RETURNING *
        `;

        const values = [roomId, userId];

        const res = await db.query(text, values);
        return res.rows[0];

    } catch (err) {
        console.log(err.stack);
    }
}

async function updateParticipant(roomId, userId, peerId) {
    try {
        const text = `  
        UPDATE participants SET peer_id = $1
        WHERE room_id = $2 AND user_id = $3
        RETURNING *
        `;

        const values = [peerId, roomId, userId];

        const res = await db.query(text, values);  
        return res.rows[0];  

    } catch (err) {
        console.log(err.stack);
    }
}

Be sure to specify the WHERE clause when you delete an entry from the table, else you risk deleting all the data!

Next, your signalling server logic in routes/socket-events.js:

socket.on('user-left-room', async (eventInfo) => {
    socket.leave(eventInfo.room_id);

    let roomParticipant;

    if (eventInfo.role !== 'HOST') {
        roomParticipant = await db.removeParticipant(eventInfo.room_id, eventInfo.user_id);
    } else {
        roomParticipant = await db.updateParticipant(eventInfo.room_id, eventInfo.user_id, null);
    }

    socket.to(eventInfo.room_id).emit('user-left-room', roomParticipant);
});

And that's pretty much it. You might need the help of your friends to test it out!

Next Steps

Obviously, what you build here is a hyper-simplified version of Clubhouse. But you can use this as a starting point and replicate more features.

Some features that are worth replicating

Creating a new room

Notice that in the bottom of the UI screen there's a button that says “Start a room”? Well, you can code the functionality for it:

On the UI, display a form where you can enter in the name of the new room, and optionally assign it to an existing Club.
On the backend, create an API that takes in the name of the room, and adds a new room entry in the database

Creating a new club

Similarly, you could also add a feature that let's people start their own clubs. Might even throw in an invite feature that lets club creators invite other users to follow a member or become a member.

Memberships

Remember how a user can be a follower in other club? Well, turns out that in the actual app, a user can also be a member of many different clubs. Starting a room within a specific club requires the user to be a member of that club.

When you incorporate memberships in your ERD, it would look like this:

Any guesses on what type of relationship exists between a user (member) and a club? HINT: It's similar to the followers relationship that you read in the previous article

And several more features!

Mute/un-mute yourself within a room
Request to speak — new participants get added to the audience list by default, and will get added to the speaker panel on request.
A Chat messaging feature

When deploying an app in production for real users, there are several other considerations as well.

Architectures, Scaling and Costs

Peer-to-peer (P2p) network

What we covered in the tutorial was an example of a WebRTC implementation using a peer-to-peer (P2P) or a mesh network architecture. In this, each peer or participant that is a speaker upstreams their media (audio, video or both) to n-1 other peers in the network. Every peer also needs to downstream the media from other peers so that they may consume it. This is decentralized in nature, as no central server is being used to route media information.

Advantages

Easy to understand and setup.
Low set-up costs and less resource intensive on the server side, because the server isn't being used to process media streams.
Better privacy due to decentralization — because the data (audio) is upstreamed and downstreamed directly by the clients in the network, without passing via a server.

But this setup also has drawbacks

Drawbacks

A mesh network suffers in streaming data reliably as the number of peers or participants in the network increase. More number of peers will overwhelm the bandwidth because they'd need to stream their audio or video to n-1 other peers. Besides, cameras that can capture video in 4K are becoming commonplace — and if you're building a video streaming platform, then this can cause issues like audio and video lags.
Besides, most browsers can only reliably support a fixed number of connections — for instance as of today, the officially supported limit for Google Chrome is 500 connections.
Although less resource intensive on the server side, a mesh network can be more CPU intensive on the client side, because each peer would need to process data while upstreaming / downstreaming for (n-1) connections in the network
Better privacy also comes at a drawback of not being able to store/record conversations either.

So commercial products mostly don't rely on a peer-to-peer architecture. In-fact, Google Hangouts uses a peer-to-peer connection only for 2 participants.

So if you're trying to build and deploy a product for commercial purposes, and you anticipate a large number of persistent peer connections, there are other network architecture that need to be considered, which implement centralized video bridging techniques:

Selective Forwarding Unit (SFU)

Unlike the peer-to-peer architecture, in this network each participant sends their encrypted media streams only once to a centralized server, called an MCU server. This server then forwards those streams to the other participants, without any processing.

Advantages

The bandwidth to upstream media becomes quite less — the participant only needs to upload the information to the central server.
More scalable than peer-to-peer network due to the less number of connections required.
Easier to manage and store media data.

Disadvantages

Although more scalable than the peer-to-peer network, it still has limits.
Some of the computation power is now offloaded to the server side. So the setup costs become a lot more.
Not ideal for privacy, as media streams are being passed via a central server.
Having a central server also means that it can potentially be a single point-of-failure.

Multipoint Conferencing Unit (MCU)

Also called Multipoint Control Unit, this architecture is similar to that of SFU — each client is connected to a centralized server. Unlike SFU however, the MCU server decodes, rescales, and mixes all incoming streams into a single new stream and then encodes and sends it to all clients.

So any client in the network only need to upstream and downstream a single media stream.

Advantages

Extremely easy on bandwidth.
Much more scalable than peer-to-peer architecture

Disadvantages

Just like the SFU architecture, the drawbacks of the MCU architecture are similar. In addition to it:

Extremely CPU intensive on the server side — since the server not only just streams media information to all the connected peers, but now also needs to transform media streams from all participants before sending it.
Very high set-up costs
Since all media streams are combined into a single one, the user can't control or disable audio/video reception from a specific participant.

You might want to leverage a cloud-based WebRTC service provider for enabling real time communication, rather than implement your own WebRTC logic. That way, your app could easily support large numbers of users. It isn't uncommon for a Clubhouse room to have several thousands of users, especially when a celebrity is speaking!

There are several WebRTC service providers, like Agora.io, Twilio and Amazon Connect that offer SDKs with APIs to integrate WebRTC into your app. In-fact, when Clubhouse launched in 2020, they seem to have relied on Agora for powering their audio chat!

I'm curious to see what you build next!

Thank you for reading.

Here's the complete source code for reference. In case you have any questions regarding the tutorial, please leave a comment below!

If you're a committee or an organization and would like help in building more such solutions, reach out at digital@egen.solutions. As a digital-native company, we at Egen know how to scale up and deliver fast, which means that we can help you create and launch such apps in days instead of months!

One good conversation can shift the direction of change forever.

References & Resources

In other architectures, you'd also need several other components, such as a media server for encoding and decoding media streams. ↩
In the native WebRTC spec, this peer information is referred to as ICE candidates, standing for Internet Connectivity Establishment. The PeerJS library abstracts away all these details and instead provides you with a simple peerID string, which clients can use to establish a WebRTC connection. ↩

How to build your own Social Audio chat application

Ashwin Hariharan — Wed, 23 Feb 2022 11:23:16 +0000

What is Social Audio
What you will build

2.1 What you will learn
- Relational database design
- Real time communication
2.2 Tech stack
Designing the Entity-Relationship Model
Using Postgres to query for information
4.1 List all rooms
4.2 List rooms for those clubs that the user follows
4.3 Retrieve information for a single room
Building a web service

If you're reading this article, you mostly likely discovered it via some form of social media. They are platforms that enable people across the world to interact with each other and share information on what's happening in their own corners of the world. And for more than 3 decades since their advent, social media apps allowed sharing information mostly in the form of text or images, particularly when it comes to sharing content in real time.

In early 2020, an app called Clubhouse popped up in the iOS app store. It was a social media app, where the primary form of content was not photos, videos or text, but audio. By the end of the year in the midst of the COVID-19 pandemic, when social distancing and remote work had become the norm, the app exploded in popularity - gaining over half a million users!

And just like usual when a new feature in an app goes viral, tech giants in the social media space take a crack at cloning and offering it in an attempt to enamor their users! Ever since Clubhouse's first release in 2020, companies like Twitter, Facebook, Slack, Spotify, Discord and even Reddit have either been creating products that compete directly with Clubhouse, or adding Clubhouse-like features to their existing platforms. These features are collectively known as social audio - a new subclass of social media. Let's have look at what those features are.

What is Social Audio?

A Social Audio application can be thought of as part-radio, part house party, part conference-call, part-podcast — all rolled into one. Once you're inside such an app, you can listen to conversations happening within a room, or start one yourself on a topic or theme of your choice. The theme could be anything — interviewing a celebrity or an expert, a discussion on a niche topic like the future of AI, or simply chit-chat.

Here's a side-by-side comparison of Clubhouse and one of its competitor app, Spotify Greenroom —

Feature #1: Discover conversations

The home-screen on both the apps display a curated list of rooms for you based on your interests, and the clubs or groups that you follow. These rooms are spaces where conversations occur in real-time. There are some slight differences between what information is shown — but both apps show the room name, the number of participants, and give you a tiny preview of the list of speakers.

Feature #2: Participate in conversations

Once you click on a room in the previous screen, the app takes you to the room and you'll be able to listen to ongoing conversations. On the top, you see the panel members, followed by people who are part of the audience. If you're a host, or invited as a speaker or moderator, you'll be able to speak as well.

Feature #3: Leave a conversation

Both these apps offer the user to leave the room anytime at the click of a button. This will have the effect of returning you to the home screen.

That's pretty much the core features! Plenty of these apps also have additional features common in social media platforms, like exchanging chat messages, subscribing to topics that interest you, and also following other users and clubs.

Using your own voice to interact with people just feels very natural. This explains why during COVID-19 lockdowns, friends, relatives and colleagues started relying on apps like Zoom, Whatsapp Video, and Google Hangouts for connecting from time to time.

While the question of whether social audio apps will continue to remain popular remains to be seen, it's very interesting to learn the underlying engineering that makes them work! How does Clubhouse, for example, enable users to join rooms and socialize using their voice?

So if you're a developer and feel the need to scratch that programming itch, keep reading!

What you will build

In this tutorial, you'll learn to clone some of the most salient features of Clubhouse. You won't focus on scale or performance right now, but you'll develop an understanding of what core technologies make applications like these possible. At the end of this tutorial series, you will build the following features:

You'll attempt to mimic the Hallways feature in the Clubhouse app, where users are able to browse and discover active rooms.
Selecting a room to join it, where you can see all the other participants in the room and participate in the conversation.

This will be a programming tutorial mainly focused on functionality, so your focus will not be much on web design, (though you're welcome to do so if you wish!).

What you will learn

#1: Relational Database Design

You'll learn how to design and architecture a relational database system for representing the most essential data entities required for this app.

#2: Real Time Communication

You'll learn how to implement WebRTC to enable real time communication of audio, to enable live conversations within a club room.

Technologies that you will learn and use

On the front-end you'll use HTML, CSS and JavaScript to build the UI. You'll also use PeerJS — a library that allows you to easily for implement Real Time Transmission of audio/video in WebRTC.
On the back-end, you'll use NodeJS/Express for building web server and APIs. You'll also use Socket.io for implementing a signalling-server as is required in WebRTC based applications — of which you'll learn more about later.
You'll learn to use PostgreSQL as your database. It's fast, extremely extensible to a wide array of needs (including hyperlocal business applications), has support for both SQL and NO-SQL data models, and is scalable.

Prerequisites

It will be helpful to have at least a theoretical understanding of how relational database systems work.
Some knowledge of JavaScript and NodeJS is also useful.

Let's get started!

In this tutorial, you'll learn how to construct an efficient relational database model for organizing data and effective querying. You'll then build an API service that allows a client to retrieve information, which can then be displayed on a user interface.

Go to this link, and follow the instructions to download the repository and install all the necessary project dependencies. It contains some boilerplate code and helper methods for you to use and build the rest of the project as you go about the tutorial.

Designing the Entity-Relationship Model

An entity can be thought of as a category of data or information in a system, which is of particular interest to us. When you design an entity-relationship model for a system, you identify entities and their relationships with each other, which can be stored in a database.

Broadly speaking, your app has the following entities:

Users: Users can join a room and participate in an ongoing conversation.
Clubs: A club will usually be centered around a particular topic, and can consist of multiple live sessions or "rooms".
Rooms: A room is a live discussion where participants can tune in and listen, or speak too. A club can have multiple rooms.
Topics: A club can belong to a certain category, like Technology or Geopolitics. Any rooms created within a club will have conversations related to this topic. As such, there can be multiple clubs around a certain theme or topic.

We could imagine more entities in the system but these will suffice for now. If you built data models for each of the above entities, what would they look like? Take a moment to think!

Here's how our Entity Relationship Diagram could look like:

Entity Relationship Diagram

What we have here is an Entity-Relationship Diagram, or ERD for short. As the name suggests, it helps us visualize the various entities in a system and their relationships with each other. The above diagram is obviously a very simplistic version of how various entities within the actual Clubhouse app might look like and relate with each other. However, it still serves as a useful starting point!

Let's take a closer look at some of these relationships:

A club has exactly one founder. This is a one-to-one relationship.
A club has many rooms. That's a one-to-many relationship. A room however, may or may not be associated with a club.

Similarly, you can identify some more relationships:

A club can have many followers (users)
A user can follow many clubs

So there exists a many-to-many relationship between a user and a club! In ERD and relational-database terminology, this relationship is referred to as an associative entity, and is implemented by introducing an associative table or, cross-reference table. So in-order to denote this relationship, we'll have a third table which you can name followers, indicating the relationship:

Similarly, we have a relationship between a user and a room:

A room can have many participants (users)
A user could be a participant at many rooms

So there exists a many-to-many relationship between users and rooms. Just like before, you'll use a cross-reference table participants in-order to map out this relationship:

Designing the Schema in PostgreSQL

PostgreSQL is amongst the most popular relational databases used all over the world, being in the top 2 widely used databases for 2 years in a row in 2019 and 2020. It has a very high compliance with SQL standards, supports a wide range of native data types (including custom types as well as JSON!), and is extremely extensible.

Let's begin creating the above tables using the psql prompt. Once you've fired up the psql terminal, run the following command to create a new database:

CREATE DATABASE clubhouse;

Next, switch to this database so that the subsequent commands apply to this database by typing \c clubhouse;. Here are some commonly used queries:

\l: List available databases.
\dt: List existing tables.
SELECT * FROM tablename: Will print all rows and columns in a table.
DROP TABLE tablename: Will delete the table.
DROP DATABASE databasename: Deletes the database.

Next, you'll start by adding the following tables, one by one:

CREATE TABLE topics (
    id BIGINT GENERATED BY default AS IDENTITY PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    description VARCHAR(500)
);

CREATE TABLE users (
    id BIGINT GENERATED BY default AS IDENTITY PRIMARY KEY,
    username VARCHAR(50) NOT NULL UNIQUE,
    email VARCHAR(50) NOT NULL UNIQUE,
    name VARCHAR(100),
    bio VARCHAR(350),
    join_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE clubs (
    id BIGINT GENERATED BY default AS IDENTITY PRIMARY KEY,
    founder_id BIGINT NOT NULL references users(id),
    name VARCHAR(100) NOT NULL,
    topic_id BIGINT NOT NULL,
    date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

    FOREIGN KEY (topic_id) REFERENCES topics(id)
);

CREATE TABLE rooms (
    id BIGINT GENERATED BY default AS IDENTITY PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    club_id BIGINT references clubs(id) ON UPDATE CASCADE ON DELETE CASCADE,
    date TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE followers (
    club_id BIGINT REFERENCES clubs(id) NOT NULL,
    user_id BIGINT REFERENCES users(id) NOT NULL,
    UNIQUE(club_id, user_id),

    PRIMARY KEY (club_id, user_id),
    FOREIGN KEY (club_id) REFERENCES clubs(id) ON UPDATE CASCADE ON DELETE CASCADE,
    FOREIGN KEY (user_id) REFERENCES users(id) ON UPDATE CASCADE ON DELETE CASCADE
);

CREATE TABLE participants (
    room_id BIGINT REFERENCES rooms(id) NOT NULL,
    user_id BIGINT REFERENCES users(id) NOT NULL,
    role VARCHAR(10) DEFAULT 'AUDIENCE' NOT NULL CHECK(role IN ('HOST', 'MODERATOR', 'SPEAKER', 'AUDIENCE')),
    join_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    peer_id VARCHAR(50),
    UNIQUE(room_id, user_id),

    PRIMARY KEY (room_id, user_id),
    FOREIGN KEY (room_id) REFERENCES rooms(id) ON UPDATE CASCADE ON DELETE CASCADE,
    FOREIGN KEY (user_id) REFERENCES users(id) ON UPDATE CASCADE ON DELETE CASCADE
);

A participant could have one of the following roles:

Host — This would be those users who created the room and are in the main speaker panel. They also have the highest privileges within that room, including moderator privileges.
A moderator is also present in the speaker panel and can invite other speakers to the room.
A speaker is also part of the speaker panel, but cannot add or invite other users.
Finally, the other participants will be part of the audience. They can only listen, but not speak with other users.

Once you've created the above tables, you'll also add some sample data. Run the commands from the models/db-schema.txt file in your repository to add some sample data.

Once you're done, you can run SELECT * FROM users; for the users table and the rest of the tables too in-order to verify if your data has been imported successfully. I highly recommend you to take a few moments to run the above query for the rest of the tables and study the data that's stored in them.

Querying for information

We can imagine writing the following queries for our application:

A query to retrieve a list of all available rooms and their information. It would be useful to display this for a user who haven't logged in, so that they may get a glimpse of the kind of conversations that occur with the app!
Similarly, a user that has logged in would be interested to browse through the rooms for those clubs that they follow.
And finally when a user joins a room, we'll need a query that retrieves the club-name, room-name and the information about all the participants in the room which we can display in the UI.

Query #1: List all rooms

This basically includes:

All the rooms that are being hosted within a club; AND
All the rooms that aren't within any specific club

Here's a screenshot of the home screen when you log into clubhouse:

The screen contains a scrollable list of rooms, displayed within a card. If you notice closely, each room-card has the following information:

If the room is being hosted within a club, the name of the club appears on the top (with a green icon next to it).
The names of some of the participants within the room. On Clubhouse, these are usually the names of the hosts, speakers or moderators within the room.
On the bottom, you also see the overall number of room participants, and also the number of people in the speaker panel.

Basically, our query needs to give us data in the following format:

Let's break it down:

Step 1: Extract the room ID, room name and the club name that the room is a part of

SELECT
    rooms.id,
    rooms.name,
    clubs.name AS club_name

FROM clubs
RIGHT JOIN rooms
    ON rooms.club_id = clubs.id

ORDER BY rooms.date DESC
LIMIT 100;

Running this query in your psql prompt will give you the following result:

How does this work?

We mention the names of the columns that we're interested in in the beginning. Since our room table and our clubs table both have a column called name, we can rename either of these columns using the ASclause to assign an alias.
In the FROM clause, you mention the table or tables from which you can extract the info of one or several of the columns that we mentioned in the beginning.
And finally, we do a RIGHT JOIN on the rooms table. A JOIN creates an link between two tables based on a given condition. This condition is given within the ON clause.
The ORDER BY clause allows you to determine in what order should the query results be returned in. In social networking apps, the latest (or most recent) information is seen in the top, we'll retrieve the rooms in descending order of their creation date.
The LIMIT clause places an upper limit on how many rows should be returned.

Types of JOINS in a relational database management system:

An INNER JOIN returns only the common elements between both the tables — in other words, the intersection where they match on the joined column given by the ON clause.
A LEFT JOIN joins the rows from the first (LEFT) table to the second (RIGHT) table. But unlike the INNER JOIN, it will also include the rows from the LEFT table even if they don't match the condition.
A RIGHT JOIN is similar to the LEFT JOIN, except that it will include rows from the second (RIGHT) table that don't match the condition.

Why did we apply a RIGHT JOIN?

By selecting the clubs table and applying a RIGHT JOIN on the rooms table, we create an intersection between these two tables. This way, each row within our query result will contain room information and its associated club name too. But we also want to include those rows (rooms) that aren't associated with any club, and hence we specifically apply a RIGHT JOIN.

Step #2: Retrive the information of the panel participants for each room

The participants table contains the information about the participants within a room. It has the following foreign keys:

room_id refers to the rooms table
user_id refers to the users table

So we'll need to add a subquery that:

Selects the column names that we're interested in: the participant's user ID, their name and their type.
Applies an INNER JOIN on the users table, so that each participant row in the subquery is associated with its respective participant name from the users table.
Apply a WHERE clause to match the participant rows with their respective rooms
And a filter to return only those participants that are either a host, moderator or speaker.

SELECT
    rooms.id,
    rooms.name,
    clubs.name AS club_name,
    (
    SELECT json_agg(participant)
    FROM (
        SELECT participants.user_id, participants.role, users.name
        FROM participants
        INNER JOIN users
            ON participants.user_id = users.id
        WHERE participants.room_id = rooms.id
            AND participants.role IN ('HOST', 'MODERATOR', 'SPEAKER')
        LIMIT 6  
    ) participant  
    ) AS panel

FROM clubs

RIGHT JOIN rooms
    ON rooms.club_id = clubs.id

INNER JOIN participants
    ON participants.room_id = rooms.id

ORDER BY rooms.date DESC
LIMIT 100;

The json_agg function is an aggregate function provided by POSTGRESQL that groups the results of the inner subquery into a JSON array.

Step #3: Retrive the count of the participants in the panel, and the total number of participants in the room

For this, we'll apply a very similar query to the previous subquery, except this time we don't need to apply an INNER JOIN on the users table.

For fetching the number of participants on the panel list, we just need to select those rows that match the participant row to its associated room id, and apply a filter on the participant-types. Then all you need to do is return a COUNT of those rows.

So here's how the final query looks like:

SELECT
    rooms.id,
    rooms.name,
    clubs.name AS club_name,
    (
    SELECT json_agg(participant)
    FROM (
        SELECT participants.user_id, participants.role, users.name
        FROM participants

        INNER JOIN users
            ON participants.user_id = users.id
        WHERE participants.room_id = rooms.id
            AND participants.role IN ('HOST', 'MODERATOR', 'SPEAKER')
        LIMIT 6
    ) participant
    ) AS panel,

    COUNT(participants.user_id) FILTER (
        WHERE participants.room_id = rooms.id
            AND participants.role NOT IN ('AUDIENCE')
    ) as panel_count,

    COUNT(participants.user_id) FILTER (
        WHERE participants.room_id = rooms.id
    ) as participant_countFROM clubs

RIGHT JOIN rooms
    ON rooms.club_id = clubs.id
INNER JOIN participants
    ON participants.room_id = rooms.id

GROUP BY rooms.id, rooms.name, clubs.name
ORDER BY rooms.date DESC
LIMIT 100;

Whew, that was a lot! If you've followed the tutorial this far, I commend you!

Let's move on to the next query —

Query #2: List rooms for those clubs that the user follows

Fetching all room information is great. But wouldn't it be better if we tailored the room discovery feature for each user just a little bit?

The following query will return the room list information for the user Elon, whose user_id is 5:

SELECT
    rooms.id,
    rooms.name,
    clubs.name AS club_name,
    (
    SELECT json_agg(participant)
        FROM (
            SELECT participants.user_id, participants.role, users.name
            FROM participants

            INNER JOIN users
                ON participants.user_id = users.id
            WHERE participants.room_id = rooms.id
                AND participants.role != 'AUDIENCE'
            LIMIT 6
        ) participant
    ) AS panel,

    COUNT(participants.user_id) FILTER (
        WHERE participants.room_id = rooms.id
            AND participants.role != 'AUDIENCE'
    ) as panel_count,

    COUNT(participants.user_id) FILTER (
        WHERE participants.room_id = rooms.id
    ) as participant_count

FROM clubs

RIGHT JOIN rooms
    ON rooms.club_id = clubs.id

INNER JOIN participants
    ON participants.room_id = rooms.id

LEFT JOIN followers
    ON followers.club_id = clubs.id

WHERE followers.user_id = 5
    OR rooms.club_id IS NULL

GROUP BY rooms.id, rooms.name, clubs.name
ORDER BY rooms.date DESC

LIMIT 100;

The above query is almost exactly the same as the previous one, except for a few key differences:

...  
LEFT JOIN followers  
    ON followers.club_id = clubs.id

WHERE followers.user_id = 5  
    OR rooms.club_id IS NULL  
...

We do a LEFT JOIN on the followers table, and apply a WHERE clause so that we get those rows where elon is a follower. But because of this third join, it also has the effect of removing the rooms that don't belong to any club, so we apply one more condition using the OR clause.

And here are the results! —

As you may observe, the previous query returned 12 rows, this query returned only 8 rows — which makes sense, given that this user isn't following all the available clubs in the app.

Query #3: Retrieve information for a single room

When you tap or click on a room in clubhouse, the app opens up the room and you're officially a participant. You're able to see the users in the speaker list, and also other participants who aren't on the panel, but are simply listening.

So next, you'll write a query that for a given room ID retrieves the following room information:

Name of the club
Name of the room
The names of speakers on the panel
The names of the rest of the participants in the audience

Here's how the resulting query response might look like:

Let's write a query that fetches the above details for the room id 3:

SELECT
    rooms.id,
    rooms.name,
    clubs.name AS club_name,
    (
    SELECT json_agg(participant)
    FROM (
        SELECT
            participants.user_id, participants.role, participants.peer_id, users.name
        FROM participants, users
        WHERE participants.user_id = users.id
            AND participants.room_id = rooms.id
            AND participants.role IN ('HOST', 'MODERATOR','SPEAKER')
    ) participant
    ) AS panel,
    (
    SELECT json_agg(participant)
    FROM (
        SELECT participants.user_id, participants.role, participants.peer_id, users.name
        FROM participants, users
        WHERE participants.user_id = users.id
            AND participants.room_id = rooms.id
            AND participants.role = 'AUDIENCE'
    ) participant
    ) AS audience

FROM rooms
LEFT JOIN clubs
    ON rooms.club_id = clubs.id
WHERE rooms.id = 3;

Since the first (left) table is rooms and the second (right) table is clubs, we apply a LEFT JOIN. This makes sure that the row from the rooms table isn't discarded even if it isn't associated with any club.

Building a web service

Perfect! Now let's try doing the same, except this time you'll create APIs that uses the above queries to return the information we seek.

A brief explanation about some of the packages that you'll use:

Express is a web framework for NodeJS. It has lots of APIs, utilities and middlewares in its ecosystem to help you build your application.
pg is a NodeJS middleware for PostgreSQL. It helps your Node app talk to your database. You'll use this to run the same commands that you ran using the psql terminal, except this time it will be within your web server.
handlebars is a templating engine. You can use it to do parsing and apply conditional logic within html templates on the server before serving them to the client.

Feature #1: List and display rooms

Backend

Let's begin by first creating functions for each one of our previous queries.

Inside models/index.js, you'll add code for the following functions:

getRooms(limit): Returns all available rooms
getRoomsForUser(limit): Returns rooms for those clubs that the user follows
getRoomDetails(roomId): Returns information for the given room id

Next, write a function getUserByUsername(username) that fetches information of a user using their username. Add the following in models/users.js:

async function getUserByUsername(username) {
    try {
        const text = `
            SELECT * from users
            WHERE username = $1
        `;
        const values = [username];

        const res = await db.query(text, values);
        return res.rows[0];
    } catch (err) {
        console.log(err.stack);
    }
}

APIs

Next, you'll write APIs that utilize the above functions and expose them in a web service:

/rooms: A GET request to this route will return all available rooms. If you pass a query param ?userId=6, then the API should return only those rooms of those clubs that the user is a member of.
/rooms/:roomId: A GET request to this route will return the participant information for a particular room.

Add the following code inside routes/rooms.js:

router.get('/:roomId', async function(req, res, next) {
    const room = await db.getRoomDetails(req.params.roomId);

    if (room) {
        res.json({
            room: {
                id: room.id,
                name: room.name,
                club_name: room.club_name,
                audience: room.audience ? room.audience : [],
                panel: room.panel ? room.panel : [],
            }
        });
    } else {
        res.status(404).json({
            message: "Room does not exist"
        });
    }
});

router.get('/', async function(req, res, next) {
    let rooms = [];
    const userId = req.query.userId;

    if (req.query.userId) {
        rooms = await db.getRoomsForUser(userId);
    } else {
        rooms = await db.getRooms();
    }

    res.json({
        rooms,
    });
});

Finally, run npm start in your project root, and open the following URLs on your browser:

http://localhost:3000/rooms: This will return a list of all available rooms.
http://localhost:3000/rooms?userId=1: Returns all rooms for a specific user.
http://localhost:3000/rooms/4: Returns information for a single room.

Frontend

Going to http://localhost:3000 displays an empty page. First, we'll add a fake login, so when you type http://localhost:3000/username=rick, it displays a message greeting the user Rick Sanchez.

On your routes/index.js, add the following code:

/* GET home page. */
router.get('/', async function(req, res, next) {
    const username = req.query.username;

    let userInfo = {};

    if (username) {
        const user = await db.getUserByUsername(username);

        if (user) {
            userInfo = {
                username: user.username,
                name: user.name,
                userId: user.id,
            };
        }
    }

    res.render('index', userInfo);  
});

In the above code, we check for the request query params' value for username. Then, we use the username to extract the user information, and interpolate these values for the HTML document inside views/index.hbs. The interpolation is done by handlebars.

Now when you load the http://localhost:3000/username=rick, it will display the message "Howdy, Rick Sanchez" on the header.

Next, let's use our APIs to fetch a list of rooms and display them on the UI

First, extract the user details in our HTML file and save them in a variable. Add the following code inside public/modules/index.js:

const user = document.getElementById('user');

updateUserInfo({
    id: user.getAttribute('data-userId'),
    name: user.getAttribute('data-name'),
    username: user.getAttribute('data-username')
});

Add the following function inside models/api.js that makes a fetch request to our endpoint to fetch the list of rooms, and stores them in a ROOMS variable:

export async function fetchRooms(userId) {
    const url = userId ? `/rooms?userId=${userId}` : '/rooms';

    const response = await fetch(url, {
        method: 'GET'
    });

    if (response.ok) {
        const result = await response.json();
        updateRooms(result.rooms);
        return result;
    } else {
        return Promise.reject(Error(response.statusText));
    }
}

Now, you'll use this function to fetch the list of rooms, and utilize a helper method to display them in the UI. Add the following code in modules/index.js:

async function displayRooms() {
    let result;

    if (USER_INFO.id) {
        result = await fetchRoomsForUser(USER_INFO.id);
    } else {
        result = await fetchRooms();
    }

    renderRooms(ROOMS);
}

And we're done!

You should see the following page when you head over to http://localhost:3000/?username-rick:

Next Steps

In the next tutorial, you'll learn how to implement the next feature — joining a room and interacting with other participants via audio!

How to build your own Social Audio Chat Application - Part 2

Ashwin Hariharan for Egen ・ Feb 23 '22

#tutorial #webdev #javascript #node

If you liked this tutorial or have any questions, do leave a like or a comment!

Special thanks to Prashanth Molakala for helping with optimizing the postgresql queries 🙏

Fashion E-Commerce: Using Computer Vision to Find Clothing that Fits Like a Glove

Ashwin Hariharan — Tue, 15 Feb 2022 10:07:05 +0000

Problems in shopping for clothing online
How does SmartTry use computer vision to make cloth shopping easier
Tutorial
Jupyter-Notebook slides
Architecture and implementation
Potential revenue models

It was October 2019, and my cousin's wedding was taking place in about a week. Like everyone else, I was faced with the dilemma — what to wear? The struggle was real.

So there I was, in the mall with my brother picking out shirts, because everyone knows you can't go wrong with a great shirt right? I picked a few and was searching for a place to try them on before deciding which ones to buy. Finally found somewhere just right.

It was a fitting room.

I tried them on, and after checking myself out (and getting my brother's nod of approval) I bought the ones that fit just right. The following week, I wore one of them to the wedding. I looked great. Just call me GQ.

Like many guys, I don't particularly like shopping, but I tolerate the fitting room because it gives me the opportunity to see how everything looks before going out in public. Getting the right clothes the first time means not having to head back to the store for a return and round 2. I'm efficient like that.

It goes by different names — changing room, dressing room, trial room. You know, those small cubicles with the big mirrors and lighting that makes you look like an extra in The Walking Dead? The place where before you enter, the staff stops you and barks "Sorry, you can only bring in 4 pieces at a time", and you're like — "Can I please bring in all 7? I won't take long!" — just so that you can play dress-up for the next 45 minutes?

Yeah, they're kinda nice aren't they?

But then 2020 happened — a year that most people will remember for a very specific reason. With social distancing protocols and strict sanitation rules, shopping for clothes just became a lot harder.

A Problem of Size

It's not like 2020 forced us to spin our own cloth. It was simple enough to order clothes online, but it created a whole new set of issues.

Most of us have purchased something online. Electronics, household appliances, books or kitchenware. But amongst all, shopping for apparels is often the trickiest. It's pretty straightforward to purchase a smartphone, a laptop or some other gadget — you only need to worry if the specs and the design meet your needs, and whether it's got positive reviews. Ditto with books, handbags, or even jewellery.

But clothes? Reviews alone just won't cut it.

You probably hate going for clothes shopping just as much as I do. It can be terribly tiresome. But you gotta admit that it's at-least somewhat un-ambiguous. In a clothing store, picking something you like is often the hardest part. But once you do find something, things get simple. Grab your size, try it on, check yourself out.

With online shopping however, everything gets flipped! earching for just the right thing is easy — thanks to the ever-improving search features from all the mainstream e-commerce platforms. But getting the right fit? It's not as easy as it looks.

It's no wonder that the average return rates for online apparel shopping is a whopping 32%, and it burns a hole in the pocket for several e-commerce retailers. Here are some wrinkle-inducing stats:

Almost 1 out every 3 apparels purchased online get returned.
In 2019, about $260 Billion worth of retail goods were returned, out of which $30 Billion were from apparels.
The cost of returns is estimated to be nearly $1.5 Billion. The cost of each return is between $5-$15.

What are the top 3 reasons for people returning apparels bought online?

Here are some stats that'll create creases... on your forehead, that is! -

Size misfits is the biggest factor, making 43% of total online shoppers citing this as the reason.
Quality and color mismatch makes up for 29%.
22% of shoppers feel the need to simulate a trial room at home.

Despite these stats, online shopping isn't going away. More and more users are shopping online post-COVID:

27% of customers switched from offline to online due to COVID-19 in USA.
12% of customers would prefer to continue purchasing online even after the COVID.

So needless to say, online shopping is here to stay. And that includes apparel shopping as well, which brings with it with the size-mismatch problem.

The size-mismatch problem contributes to $1 Billion returns annually and affects inventory planning, leading to heavy losses for e-retailers.

Size-mismatch is not a mere wardrobe malfunction, it is a huge-sized problem!

At Egen, we decided to iron this one out.

SmartTry

Before the e-commerce era began, how did someone know if a certain clothing fit them?

The solution was simple. If you were buying a new shirt, you'd simply compare it with another shirt that you already possess — one that was already the best fit for you. The best part is, you wouldn't even need to try the shirt on in a fitting room, and it'd still be a great fit for you most of the time!

That's what SmartTry does — it enables users to compare their existing best fit clothes to new ones they want to purchase online.

Simplicity, carried to an an extreme, becomes elegance.

How does SmartTry work?

Say you're trying to buy a T-shirt online from Amazon, or from mango.com like the below screenshot -

You find the ultimate white T-shirt for the ultimate in effortless cool in your next Zoom meeting. But you've never bought a T-shirt from Mango before, so how will you know how it fits. (We'll spare you the rant on inconsistent sizing. You know it. We all know it.)

What do you do? You don't wanna go through the whole ordeal of having to return it if it's the wrong fit!

Precisely the pain point for you and many other people. And here's where SmartTry's user experience is guaranteed to knock your socks off!

Features

Clicking on the green SmartTry button opens the SmartTry plugin in a popup on your screen:

The SmartTry plugin shows you the T-shirt or the outfit that you want to buy, along with your previous best fits for you to compare.
Clicking on the Add a new apparel to Compare icon lets you upload a picture of an outfit that you currently own, into the SmartTry platform . You'll use this to upload a photo of the T-Shirt that is your current best-fit.
On uploading an apparel, it gets added to the Recommended Items section and to your Collection, with a button underneath that says Compare. Clicking on the Collection tab on the left will also show you all your previous uploads neatly organized within categories:

Finally, when you click on Compare, SmartTry runs its magic in the background and shows you how good of a fit your selection is!

The blue T-shirt border represents your current best-fit that you used for the comparison, while the black border is the T-shirt that you want to buy.
On the right, you'll be able to see precise measurements for the T-shirt you selected and your current best-fit. This helps you easily compare the differences between the two. (Let's face it. Every site has a size chart. No one is whipping out a measuring tape before clicking add to cart. Problem solved.)
In this example, the Small size seems a little too small — a 67% match. But not to worry! Since you like the T-shirt so much, you can switch between different sizes:

Medium size is a 94% fit.

You can now buy the outfit with the right size, knowing fully well that it'll fit you to a T!

Life ain't perfect, but your outfit damn well can be!

Unmasking the Magic

If you're a programmer, I'm sure you're wondering by now —

How does SmartTry really work behind the scenes, and how do you build it?

Well roll up your sleeves, because we put together a tutorial for you to follow along!

What you will learn

Whatever the mode of user-interaction may be, the power of SmartTry comes from Computer Vision techniques, that allows it to compare two different outfits and analyze the difference in their measurements. Over the next several minutes, I'll walk you through how to build this algorithm.

You'll use OpenCV, one of the most popular and widely used computer vision and machine learning libraries.
For writing code, you'll use a Jupyter-Notebook, an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

A word to the wise

Some knowledge of Python is essential in-order to understand this tutorial. If you haven't worked on Python before, I highly recommend this free course on Introduction to Python, before proceeding with the rest of this article.

Why Python? Because it is fairly easy to learn, and the Python ecosystem has a lot of tools and libraries for computer-vision, machine learning, and more. Once you've built the algorithm, Python also has several web-frameworks that allow you to build a service around this feature.

Tutorial

Go to this github link, and follow the instructions to download and install the the project dependencies. Then, run jupyter-lab in your project root and create a new Python3.x notebook file. Next,

Head over to the Jupyter tutorial and follow the steps outlined, to build your very own program that calculates accurate measurements for a given T-Shirt!

Alternatively

Head over to the below Binder link and execute the code right within your browser without the hassles of installation.

The smarttry.ipynb notebook has the complete source code with instructions for you to read. You could choose to run this directly, or write code on your own in a new notebook file as you follow the tutorial.

Slides

If you prefer taking just a quick peek into the logic, here's a short slide-deck:

So go ahead, put on your thinking cap and have fun!

The first version of SmartTry that we developed at Egen achieved an accuracy of 91.7% for clothing measurements! Hopefully, you'll be able to use this as a performance benchmark when following the tutorial and writing the program. I'm curious to know what your accuracy score looks like!

Next Steps

Once you've developed your program, you can integrate this into a web-service and expose it via an API. Here's the architecture implementation that we designed at Egen:

It doesn't stop here! The future scope for such a feature is immense:

On websites, SmartTry can be easily integrated into any existing e-commerce platform — as a popup, or an installable plugin, or a platform integration, or even as a browser extension.
Integrate SmartTry into mobile applications, which will allow you to upload photos of your outfit more easily.
You could maintain a common SmartTry profile that'll exist across multiple e-commerce websites.
A history page to quickly review past comparisons.

and more!

Potential Revenue Models

Being an intermediary between different parties that interact with a product is a model that a lot of businesses are exploring in recent times. SmartTry being designed as an easy-to-add plugin with a high accuracy makes it a really good contender for adoption in new businesses and existing e-commerce platforms. Here are some revenue models that can be quite viable (although they're by no means an exhaustive list):

At-cost SaaS model for online shopping. The pricing could be based on the number of API calls made to the SmartTry platform, for example — $0.001 per 1 API call.
Size and shopping data of users and their different demographics
Sending traffic from our platform to "preferred" e-commerce websites
In-store enterprise yearly licensing model
A freemium model with basic features accessible for free, and upgrades or more advanced features available at a monthly subscription.

Summary

What is right is often forgotten by what is convenient. And evidently so, it's true in the world of fashion e-commerce — with the potential of nearly $1 Billion USD in cost savings and returns, if done the right way! Reduced apparel returns would also translate into reduced carbon emissions, logistics and waste — thus helping the planet be clothed in green!

Credits

All credits for developing the idea and concept goes to the SmartTry team at Egen, consisting of Arif, Gagan, Anmol and Aatman. 🙌🏼

Build a community-driven delivery app using Django, PostgreSQL & JavaScript - Part 2

Ashwin Hariharan — Tue, 01 Feb 2022 12:03:19 +0000

Welcome to the 2nd part of this series on using technology to foster sustainability in your community! In this tutorial, you'll continue building Kartpool — a community driven delivery platform for the ones who need it the most!

Be sure to read Part 1 of the tutorial series thoroughly and complete the exercises before you proceed with this tutorial!

Problems in traditional delivery models
Local Search and Discovery platforms
Kartpool app features
- Advantages
Engineering the app
- Django
Feature #1: Adding a wishlist
Feature #2: Listing nearby wishlists
Feature #3: Store navigation and info
Feature #4: Updating a wishlist
Next steps
Source code

To recap, here's the list of features:

Feature #1: A location-based store discovery service from where users could buy grocery and other essentials. You already built this in Part 1.

Feature #2: Users can select a store and add a wishlist of essentials that they intend to buy. This wishlist would be visible to other residents.

Feature #3: Any other resident can choose to accept this person's request and become a wishmaster. Then, they can purchase the items from the store on behalf of the requestor and deliver it to them.

Feature #4: Users can give karma points to runners via a recognition and appreciation system, for being good samaritans and helpful members of the community.

Sounds similar to online grocery delivery platforms right? What's so different about this one, when so many others are already out there?

Fair question indeed! Let's look at some problems in the existing delivery business models, and what your platform will help solve:

Problems in traditional delivery models

You may already be aware of several retail delivery platforms out there. Walmart, founded in 1962, operates a multinational chain of hypermarkets, grocery stores and discount department stores along with home delivery, and is arguably the largest retailer in the US in terms of revenue.

In June 2017, Amazon acquired Whole Foods for $13.7 Billion USD and amped up their retail delivery offerings as well. There's also Instacart — another grocery delivery and pick-up service in Canada and USA. Despite losing Whole Foods as a customer, Instacart holds a whopping 59% of the delivery market. And Kroger, another American retail company, is the second largest retailer in the United States, just behind Walmart.

These developments in the retail and delivery sectors have had various impacts on local businesses:

While these platforms offer convenience, there can be challenges in ensuring a consistently positive experience for customers who shop at local stores.
All these platforms have also been at the center of a large number of controversies and lawsuits on issues involving low wages, poor working conditions, treatment of suppliers, and waste management. When local businesses are on-boarded onto these larger platforms, any bad press coverages and negative consequences tend to spill over into your store's reputation and reviews as well, for probably no fault of their own.
Large companies slowly end up transforming into a monopoly — taking over smaller businesses and becoming the sole retailer and distribution chain in the area. Eventually your local businesses become very dependent on these platforms, which is a bad idea.
There are labor costs, and service and delivery charges associated while utilizing larger monopolized platforms. Due to these, businesses would make lesser profits than they did if they were to sell the items directly. In-order to maintain their current profits or to grow, they would inevitably need to raise the prices of items — once again, bad news for both customers and grocers.

It seems like there are a lot of opportunities for innovation in the delivery model. Time for a fresh new approach!

Local Search and Discovery Platforms

In the previous part, you learned to build a store discovery service that fetches all nearby stores in your neighborhood and displays them on a map.

Over the last decade, local search-and-discovery applications have been seeing a steady rise in usage and popularity. In 2009, Foursquare — a platform of nearly 50 million users — launched a platform that let users search for restaurants, nightlife spots, shops and other places in a location. In 2012, Facebook launched Nearby, Foursquare's competitor that pretty much did the same thing. And in 2017, Google Maps announced a similar feature that let users create lists of their favorite places to visit.

When you look at the user interfaces in several of these platforms, you observe a lot of similarities — especially on the layout on the home page that shows the places of interest:

Indeed, if you look at Foursquare's city guide — the user-interface consists of a small column on the left that displays a list of areas of interest, along with their locations to the right on a wide map. Google Maps also has a similar interface:

And here's AirBnb:

Clicking on one of the items on the left makes the map fly to the associated location and zoom in to the marker icon. Sometimes, it also shows a popup on the marker with some useful information.

So needless to say, these user-interfaces are in vogue because its convenient to navigate through the list on the left, and look at their associated locations on the right on the map.

Deriving lessons from both the online grocery delivery models and local search and discovery applications, this platform that you'll build might just be what your community needs!

Kartpool app screenshot - what you'll build!

Features

On the right-side you have a map where you'll type in the name of a location, which then displays stores in the area. You already did this in the previous tutorial.

The left column is a bit different — unlike Foursquare or Google Maps, you won't be displaying stores here, but wishlists. Clicking on one of the wishlist cards will make the map "fly" to the location of the store, where the items can be purchased from. These list of cards are arranged across 3 different tabs:

The 1st tab displays all nearby wishlists created by users in the neighborhood. From here, you can accept a wishlist and it will be assigned to you to collect from a store nearby.
Wishlists created by you will be visible on the 2nd tab.
The 3rd tab shows wishlists that you accept from the 1st tab. If you mark a wishlist as accepted, you become a wishmaster for that user and it gets added to your trips. You can then make a trip to the store to purchase the items, and mark them as fulfilled once your neighbor receives them.

To create a wishlist, you'll select a store icon from the map and add the items that you need by using the input-field on the bottom left.

How are these features useful?

Advantages

While most of the year 2020 was spent in lockdowns and quarantines, it also revealed many heart-warming examples of how powerful organized efforts and informed choices of individuals within a community can be.

Providing a digital tool that leverages this power can create an immensely positive social and economic impact:

You could foster virtually an endless shopping experience focused exclusively on local stores and businesses.
User on-boarding becomes simpler.
Enable massive reduction in delivery/service fees.
The business model is socially driven and community-driven, which will foster a feeling of togetherness and readiness to help those in need.
Not having to rely on middlemen and eliminating needless logistics and packaging would translate into drastic reductions in pollution and consumer waste, thus helping the planet stay green.

I hope you're excited. Let's begin!

Engineering

Django

A Django project consists of one or more applications. At the moment, your project root directory contains two applications — stores and home. An application encapsulates a set of related features along with its own models, views, serializers and business logic.

It is useful to group your project logic in this way as it offers a lot of advantages:

It gives you much better organization and structure of your project, and allows you to maintain separation of concerns.
Flexible development — one developer could chose to work on features related to stores, while another could chose to work on the wishlists feature.
Re-usability — you can easily reuse an app and migrate it to another project.

So in your current project, everything that is related to stores is in the stores directory, and everything related to rendering the home page is in the home directory. Similarly, you'll create a new Django app for the wishlists feature. In your terminal type python manage.py startapp wishlists. This will create a new directory wishlists with its structure similar to the stores directory.

Wishlists

Step #1: Create the database model for storing wishlists

Open wishlists/model.py and add the following code:

from django.db import models  
from django.contrib.postgres.fields import ArrayField

# Create your models here.

WISHLIST_STATUSES = [  
    ("PENDING", "PENDING"),  
    ("ACCEPTED", "ACCEPTED"),  
    ("FULFILLED", "FULFILLED")  
]

class Wishlist(models.Model):  
    created_at = models.DateTimeField(auto_now_add=True)  
    buyer = models.CharField(max_length=100)  
    wishmaster = models.CharField(max_length=100)  
    items = ArrayField(models.CharField(max_length=100))  
    status = models.CharField(  
        choices=WISHLIST_STATUSES,  
        default="PENDING",  
        max_length=10  
    )  
    store = models.ForeignKey(  
        "stores.Store",  
        related_name="wishlists",  
        on_delete=models.SET_NULL,  
        null=True  
    )

Each wishlist can have one of three statuses, with the default status being PENDINGat the time of creation.
A buyeris that user who creates the wishlist, while the wishmasteris the user that makes the trip to the store and collects the items on behalf of the buyer.
Each wishlist also has a foreign key that's associated with a valid store-id from the stores model that you implemented in the previous tutorial.

Now you'll run python manage.py makemigrations followed by python manage.py migrate . Django's ORM will create the table with the defined schema in the database!

Step #2: Add a serializer

In wishlists/serializers.py, add the following:

from rest_framework import serializers  
from .models import Wishlist

class WishlistSerializer(serializers.ModelSerializer):  
    class Meta:  
        model = Wishlist  
        fields = [  
            'id', 'created_at', 'buyer', 'wishmaster', 'items',  
            'status', 'store'  
        ]

Step #3: Define the View Class

Add the following in wishlists/views.py:

from rest_framework import viewsets  
from rest_framework.response import Responsefrom .models import Wishlist  
from .serializers import WishlistSerializer

# Create your views here.
class WishlistView(viewsets.ModelViewSet):  
    queryset = Wishlist.objects.all()  
    serializer_class = WishlistSerializer

You'll add the controller logic for creating, listing and updating wishlists within this class.

Step #4: Define the API service

Add the URL for your wishlists service in kartpool/urls.py:

from wishlists import views as wishlists_viewsrouter.register(r'wishlists', wishlists_views.WishlistView, basename='wishlists')

Any request made to the endpoint /wishlists/ will execute the relevant controller within your WishlistView class.

Now you're ready to begin developing the wishlist feature for your app.

Note: Some helper methods have already been provided for you in the code, so that you may devote most of your time towards writing the core logic:

helpers.js: Contains methods to render wishlists.
api.js: Has functions for making network requests to the /stores/ and /wishlists/ endpoints.

Feature #1: Adding a Wishlist

Backend

Create a new file services.py in the wishlists directory.

Here, you'll write a function that takes in 3 arguments — a buyer, an items array, and a store. This function will create a new Wishlist, save it in the table, and return it.

from django.core.exceptions import ObjectDoesNotExist  
from .models import Wishlist  
from stores.models import Store

def create_wishlist(buyer: str, items: list, store: Store):  
    wishlist = Wishlist(  
        buyer=buyer,  
        items=items,  
        store_id=store  
    )

    wishlist.save()

    return wishlist

Next, you'll import this function in wishlist/views.py and add the controller logic in the WishlistViewclass.

def create(self, request):  
    buyer = self.request.data.get('buyer')  
    items = self.request.data.get('items')  
    store = int(self.request.data.get('store'))

    wishlist = create_wishlist(buyer, items, store)  
    wishlist_data = WishlistSerializer(wishlist, many=False)

    return Response(wishlist_data.data)

When when someone makes a POST request to the /wishlists/ endpoint, it'll run the create method, extract the values for the buyer, items and the store id, and pass them to create_wishlist to create a new wishlist in the db.

Front-end

In order to add a wishlist on the front-end, you'll need to click on a store marker on the map, and add items on the input-box#wishlist-items separated by commas. Then when you click on the "Add a wishlist" button, it will make a POSt request to /wishlists/ with the required data.

Open wishlists.js and add the following:

async function createWishlist() {  
    const wishlistInput = document.getElementById("wishlist-items").value.trim();  
    if (USERNAME && SELECTED_sTORE_ID && wishlistInput) {  
        addWishlist(USERNAME, wishlistInput.split(","), STORE);  
    }  
}

This function extracts the value from the input-field, converts it into an array, and passes these values to the method addWishlist, which will make the POST request to add the wishlist into the database!

You'll now need to run this function upon clicking the Add a wishlist button. Let's define the event handler for this in index.js:

document.getElementById("add-wishlist").onclick = function(e) {  
    createWishlist();  
}

Run python manage.py runserver and head over to localhost:8000/?username=YOURNAME. Try adding your first wishlist and some sample wishlists for a few other users as well. You should be able to see them in your database.

Next, you'll build the service to fetch nearby wishlists and display them in the UI.

Feature #2: Listing nearby wishlists

Backend

For retriving nearby wishlists, you'll define a function get_wishlists in wishlists/services.py, that accepts 3 arguments — a latitude, a longitude, and an optional optionsdictionary.

from stores.services import get_nearby_stores_within

def get_wishlists(latitude: float, longitude: float, options: dict):  
    return Wishlist.objects.filter(  
        **options,  
        store__in=get_nearby_stores_within(  
            latitude=latitude,  
            longitude=longitude,  
            km=10,  
            limit=100  
        )  
    ).order_by(  
        'created_at'  
    )

Using the get_nearby_stores_within function that you wrote in Part 1, we can use the foreign key store and retrieve only those wishlists for which their associated stores are near the given pair of coordinates. That way in the UI, you'll never have a wishlist for which its store isn't visible on the map! Makes sense?

With the get_wishlists method, you can retrieve the required data for all 3 tabs for the left column using the options argument:

If you wish to return your own requests, you just need to retrieve those wishlists for which you're the buyer. So you'd pass in {buyer=ashwin} in the options argument.
Likewise, for retrieving your trips, you'll just need to retrieve those wishlists for which you're the wishmaster, by providing {wishmaster=ashwin}.

Next, you'll import the above function and add the controller logic in wishlists/views.py:

def list(self, request):
    latitude = self.request.query_params.get('lat')
    longitude = self.request.query_params.get('lng')
    options = {}
    for key in ('buyer', 'wishmaster'):
        value = self.request.query_params.get(key)
        if value:
            options[key] = value

    wishlist = get_wishlists(
        float(latitude),
        float(longitude),
        options
    )

    wishlist_data = WishlistSerializer(wishlist, many=True)
    return Response(wishlist_data.data)

Frontend

Inside wishlists.js, you'll have 3 functions:

displayNearbyWishlists: To show all nearby wishlists in the 1st tab.
displayMyRequests: To show wishlists that you created in the 2nd tab.
displayMyTrips: To show the wishlists that you accepted in the 3rd Tab.

export async function displayNearbyWishlists(latitude, longitude) {
    try {
        const nearbyWishlists = await fetchNearbyWishlists(latitude, longitude);
        renderWishlists('nearby-wishlists', nearbyWishlists);
    } catch (error) {
        console.error(error);
    }
}

fetchNearbyWishlists makes an HTTP GET request with the given pair of coordinates to the endpoint /wishlists/. Once the wishlists are fetched, you'll render it inside the tab section with the id nearby-wishlists, using the helper method renderWishlists.

Likewise, add the other two functions as well:

export async function displayMyRequests(latitude, longitude) {  
    try {  
        const myWishlists = await fetchNearbyWishlists(latitude, longitude, {buyer: USERNAME});  
        renderWishlists('my-wishlists', myWishlists);  
    } catch(error) {  
        console.error(error);  
    }  
}export async function displayMyTrips(latitude, longitude) {  
    try {  
        const myTrips = await fetchNearbyWishlists(latitude, longitude, {wishmaster: USERNAME});  
        renderWishlists('my-trips', myTrips);  
    } catch(error) {  
        console.error(error);  
    }  
}

Refresh the page and try it out!

Feature #3: Store Navigation and Info

Displaying wishlists is great, but how do you know which store to collect it from?

That's where the foreign key store on our Store model comes in handy, which is present in the JSON response when you make the request to fetch wishlists:

On the DOM, each wishlist card has a data-attribute with the value of the associated store-id:

Inside stores.js, add a function setStoreNavigation that takes in 2 arguments — map and storesGeoJson . The function will loop over all the wishlist elements and add a click event listener on all of them. Upon a click,

Fetch the wishlist's associated store-id from the data-store-id attribute.
Then using the store-id, find the relevant store's GeoJSON information (that also contains the latitude and longitude information) from storesGeoJson.
Using the store's coordinates, you can now programmatically make Mapbox zoom into the store's location.

export function setStoreNavigation(map, storesGeoJson) {  
    const wishlistElements = document.getElementsByClassName('wishlist');

    for (let i=0; i<wishlistElements.length; i++) {  
        wishlistElements[i].onclick = (event) => {  
            const storeId = event.currentTarget.getAttribute('data-store-id');

            for (let point of storesGeoJson.features) {  
                if (storeId === point.properties.id) {  
                    flyToStore(map, point);  
                    displayStoreDetails(map, point);  
                    updateSelectedStore(storeId);  
                    break;  
                }  
            }  
        }  
    }  
}

Next, add the function flyToStore that zooms the map into a given pair of coordinates within map.js:

export function flyToStore(map, point) {  
    map.flyTo({  
        center: point.geometry.coordinates,  
        zoom: 20  
    });  
}

Refresh the page, type in a location in which you created the wishlists in the previous step. Once the wishlists show up, click on one of them and watch the map zoom in to the store marker!

But we're not nearly done yet.

Accessibility

In the previous tutorial, you added a title attribute to each marker that shows you the store information when you hover your cursor over a store icon. Though it gets the job done, it isn't nearly good in terms of accessibility.

Along with flying to the store location, what would really be good is to also show a popup on the marker. Lucky for you, Mapbox has a neat little API that does the job!

Add the following function within map.js, and call it inside setStoreNavigation, right after you fly to the store:

export function displayStoreDetails(map, point) {  
    const popUps = document.getElementsByClassName('mapboxgl-popup');  
    /** Check if there is already a popup on the map and if so, remove it */  
    if (popUps[0]){  
        popUps[0].remove();  
    } const popup = new mapboxgl.Popup({ closeOnClick: false })  
        .setLngLat(point.geometry.coordinates)  
        .setHTML(`  
            <details>  
                <summary><h2>${point.properties.name}</h2></summary>  
                <dl>  
                    <dt>Distance</dt>  
                    <dd>Approximately <strong>${point.properties.distance.toFixed(2)} km</strong> away</dd>

                    <dt>Address</dt>  
                    <dd>${point.properties.address || 'N/A'}</dd>

                    <dt>Phone</dt>  
                    <dd>${point.properties.phone || 'N/A'}</dd>

                    <dt>Rating</dt>  
                    <dd>${point.properties.rating || 'N/A'}</dd>  
                </dl>  
            </details>  
        `)  
        .addTo(map);  
    return popup;  
}

Moving on to our final set of feature in this tutorial:

Feature #4: Updating a wishlist

So far, you've managed to build stuff that adds quite some oomph factor on the UI. But your app isn't usable yet.

The real fun begins when a user can pick up one of the wishlists created by someone in the neighborhood. This is where the true value of the application lies — the community aspect that makes it possible for neighbors to help each other and be good samaritans during times of need!

When a wishlist item is first created on the platform, it isn't assigned to any wishmaster yet, and the default status is set to PENDING. So this is how the card looks like on the UI:

In-order to accept a wishlist:

Click on the little grey icon on the right-side of the card. This icon has a class value acceptin the DOM.
Upon clicking the icon, the app will make a PATCH request to the /wishlists/ endpoint.
On the backend, update the wishlist item's status to ACCEPTED, and also update the wishmaster field to the current user.
Finally in the UI, accepted wishlists will be indicated by a little green shopper icon with an acceptedclass, like this:

Once the items have been picked up by the wishmaster and handed over to the buyer, they can then click on the green icon and mark it as FULFILLED with a similar PATCH request, after which it'll look like this:

Backend

Create a function update_wishlist inside wishlists/services.py. This function will require 3 arguments — the primary key of the wishlist pk, the wishmaster, and status:

def update_wishlist(pk: str, wishmaster: str=None, status: str="ACCEPTED"):  
    try:  
        wishlist = Wishlist.objects.get(pk=pk)

        wishlist.wishmaster = wishmaster  
        wishlist.status = status

        wishlist.save(update_fields=['wishmaster', 'status'])  
        return wishlist  
    except ObjectDoesNotExist:  
        print("Wishlist does not exist")

You'll call this method upon receiving a PATCH request to the /wishlists/ endpoint. For PATCH requests, the controller logic needs to be written within the partial_update inside your Class View.

Import the above method inside wishlists/views.py and add the following code inside the WishlistView class:

def partial_update(self, request, pk):  
    wishlist = update_wishlist(  
        pk=pk,  
        wishmaster=self.request.data.get('wishmaster'),  
        status=self.request.data.get('status')  
    )

    wishlist_data = WishlistSerializer(wishlist, many=False)  
    return Response(wishlist_data.data)

That's all you need for the backend!

Frontend

First you'll register an event listener for any click events that occur on the wishlists container elements. Add the following code inside index.js:

const wishlists = document.getElementsByClassName('wishlists');  
    for (let i=0; i<wishlists.length; i++) {  
        wishlists[i].addEventListener('click', updateWishlistStatus);  
    }  
}

This is how the markup of the card looks like:

In wishlists.js, you'll define a function updateWishlistStatus that runs whenever a click occurs anywhere within the three wishlist container elements. Within this function:

First check whether the click occurred on any of the icons on the right-side of the card. If it did, then
Grab the wishlist's primary key (id) from the data-id field.
Determine the right status value to be set, using the class-name of the icon.
Finally call the updateWishlist function from api.js to make the PATCH request and update the wishlist!

export async function updateWishlistStatus(event) {  
    switch(event.target.className) {  
        case 'accept':  
            event.preventDefault();  
            updateWishlist(  
                event.target.getAttribute('data-id'),  
                {  
                    status: 'ACCEPTED',  
                    wishmaster: USERNAME  
                }  
            ).then((result) => {  
                updateWishlistNode(event.target, 'ACCEPTED');  
            }).catch(error => console.error(error));

            break;  
        case 'accepted':  
            event.preventDefault();  
            updateWishlist(  
                event.target.getAttribute('data-id'),  
                {  
                    status: 'FULFILLED',  
                    wishmaster: USERNAME  
                }  
            ).then((result) => {  
                updateWishlistNode(event.target, 'FULFILLED');  
            }).catch(error => console.error(error));

            break;  
    }  
}

And you're done. Refresh the page, play around with your app and watch it in action!

What's next?

Congrats on successfully building a Minimum Viable Product. As an exercise, I leave it to you implement the karma points feature. Don't hesitate to leave a comment if you need any help!

Once you finish developing all the essential features, it's time for you to speak to your neighbors, demonstrate to them the usefulness of this service, and get some real active users on the platform. Alone, you can do little — but together, you can do so much more!

Technology is best when it brings people together — Matt Mullenweg

Speaking to members within your community will help you receive valuable feedback for your platform. Here are some nice-to-have features that'll make your app even more powerful:

Add the ability for users to sign-up and create accounts in the platform.
A community hall-of-fame page that displays Samaritans of the month.
Show a store's inventory so that users may know beforehand whether they can buy a certain item from a store. You will need to connect with your local businesses for this one!
Continuously refreshing the page after adding or updating a wishlist is annoying. Why don't you try adding web-sockets?
Implement payment integration so that users can make payments directly to the store from within the app.
Build a Progressive Web App or a native mobile application UI.

Conclusion

Crisis are part of life. Everybody has to face them, and it doesn't make any difference what the crisis is. Most hardships are opportunities to either advance or stay where you are.

That being said —

You shouldn't necessarily have to wait until you're in a crisis to come up with a crisis plan.

The utility of Kartpool will extend beyond emergencies. With big players and large retail chains already eating up local businesses and killing most competition, platforms like these will give the small guys a fighting chance. Your local economy and community will thrive, adapt and grow together in the ever-changing e-commerce landscape and become sustainable!

I leave you with this:

Thou shalt treat thy neighbor as thyself

Source code

Here's the github repository for reference. In case you have any questions regarding the tutorial, please leave a comment below!

Build a community-driven delivery app using Django, PostgreSQL & JavaScript

Ashwin Hariharan — Mon, 31 Jan 2022 16:25:08 +0000

This post was originally published an year ago. I hope the dev.to community finds it useful!

Some COVID-19 stats
Kartpool app features
What you will build
- Tech stack you will learn and use
- Prerequisites
- Project boilerplate
PostgreSQL
- Designing the schema
Django
Open Street Maps
Building a location-based store discovery service
- Backend
- Frontend
Next steps

2012, Terminator, World War Z, Avengers — Endgame. Over the years, movies featuring apocalyptic events have been a favorite amongst movie-goers. Many of them featured the Earth nearly coming to an end in a myriad of ways — from killer zombies to aliens, or something about the planet's core getting unstable. They all usually end up with a hero (or a team of heroes) saving the world from the brink of destruction. And everything's back to normal.

Except the year 2020 brought us the Coronavirus — something that none of the movies prepared us for. Although calling COVID-19 an apocalypse might be a bit of a stretch, it's not too far from the next worst thing — a Pandemic. The thing about post-apocalyptic movies, you see, is that there's always a Deus ex machina that brings everyone back from the edge of the abyss. I mean, how many of us have at least fantasized about this, for example?

Unfortunately in the real world, we don't have one.

So while our scientists toil hard to produce vaccines for COVID-19, countries around the world have been making attempts to address the problems that have arisen due to the pandemic. Technology solutions, in particular, have an immense potential in raising awareness and slowing down the spread of disease. Unfortunately, besides contact-tracing applications and daunting dashboards that display disease stats, most efforts have only been marginally effective.

Let's look at some stats

You probably are keeping an eye on the numbers related to COVID-19 case reports already. But there are several other kinds of stats that matter too, in the long run!

Over the course of the year, numerous studies have shed light on how COVID-19 has affected people and shaped up some significant behavioral patterns. Here are some highlights:

Most people are uncomfortable with visiting public spaces and are apprehensive in stepping out, doing so only for purchasing essentials, like Groceries, household supplies, and medicines.
About 40% of consumers prefer to have their products locally sourced. However, the projections for online grocery sales also indicate an increase by almost the same amount. More people are switching to e-commerce platforms for purchasing groceries and medicine, citing hygiene and fear of exposure to the virus as the top concerns.
Online shopping has created a massive dislocation in local businesses. A survey in the US revealed that out of a sample size of 5800 businesses, 43% had temporarily closed! The biggest reason being that there was no reliable shipping, and most didn't have enough capital to set up a digital infrastructure for their consumers. At the same time, businesses that do manage to provide a reliable delivery option are expected to see nearly 16% increase in revenue.

Now that you know this, how do you make sense of it all?

To put it succinctly, the need of the hour is:

Effective means to shop for essentials online and optimize distribution.
Improve offline shopping experience in a manner that poses the least risk of exposure.
Serve the neighborhood and local businesses.

Enter Kartpool

We at Egen decided to build Kartpool — a community-driven delivery app and platform for connecting residents with local businesses.

After some good old fashioned design thinking, these were the following key features that we agreed upon:

Feature #1: Residents will be able to look at stores in their vicinity and look at the inventory. They can then add a wishlist of essentials that they intend to buy and place a request, which would be visible to other residents.

Feature #2: Any other resident can choose to accept this person's request and become a wishmaster. Then, they can purchase the items from the store on behalf of the requestor and deliver it to them.

Feature #3: Users can give karma points to runners via a recognition and appreciation system, for being good samaritans and helpful members of the community.

Why Community-first?

Government imposed lockdowns will only take us so far. The focus must change to community participation — a bottom-up approach, unlike the top-down directives initiated by the world's governments so far. Talking about the biggest lessons from the Ebola epidemic of 2014, the WHO said that "community engagement is the one factor that underlies the success of all other control measures."

This means, providing your community with access to digital tools that assist in getting timely access to essentials, while at the same time observing containment protocols, and also helping local businesses and livelihoods.

So, if you're a developer and feel the need to scratch that programming itch, keep reading! I'll walk you through how to build this application that will be useful to your community and neighborhood during this crisis!

What you will build

In this tutorial, you'll learn to build a minimum viable solution that you'll be able to present within your community. At the end of this tutorial, you will have:

A client-facing app that runs in a browser, in which users will be able to view nearby stores and wishlists. They can accept to pickup wishlists from other users and also make wishlists of their own.
A backend server to handle incoming requests and to interact with the database.
A database to store and retrieve information on nearby stores, wishlists, and users in the neighborhood.

Since this is a programming tutorial, your focus will not be much on web design, (though you're welcome to do so if you wish! ). It'll primarily be about building this service that has an immediate and long-lasting practical and utilitarian benefit.

Technologies that you will learn and use

You'll learn to use PostgreSQL as your database. It's fast, extremely extensible to a wide array of needs (including hyperlocal business applications), has support for both SQL and NO-SQL data models, and is scalable.
You'll use Python with Django for the back-end. It's a very robust and well-maintained framework, specially designed to accelerate building web services. It's also compatible with major databases like PostgreSQL.
On the front-end, you'll be using HTML, CSS and JavaScript, and also a few other libraries like Mapbox.

Prerequisites

It will be helpful to have at least a theoretical understanding of how relational database systems work.
Some knowledge of JavaScript and Python will also help.

Part 1 (what you're reading right now)

You'll begin with the first feature of this application. For this feature to work, you'll build an API that fetches all stores from a nearby location along with their inventory information, and display it to the user. Here's what we'll cover:

PostgreSQL schema design and modeling
Using PostgreSQL geospatial querying capabilities
Using Django and Mapbox to build a Location-based store discovery service

Part 2

we'll extend this service and add the ability for users to create wishlists and accept wishlists from other users.

Let's get started!

Go to this link, and follow the instructions to install all the necessary project dependencies. Some of the libraries, python modules and methods have already been imported in the code for you to use.

PostgreSQL

You'll be using PostgreSQL along with PostGIS - an extension that allows you to store geographic and spatial data in PostgreSQL. It makes your DBMS location aware. and comes with some really cool features!

Designing the schema

Broadly speaking, your app has 3 kinds of entities:

Store: Any shop that sells items to people in the nearby vicinity.
User: A user in the system can be someone who issues a request for items to purchase from a store, or they can also be a runner who can collect items on behalf of another user.
Wishlist: A user can create a wishlist of items that they need, which can be bought from a nearby store. This wishlist will be visible to other users in the neighrborhood, and one of them can volunteer to take a trip to the store and buy those items.

We could imagine more entities in the system but these will suffice for now. If you built data models for each of the above entities, what would they look like? Take a moment to think!

Next, fire up your terminal and run psql to enter the PostgreSQL interactive terminal. Once it opens up, you can look at the existing databases in PostgreSQL by typing \l.

You'll need to create a new database for your application. Let's call it kartpool. Run CREATE DATABASE kartpool; to create a new database.

For applying queries to this database, you'll need to switch to it by typing \c kartpool. Here are some commonly used queries:

\l: List available databases
\dt: List existing tables
SELECT * FROM stores: Will print all rows and columns in the stores table.
DROP TABLE <tablename>: Will delete the table
DROP DATABASE kartpool: Deletes the database

Now as you may already know, in a relational database, data is usually stored in rows and columns, and the usual data-types can be text, numeric, unique identifiers, images — which is sufficient for most applications. However, when you're building an application that has a location-based feature, you need for a way to be able to store spatial information as well!

And here is where PostGIS helps! In addition to the existing data-types and querying that PostgreSQL provides, PostGIS introduces some more data-types, and also provides you with querying capabilities, especially for geographical information.

You'll be working with the Point datatype, because that will allow you to store location information of a store using its geographic coordinates. Then, if you have the user's location, you'll be able to query for stores within their proximity!

For the purposes of this tutorial, you won't dive into the SQL commands — as it can become quite hard to get a hold of the syntax!

The more I look at it, the less I understand!

So instead, you'll learn to use Django's Object Relational-mapping (ORM) layer to interact with the database. That way, if you switch the database from PostgreSQL to another one that's supported by Django, your python code still remains the same!

Django

Django has been consistently rated among the most popular Python frameworks for multiple years in a row. It supports the Model-View-Template pattern, which allows you to maintain a robust architecture in your code, as the application grows with time. It also has "batteries" included — meaning that it comes with an admin panel and an extensive set of plugins for building just about anything ranging from authentication services to payment systems.

Django also comes along with an ORM system by default, which you'll be using to interact and query your PostgreSQL database.

Now, each Django project consists of one or more applications. A Django app allows you to encapsulate a set of related functionality along with its models, views and application logic together. For this project, you'll have the following apps:

stores : All store-related functionality (such as storing or fetching nearby stores) will go here.
wishlists: This will contain all the logic required to create/update or retrieve wishlists.
home: You'll use this app to serve the app page in the browser along with client side JavaScript files, stylesheets, and for loading external libraries like Mapbox.

You'll first define the model for store-related information instores/models.py. Open the file in your favorite editor and write the following:

Here you've basically defined a class for your Store, along with its fields and data-types. The location field, in particular, is of type Point — so that allows you to store information using a pair of coordinates.

Now, you'll need to tell Django to create the store table in PostgreSQL. In your terminal, run the following:

python manage.py makemigrations  
python manage.py migrate

This will have the effect of creating a table called stores_store! This is the equivalent of running an SQL query using psql :

CREATE TABLE stores_store (  
    id BIGINT GENERATED BY default AS IDENTITY PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    store_type VARCHAR(50),
    created_at TIMESTAMPTZ,
    rating float,
    opening_hour TIME,
    closing_hour TIME,
    city VARCHAR(50),
    latitude float,
    longitude float, 
    location GEOGRAPHY,
    address VARCHAR(255),
    phone VARCHAR(100)
);

Next, you'll need to populate some data! Wouldn't it be awesome if you could find some real data to populate your database?

Well lucky for you, we have -

Open Street Maps

A collaborative project, that has real-world location information of billions of areas of interest — collected and aggregated by good samaritans all over the world! Plenty of non-profits have built useful services ¹ on top of OSM, like Wheelmap for example:

A list of places in Chicago that are wheelchair accessible

Pretty amazing right?

Now, what if I told you that all OSM data is in-fact stored in — (surprise), PostgreSQL?! Thanks to these folks, you can pretty much pull information from anywhere you live!

In order to mine information from OSM, there is a neat little tool called Overpass Turbo ². Header over there, and click on the Wizard option. If you live in Chicago, you can type something like shop in Chicago which will run a query to pull out all shop-related information within the region. Once the query has run, download the data in raw OSM format and save it as data.json in the data folder of your project.

Now, run python manage.py makemigrations stores --empty to create a new empty migration file, and add the following code:

In this code, you've defined an operation to be run called load_data . This method uses the json library to load the file you just downloaded, and loops over each object that contains information for a particular store. You then pick the information you need, create a Store instance and then save it.

Now, you can run python manage.py migrate once again, and it will extract the stores from the JSON dump, and save them in the db. You can run SELECT * FROM stores_store from psql and see the results!

Now that you have a database with spatial querying capabilities that has actual real data stored in it, you can now create a useful application:

A Location-based store discovery service

Here's what you'll build:

On the UI, you'll have a map with a search box. A user can type in the name of a place (where they live) on the search box to perform a search.
The search box will make automatic suggestions as the user types.
On selecting the place from the search results, the app hits an API and fetches the relevant stores information near to that location
The store locations will then be displayed on the map.

Here's a visualization of how this service might look like

Backend

Write an API to retrieve a list of nearby stores from a pair of coordinates

Here are the components that you will need in-order to build this service in Django —

A URL/API Route: You'll define an endpoint called stores/ to which you could pass a pair of coordinate values lat and lngin the request query parameters.
A View: This is essentially a request controller, that helps extract the information from a incoming request, do something with it, and send a response back. So you'll define a view to handle the stores endpoint, extract the latitude and longitude information from the request, and ask PostgreSQL to look up information for all nearby stores for those pair of coordinates. It is always a good idea to separate your business logic from the request, so you'll write the PostgreSQL querying part inside services.py file of your project.
A Serializer: This helps you to control the output of your responses and what fields must they contain.

In order to build the above components, you'll use the Django REST framework, and the built-in GeoDjango module in your project.

Step #1: Define an endpoint

Open urls.py in your code, and type the following:

from stores import views as stores_viewsrouter = DefaultRouter()  
router.register(r'stores',stores_views.StoreView,basename='stores')

Here, you import your Store views from the stores/views.py file, and create a router instance using Django REST Framework's DefaultRouter method. Then you'll need to add this route to the list of urlpatterns — path('', include(router.urls)),.

Step #2: Define the view for your endpoint

The next step is to create the Store view that you just imported in the previous step. Add the following in stores/views.py:

A view allows you to write a handler for your API. To put it simply, it's just a Python Class or a function that takes in a web request and returns a response.

You can use either Django's built-in views, or in this case, a Viewset — an abstration provided by the Django rest Framework module. Using a Viewset allows you to standardize handling of APIs for commonly requested information — like returning a list, or details of a particular Model.

Since you need to return a list of Stores, you'll use the ModelViewset and use the built-in list action for our handler. Along with this, you'll also need to provide the following:

A Queryset: This has the effect of querying your underlying database
A Serializer Class: To configure your view to use a serializer.

You'll now need to define the method get_nearby_stores_within method for querying the database for store related information. Finally, you'll create the serializer NearbyStoreSerializer which will allow you to define what Store related information you want to return.

Step #3: Define a method to query your database for nearby stores

You'll create the method get_nearby_stores_within to which you'll pass latitude, longitude, a distance parameter, and an srid:

annotate: This method adds a new field (in this case, distance) to each object that is returned within a query. The value of each distance field is calculated using GeoDjango's Distance function.
Distance: This calculates the distance between two sets of coordinates — coordinates of the store location, and coordinates of the current user's location.
filter: This allows you to return only a subset of the set of objects. In this case, we want to apply a filter that selects only the objects that lie within the first x kilometres. For this, we'll use GeoDjango's measurement API D (alias for Distance)
order_by: Finally, we'll order the results sorted by the distance field that we have from the annotate step.

Step #4: Define a serializer

Open serializers.py and add the following code:

Your Store model doesn't actually have the distance field, but was annotated while constructing the Response in the earlier step. Therefore, your Serializer must explicitly be told how to look up this information, by providing a method of the form get_annotatedfieldname.

Whew that was a lot! Run python manage.py runserver, fire up Postman or any API client of your choice (or you could use the browser as well), and go to http://localhost:8000/stores?lat=12.978624&lng=77.645296 (Replace these values with the latitude/longitude of your own). Based on the latitude and longitude information you passed in the request, you'll see all nearby stores for that location!

Seeing the list of stores as JSON is a good start. But that's not nearly as fun as seeing all the stores on a map!

For this, you'll need an HTML page. So here's what you'll do next:

Write an API to serve an HTML page

Your app will serve an HTML page with a welcome message when someone heads over to the URL http://localhost:8080/home?username=john.

Step #1: Define the endpoint

In-order for this feature to work, you'll use the home Django application.

Open urls.py and add the route for your home page:

from home import views as home_viewsrouter.register(r'home', home_views.HomePage, basename='home')

Step #2: Add the view

Write this inside home/views.py:

from django.shortcuts import render  
from rest_framework import viewsets  
from rest_framework.decorators import action  
from rest_framework.response import Response  
from rest_framework.renderers import TemplateHTMLRenderer\# Create your views here.  
class HomePage(viewsets.GenericViewSet):  
    renderer_classes = [TemplateHTMLRenderer]  
    template_name = 'home/index.html'def list(self, request):  
        username = self.request.query_params.get('username')  
        return Response({'username': username})

Upon hitting the endpoint, Django will now look for the file index.html within the home/templates folder. Then, it intrapolates the variable username in the template file with the value that you provide in the query params — so when you open the page, you'll see "Welcome, john" displayed in the UI.

Frontend

In your home application, Here are the relevant functions that you'll work on:

map/addMap() : For loading the map.
map/addGeocoder() : For adding a search box in the map with autocomplete feature.
api/fetchNearbyStores() : For making a network request to fetch nearby stores from a given latitude and longitude.
map/convertToGeoJson() : For converting the fetched stores to GeoJSON.
map/plotStoresOnMap : For plotting the stores on the map.

There is some boilerplate JavaScript code inside index.js that calls the above methods. So you'll just need to write the logic for making the relevant functions work.

Note: Since the focus of this tutorial is on logic and not styling, all the relevant boilerplate HTML and CSS code along with the relevant files like stylesheets, icons, and client-side libraries etc have already been included for you. You'll find these files within the /home/static/home folder.

Step #1: Load the map

First, create an account on Mapbox and add an access token. Copy and paste it inside home/js/map.js:

mapboxgl.accessToken = 'your-secret-token';

In the same file, add the following the addMap() function:

const map = new mapboxgl.Map({  
    container: 'map',  
    style: 'mapbox://styles/mapbox/light-v10',  
    center: [77.645296, 12.978624],  
    zoom: 2  
});

map.addControl(new mapboxgl.NavigationControl());  

return map;

Refresh the page, and you should now see a map rendered inside the div element #map! Let's also add an autocomplete field in the map. Add the following inside the addGeocoder() function:

const geocoder = new MapboxGeocoder({ accessToken: mapboxgl.accessToken, mapboxgl: mapboxgl });map.addControl(geocoder);geocoder.on("result", (data) => {  
    geocoderCallback(data);  
});

So you'll be able to type the name of your neighborhood on the search box (for example, Lincoln Square, Illiniois, Chicago). Upon selecting the place, it will pass the data of that place (along with its coordinates) to the geocoderCallback. You'll then be able to use the latitude and longitude information to hit the /stores endpoint that you wrote earlier, and get back a list of stores as JSON.

Step #2: Fetch nearby stores and convert them to GeoJSON

For displaying those little green store markers on the map, Mapbox needs data to be in the form of GeoJSON. GeoJSON is a format for encoding a variety of geographic data structures. This is how it looks like:

{  
    type: "FeatureCollection",  
    features: [  
        {  
                type: "Feature",  
                geometry: {  
                    type: "Point",  
                    coordinates: [<longitude>, <latitude>]  
                },  
                properties: {  
                    <field1>: <value1>,  
                    <field2>: <value2>,  
                        ...  
                }  
        }  
        ...  
    ]  
}

Write the following within the convertToGeoJson function inside map.js:

function convertToGeoJson(stores) {return {  
    type: "FeatureCollection",  
    features: stores.map(store => {  
        return {  
            type: "Feature",  
            geometry: {  
                type: 'Point',  
                coordinates: [store.longitude, store.latitude]  
            },  
            properties: {  
                id: store.id,  
                name: store.name,  
                address: store.address,  
                phone: store.phone,  
                distance: store.distance,  
                rating: store.rating,  
            }  
        }  
    })  
  }}

Finally, you'll have a function called plotStoresOnMap inside that takes in 2 arguments — map, and stores in the form of GeoJSON, and plot those pretty little green icons on the map!

function plotStoresOnMap(map, storesGeoJson) {  
    for(let store of storesGeoJson.features) {  
        // create a HTML element for each feature  
        let el = document.createElement('div');  
        el.className = 'store';  
        el.title = `${store.properties.name}\n` +  
        `approximately ${store.properties.distance.toFixed(2)} km away\n` +  
        `Address: ${store.properties.address || "N/A"}\n` +  
        `Phone: ${store.properties.phone || "N/A"}\n` +  
        `Rating: ${store.properties.rating || "N/A"}`; // make a marker for each feature and add to the map  
        new mapboxgl.Marker(el)  
            .setLngLat(store.geometry.coordinates)  
            .addTo(map); el.addEventListener('click', function(e) {  
            updateSelectedStore(store.properties.id);  
        });
    }  
}

For each store, you create an HTML Div element and add the relevant information of the store such as the name, contact details, how far away it is, etc on the title attribute. So when you place your mouse over the marker, you'll see this info on a tooltip!

Then, you create a new Mapbox marker using the element, set its coordinates, and add it to the map! You also have an event listener that watches for any click events on a marker. If you select a particular store by clicking on it, it will assign a global variable STORE with the store's unique ID.

Note: The title attribute isn't great for accessibility. You'll learn a better technique later.

And that's it! Refresh the page, and on the input box displayed on the map, type in a place within the city that you added the data for earlier (for example, Lincoln Park, Chicago). Upon selecting the place, the geocoderCallbackruns a function called displayNearbyStores(inside stores.js). This method fetches the nearby stores using the Fetch API, converts them to GeoJSON using convertToGeoJson, and plots them on the map using plotStoresOnMap.

And, we're done!

What's next?

In the next part, you'll learn to extend this service further with additional features needed for the service:

A feature to add a wishlist of essential items to purchase from a nearby store using a simple user interface.
A location-based wishlist discovery service that will show you wishlists created by other people in the neighborhood.
Ability to accept a wishlist from another user and purchase the items on their behalf.

So take a break, get something nice for yourself from your favorite store, and proceed with Part 2!

Here's a list of several such useful applications built using OSM. Many of these projects need contributors in various roles — I highly encourage you to find one that you like and offer any help that you can! ↩
BTW, did you know that gamers have used this tool to find pokemon habitats in the Pokemon Go Game? 😉 ↩

While applying for jobs, do you mention skills that you learned via side-projects but never used them at your work?

Ashwin Hariharan — Wed, 02 Jun 2021 12:11:55 +0000

Hi everyone! I'm a software engineer who recently transitioned into Developer Relations. I like programming and enjoy learning and tinkering with new technologies - and they aren't necessarily stuff that I use or require everyday at my workplace. Pretty sure that I'm not the only one! 😇

When I look at the stuff that I know, I can divide them into 2 broad categories:

Grade A skills:

These are stuff that I require very less ramp-up time, because I've had enough exposure with them and use them almost everyday at work. For me, they happen to be the following:

JavaScript / TypeScript
Node.js / Express
MongoDB
React

If I'm ever asked questions like "How many years of experience do you have in x", and it happens to fall under Grade A, I can answer it somewhat accurately (for instance, 2 years or 4 years or whatever).

Grade B skills:

These are skills that I never had a chance to work on or use in production at work, but I acquired them while working on my side projects. Here are some of my own skills that fall under this category:

Python
Django
PostgreSQL
Docker
Scikit-learn
Keras
Figma

For this category, answering questions like "how many years of experience do you have" becomes difficult, because I use them every now and then, not every single day. However, I can still get things done with them and would like the person going through my profile to know that. I may not be as comfortable in these as my Grade A skills, but I still know much more than just writing a simple hello world program (or its equivalent).

If you can relate with the above, I would love to know from you on:

Along with Grade A, do you also mention Grade B skills in your resume, CV or in job applications?
If the answer to the above is yes, how do you list them? Do you make a distinction between Grade A and Grade B, or do you just club them all together?
If you're asked "How many years of experience do you have in x skill", and that skill happens to be a Grade B skill, how do you answer it?
If in a job posting, you find that 60% of the required skill-set fall under your Grade B and 40% fall under Grade A, do you still apply for the role?

The Idea of an Invigorating Startup Life can Often be an Illusion

Ashwin Hariharan — Fri, 18 Dec 2020 05:37:32 +0000

Photo by Sebastian Herrmann on Unsplash

Working in a startup can be fun. Many people coming from large companies to work with startups claim that the environment is full of positive vibes, every single day truly feels like an opportunity to do something innovative. The words 'disruptive’ and 'startups’ often go along together.

Some few years ago I decided to quit my job, mentor in web development workshops and then take a vacation to see my family. They lived in a different city, and I hadn’t seen them for almost an year. Felt like it would be nice to spend some quality time with them, before I start figuring out what the hell to do next.

I was working with startups all this time in Bengaluru, often described as the Silicon Valley of India LOL — and while it can be exciting to work in one, it can also be gruelling. In the beginning it feels great. When I just got out of college I was really desperate about landing a good role, and to prove just how much I cared about this wonderful app that they were building.

I would basically tell them anything that they needed to hear, and interactions with other fellow developers tell me that they too more-or-less do the same thing!

Often in the beginning of our career we’re naive and uninformed, and so we’re unable to evaluate our career prospects thoughtfully. If you’re thinking of taking a job in a new venture or already working in such a company, you should really do some introspection — mostly in the lines of self growth. Basically trying to find the answer to

Can I grow as a person and as a professional?

In the context of the startup world, this question can be answered by asking yourself a series of sub-questions —

Do you find yourself being able to constantly improve your skills or explore new stuff throughout the time that you’re planning to spend there?

During interviews, startup founders talk about the ground breaking stuff that they're doing and often oversell themselves, and you’re inclined to believe them. You’ll be told about the incredible learning opportunity that you’d get, if you joined. For a long time, it was my favourite thing to hear, and often I wouldn't negotiate salaries thinking that the offer was already so generous.

When I finally update my LinkedIn profile, it’s gonna look so awesome!

But that’s oversimplified thinking. It took me a while to realize that there's always gonna be learning in the beginning. If the company is into product development that consists of front-end and back-end, you’ll learn enough to use hacks and make something work.

A major part of your work will be about implementing new features and fixing bugs — which becomes straight-forward once you learn how to hack your way around code. But after the initial phase, you might not have the feasibility to learn anything more — you’re gonna be restricted to a specific role for a long time and can learn something only if the situation demands for it.

There might be a hiatus in your learning curve after a while.

For instance, in my case I always wanted to explore machine learning. Or learn Photoshop maybe? But since my job didn’t necessitate for these skills, I had no other choice but to try carving out time after office hours to do all those things.

Which brings us to the next question —

Do you get enough time to do the things that you really really want to do?

In startups, a 9 AM to 6 PM job is almost unheard of. Most of them would tell you that they have flexible timings, but it's mostly for the company's benefit, not yours. Usually you’d see people working late, staying up until midnight almost every other day.

You might try out time management hacks, take fewer breaks, try to do some of your learning in your office itself. But that’s about the best you can do. There’s always more work.

In my instance, I thought if I came home early I could spend few hours working on open source projects or watch some videos on Coursera. Maybe work on that half built app that I always wanted to finish. But I couldn’t, since I often had to stay back in office late.

Trying to do too many things also drains your productivity — you need to be taking enough breaks if you don’t wanna experience burn-outs. Hard work is important, but if you're constantly getting stressed out then you cease to be effective.

Busy does not necessarily equate to productive.

Hitting that sweet spot

You don’t wanna come home everyday looking like this

We’re humans after all, we’re not machines that are designed to take in coffee and productivity blogs as inputs and just transform into Elon Musk, being super productive, super motivated all the time.

So do you have a life outside of work?

Call it work-life balance, pursuing your interests, passions, hobbies, whatever. They’re all the same.

Many startups have 6-day work weeks. No Saturdays off meant that you couldn’t attend tech meetups, or mentor in workshops if you wanted to. Good luck in trying to finish that course work on Sunday or pursuing your hobbies, juggling between doing your laundry, spending time with your family and running other errands — because before you even realize it, the day is over.

You’ll frequently hear this during interviews, or when you talk about getting few days off —

“We’ve got lot of work to do, and we’re all trying to be as agile as possible and working really hard."

or this —

“In another few months once we’ve released our first public beta then things will start going a little easier and we can take it in a relaxed pace"

There’s always pressure to stay ahead of the competition, to ship faster, get users and start becoming profitable. I can't say I blame them entirely — when you’re overwrought, no amount of work seems enough. Convincing your boss to give you a few days off seems like an impossible thing to do.

What about those startups that offer an unlimited vacation policy? I’ve never been in one of those, but statistics indicate that employees take even fewer time off for themselves in those start-ups.

Giving up on your interests and hobbies is a bad idea, because these are the very things that make you who you are. They’re not useless, they have an effect on how you approach problems, how you deal with a crisis at work, in your interactions with your co-workers. All adding up to how others perceive you as a professional.

So, how was the journey so far?

I have mixed feelings about the whole experience. Often it felt like we were sailing in this small speedboat, trying to overtake those gigantic, slow moving ships. The enthusiasm was often intoxicating —

The unpredictability also meant that things would go bat-shit crazy at times, especially when something breaks in production and it’s demo day…

So the answer to whether you should be working in a startup or not is a little more complex than a simple yes or no —

Identify a place where you have a good pace of learning, a place where the work doesn’t overwhelm you. It’s very important that you not just negotiate for salaries, but that you also negotiate your work hours — a fixed number of hours that you’ll put in every weekday and no more than that, or perhaps Saturdays off, or perhaps remote working options. Also, read these Ten rules for Negotiating a job offer.
It’s always a good idea to have personal side projects, so ensure you’re able to carve out some time everyday to work on those.
Many early stage startups will not compromise on work hours at all. This is especially true in Asian countries like India. Moreover if you have less than a couple years experience, then it’s likely that you’ll be paid very less.

In my case it was the latter, so here’s what I did —

I decided to take a sabbatical.

If you’ve worked for couple of years and saved some money, consider taking a career break for anytime between few months to half a year and do everything that you always wanted to do. You’ll know when it’s the right time to take one.

It might feel unnerving at first, but later it starts feeling refreshing. You can take the week long vacation that you always dreamed about. Develop good habits. Meet interesting people. Learn to play a musical instrument, because that can help you write software better. Or go on that road trip that you always wanted to go but never could.

And in the meantime, constantly

Identify your own issues and fix those bugs that exist within you. Implement new features in yourself — so that when you get into the swing of things, the next release of you will be super awesome and something to look forward to.

BTW, Would you like to learn my negotiation strategies? Do let me know!

How to Know if Programming is your Cup of Tea

Ashwin Hariharan — Wed, 16 Dec 2020 10:46:19 +0000

Photo by Nathan da Silva on Unsplash

I've been working in tech for a while now. When I look back, it seems unreal. Back in 2014 while I was still in college, if someone told me that I'd be employed, I'd be overjoyed — that's how screwed up I was at academics.

In college I found most of the computer science subjects dry. I mean sure, few of them seemed interesting. But none of them were interesting enough to keep me occupied for very long. C? A nightmare, learning about pointers, malloc and stuff. Relational databases? Boooring! Visual C? Yuck, all that code just to render some crappy looking house ? I can draw better! Java? Meh.

"Can I just go watch Friends on my laptop ? That's fun!"

For the first three years in engineering, that's pretty much how my feelings towards these subjects were. And as you may have guessed, my grades got screwed up, big time (However in my defense, I did enjoy subjects like Unix and Operating systems!).

But when you're surrounded by people who are (A) Great at academics, OR (B) Programming Geeks, OR (C) Both A and B — it makes you think. Makes you worry. It makes you fear that you may never land a job, that even if you do — you'll likely get one that sucks. I had no one else to blame but myself. As someone who was still figuring out on what to do, the world of programming not only intrigued, but also scared me.

Where do I even begin? What do I do? Programming seems tough. Would I even enjoy it? If people keep saying that it's easy, does that mean I'm just plain dumb ?

It was around this time when web development was taking over the world by storm; people had already begun to explore and see what it had to offer. We had Web Programming as a subject in the final year of engineering, so I began to read HTML5, JS and CSS3. After lots of Googling and studying few examples from programming tutorials, very soon I could write a small program that did this —

This was easy enough to do — define classes in CSS3 for the sphere with position attributes and define transitions for them. Then use onclick event handlers in JavaScript to change the class-name of the sphere when you click one of the
boxes, and voila- you got a nice sphere that moves where you tell it to!

And when I saw that little blue ball moving, this is how I felt —

This is so cool! It wasn't nearly as complicated as I thought it would be!
The whole program took less than 20 lines of code
It certainly wasn't boring as writing on some black screen using vi and seeing the outputs again on a black screen that they called the terminal when I wrote C or Perl code

I got hooked, and started to learn more and more. Very soon, I evolved from that simple program that moves a blue sphere, to writing my resume using HTML5, CSS3 and JavaScript —

I knew back then that this was still pretty amateur stuff, but I didn't care. For the first time, I knew that programming was indeed my cup of tea, and this wouldn't have been possible if I didn't write that first program in JavaScript.

Now there are many people who started out with C++ or Java, loved it, and still are writing software using that. But not all of us are like that. One does not simply learn to code'. Many people give up without even testing the waters, and the ones that do — they quit later, because the water's too deep for them. They eventually think that programming isn't their cup of tea.

My first programming language was C. Which is allright, but it can take a while before you finally begin to develop applications. Web development, on the other hand, probably has the lowest barrier to entry.

JavaScript, the primary language of the web, invokes strong reactions from the programming community — people either love it or hate it. I kinda have a love-hate relationship with it as well, and I guess I'll be stuck with it for as long as I make web applications. I for one can't wait to see the web evolve to that point where we can have alternatives.

That being said, I still think you should pick it as your first language.

JavaScript is not judgmental. It does not force you learn a dozen concepts before you can display a static web site on the browser. You don't even have to declare data types. You can execute JavaScript code on a terminal, or if you hate the terminal, you can use the developer console in a web browser, or you can embed it in an HTML page and see the output in the browser. It allows you to do both functional programming and object oriented programming.

I finally got a job at a startup as a Front End Developer. As it turns out, it acted as a sort of ‘gateway' towards learning further programming concepts. Things that sounded boring before, sound very interesting to me now. Pretty soon
I moved on from front-end to full stack — I started to enjoy the process in architecting data models in MongoDB and PostgreSQL, writing RESTful APIs in NodeJS and Golang, using ReactJS for the client, to even starting open source projects.

I do not intend to 'promote' JavaScript in this post — I'm well aware of it's quirks. But we waste so much time — hours and days, or even months trying to figure out whether programming is meant for us, doing things that we don't want to. We'd rather just quit and choose not to do it, because learning things can be a pain in the ass. At such times, having a tool or a method that takes minimal effort to use and that's beginner-friendly can be very helpful. So before you begin to learn, keeping these points in mind will help you manage your expectations-

Every language has it's pitfalls, so does this one. JS is very easy to pick up, but it is pretty hard to master; so don't take it for granted thinking you'll become some sort of 'ninja' or 'rockstar' within a week.
Understand that what you write in the beginning will mostly be considered ‘crap' by other programmers. Don't give up. It's okay to write shitty code in the beginning when you learn. But if you discover that you like it, then it's very essential that you don't write shitty code.
Indentation matters a lot. It helps you avoid bugs, it also makes sure that your code doesn't look like a sea of vomit when someone sees it.
And finally - if you have alternatives to JavaScript, consider using them. The JS ecosystem is highly chaotic, new frameworks keep propping up like flies, and JavaScript fatigue is a real thing.

"All-right. Where do I begin?"

I highly recommend starting from here — From Zero to Front-End Hero (Part 1). It's very cool, I wish I had this tutorial when I began my journey.

This would be enough to help you get started. There are about a million things more to remember, but you wouldn't need to know all of them — not in the beginning at least. What's important to know is whether you would enjoy doing this for the foreseeable future — and the sooner you come to a realization, the better.

So, take a leap of faith. Who knows, maybe you might actually like it! And if you hone your craft and work diligently, you might just find yourself cast in the role of a lifetime.

Interested to learn something new? Check out my repositories:

How to build Your Own Uber-for-X App
Beginner-friendly projects built using JavaScript, Python and more!

DEV Community: Ashwin Hariharan

Organizing AI Applications: Lessons from traditional software architecture

What Makes AI projects maintainable

Clear Ownership Boundaries

Reusability across entry points

Testability

Provider Independence

Architectural patterns for AI apps

Pattern 1: Structure by business components or modules

Pattern 2: Layer your feature modules with 3-tier architecture

Pattern 3: Tools and prompts should call domain logic, not implement it

Pattern 4: Dependency Inversion

Pattern 5: Use environment-aware, secure, and hierarchical config

Pattern 6: Separate persistent data from agent memory

redis-developer / restaurant-discovery-ai-agent-demo

An Agentic AI restaurant discovery platform that combines Redis's speed with LangGraph's intelligent workflow orchestration. Get personalized restaurant recommendations, make reservations, and get lightning-fast responses through semantic caching.

🍽️ Restaurant Discovery AI Agent

App screenshots

Tech Stack

Product Features

Next steps

How to use Elasticsearch to Create an Exceptional User Experience in Retail

Table of contents

Elasticsearch

Search in Retail

Search Statistics

Elasticstack in Retail

Search Experience in online shopping

Text Search

Search within departments or categories

Filtering

Sort search results

What you will learn

booleanhunter-tech-blog / e3-retail

An E-commerce retail website demonstrating Elasticsearch

E³Retail

App Screenshots

Home Page

Products filter page

Features

Search features

Requirements

Instructions to install

Common commands

Creating an index

Bulk uploading Data

Data Exploration

Data types and field mappings

Kibana Developer Console

Get information about documents in an index

Find the number of products in each category

Common terminologies

Ranking

Score

Searching for information

Simple Text Queries

Query #1: Text search for a product

Query #2: Text search for a product with a brand name or a certain phrase

Query #3: Search for a particular category of products

Compound Queries

Query #4: Text search for products within a certain category

Query #5: Search and filter for products by ratings

Query #6: Filter by pricing parameters

Sorting search results

Improving search relevance

Precision

Recall

Boost the relevance of individual fields

Decrease the relevance of documents for certain search terms

Enhancing e-shopping experience even further

Summing up

Resources

What the heck is a Service Mesh, anyway?

A conceptual container orchestration setup

What is a Service Mesh?

Wait, but why though?

Istio

Other implementations

Istio under the hood

Data plane