Large language models (LLMs) powered by artificial intelligence are gaining immense popularity, with over 325,000 models available on Hugging Face. As more models emerge, a key question is whether to use proprietary or open-source LLMs.

What are LLMs and How Do They Differ?LLMs leverage deep learning and massive datasets to generate human-like textProprietary LLMs are owned and controlled by a companyOpen-source LLMs are freely accessible for anyone to use and modifyProprietary models currently tend to be much larger in terms of parametersHowever, size isn’t everything – smaller open-source models are rapidly catching upCommunity contributions empower the evolution of open-source LLMs

Benefits of Open Source LLMsTransparency – Better visibility into model architecture, training data, output generationCustomization through fine-tuning custom datasets for specific use casesCommunity contributions across diverse perspectives enable experimentationUse Cases

Open-source LLMs are being deployed across industries:

HealthcareDiagnostic assistanceTreatment optimizationFinanceApplications like FinGPT for financial analysisScienceModels like NASA’s trained on geospatial dataLeading Models on Hugging Face

The Hugging Face model leaderboard’s latest benchmarks.

Top LLMs on Hugging Face

Downside of Open-source LLMs

Despite advances, LLMs have concerning have 3 major limitations:

Inaccuracy – Hallucinations from inaccurate/incomplete training dataSecurity – Potential exposure of private data in outputsBias – Embedding biases that skew outputs

Mitigating these risks in early-stage LLMs remains vital.

The Bottom Line

Open-source big language models make AI more available to everyone. This widens who can use them. But risks are still there. Even so, putting information out in the open and letting users adjust models to their needs gives power to people across fields.

In-memory caching and in-memory data storage are both techniques used to improve the performance of applications by storing frequently accessed data in memory. However, they differ in their approach and purpose.

What is In-Memory Caching?

In-memory caching is a method where data is temporarily stored in the system’s primary memory (RAM). This approach significantly reduces data access time compared to traditional disk-based storage, leading to faster retrieval and improved application performance.

Key Features:Speed: Caching provides near-instant data access, crucial for high-performance applications.Temporary Storage: Data stored in a cache is ephemeral, and primarily used for frequently accessed data.Reduced Load on Primary Database: By storing frequently requested data, it reduces the number of queries to the main database.Common Use Cases:Web Application Performance: Improving response times in web services and applications.Real-Time Data Processing: Essential in scenarios like stock trading platforms where speed is critical.

What is an In-Memory Data Store?

An In-Memory Data Store is a type of database management system that utilizes main memory for data storage, offering high throughput and low-latency data access.

Key Features:Persistence: Unlike caching, in-memory data stores can persist data, making them suitable as primary data storage solutions.High Throughput and Low Latency: Ideal for applications requiring rapid data processing and manipulation.Scalability: Easily scalable to manage large volumes of data.Common Use Cases:Real-Time Analytics: Used in scenarios requiring quick analysis of large datasets, like fraud detection systems.Session Storage: Maintaining user session information in web applications.

Comparing In-Memory Caching and In-Memory Data Store

Aspect
In-Memory Caching
In-Memory Data Store

Purpose
Temporary data storage for quick access
Primary data storage for high-speed data processing

Data Persistence
Typically non-persistent
Persistent

Use Case
Reducing database load, improving response time
Real-time analytics, session storage, etc.

Scalability
Limited by memory size, often used alongside other storage solutions
Highly scalable, can handle large volumes of data

Advantages and LimitationsIn-Memory Caching

Advantages:

Reduces database load.Improves application response time.

Limitations:

Data volatility.Limited storage capacity.In-Memory Data Store

Advantages:

High-speed data access and processing.Data persistence.

Limitations:

Higher cost due to large RAM requirements.Complexity in data management and scaling.

Choosing the Right Approach

The choice between in-memory caching and data store depends on specific application needs:

Performance vs. Persistence: Choose caching for improved performance in data retrieval and in-memory data stores for persistent, high-speed data processing.Cost vs. Complexity: In-memory caching is less costly but might not offer the complexity required for certain applications.Summary

To summarize, some key differences between in-memory caching and in-memory data stores:

Caches hold a subset of hot data, and in-memory stores hold the full dataset.Caches load data on demand, and in-memory stores load data upfront.Caches synchronize with the underlying database asynchronously, and in-memory stores sync writes directly.Caches can expire and evict data, leading to stale data. In-memory stores always have accurate data.Caches are suitable for performance optimization. In-memory stores allow new applications with real-time analytics.Caches lose data when restarted and have to repopulate. In-memory stores maintain data in memory persistently.Caches require less memory while in-memory stores require sufficient memory for the full dataset.

NGINX as a reverse proxy has long been a popular choice for its efficiency and reliability. However, Cloudflare announced their decision to move away from NGINX to their homegrown open-source solution for reverse proxy, Pingora.

What is Reverse Proxy?

A reverse proxy sits in front of the origin servers and acts as an intermediary, receiving requests, processing them as needed, and then forwarding them to the appropriate server. It helps improve performance, security, and scalability for websites and web applications.

Imagine you want to visit a popular website like Wikipedia. Instead of going directly to Wikipedia’s servers, your request first goes to a reverse proxy server.

The reverse proxy acts like a middleman. It receives your request and forwards it to one of Wikipedia’s actual servers (the origin servers) that can handle the request.

When the Wikipedia server responds with the requested content (like a web page), the response goes back to the reverse proxy first. The reverse proxy can then do some additional processing on the content before sending it back to you.

Reverse Proxy is used for:Caching: The reverse proxy stores frequently requested content in its memory. So if someone else requests the same Wikipedia page, the reverse proxy can quickly serve it from the cache instead of going to the origin server again.Load balancing: If there are multiple Wikipedia servers, the reverse proxy can distribute incoming requests across them to balance the load and prevent any single server from getting overwhelmed.Security: The reverse proxy can protect the origin servers by filtering out malicious requests or attacks before they reach the servers.Compression: The reverse proxy can compress the content to make it smaller, reducing the amount of data that needs to be transferred to you.SSL/TLS termination: The reverse proxy can handle the encryption/decryption of traffic, offloading this work from the origin servers.Why Does Cloudflare Have a Problem with NGINX?

While NGINX has been a reliable workhorse for many years, Cloudflare encountered several architectural limitations that prompted it to seek an alternative solution. One of the main issues was NGINX’s process-based model. Each request was handled by a separate process, which led to inefficiencies in resource utilization and memory fragmentation.

Another challenge Cloudflare faced was the difficulty in sharing connection pools among worker processes in NGINX. Since each process had its isolated connection pool, Cloudflare found itself executing redundant SSL/TLS handshakes and connection establishments, leading to performance overhead.

Furthermore, Cloudflare struggled with adding new features and customizations to NGINX due to its codebase being written in C, a language known for its memory safety issues.

How Cloudflare Built Its Reverse Proxy “Pingora” from Scratch?

Being Faced with these limitations, Cloudflare considered several options, including forking NGINX, migrating to a third-party proxy like Envoy, or building their solution from scratch. Ultimately, they chose the latter approach, aiming to create a more scalable and customizable proxy that could better meet their unique needs.

Feature
NGINX
Pingora

Architecture
Process-based
Multi-threaded

Connection Pooling
Isolated per process
Shared across threads

Customization
Limited by configuration
Extensive customization via APIs and callbacks

Language
C
Rust

Memory Safety
Prone to memory safety issues
Memory safety guarantees with Rust

To address the memory safety concerns, Cloudflare opted to use Rust, a systems programming language known for its memory safety guarantees and performance. Additionally, Pingora was designed with a multi-threaded architecture, offering advantages over NGINX’s multi-process model.

With the help of multi-threading, Pingora can efficiently share resources, such as connection pools, across multiple threads. This approach eliminates the need for redundant SSL/TLS handshakes and connection establishments, improving overall performance and reducing latency.

The Advantages of Pingora

One of the main advantages of Pingora is its shared connection pooling capability. By allowing multiple threads to access a global connection pool, Pingora minimizes the need for establishing new connections to the backend servers, resulting in significant performance gains and reduced overhead.

Cloudflare also highlighted Pingora’s multi-threading architecture as a major benefit. Unlike NGINX’s process-based model, which can lead to resource contention and inefficiencies, Pingora’s threads can efficiently share resources and leverage techniques like work stealing to balance workloads dynamically.

Pingora: A Rust Framework for Network Services

Interestingly, Cloudflare has positioned Pingora as more than just a reverse proxy. They have open-sourced Pingora as a Rust framework for building programmable network services. This framework provides libraries and APIs for handling protocols like HTTP/1, HTTP/2, and gRPC, as well as load balancing, failover strategies, and security features like OpenSSL and BoringSSL integration.

The selling point of Pingora is its extensive customization capabilities. Users can leverage Pingora’s filters and callbacks to tailor how requests are processed, transformed, and forwarded. This level of customization is particularly appealing for services that require extensive modifications or unique features not typically found in traditional proxies.

The Impact on Service Meshes

As Pingora gains traction, it’s natural to wonder about its potential impact on existing service mesh solutions like Linkerd, Istio, and Envoy. These service meshes have established themselves as crucial components in modern microservices architectures, providing features like traffic management, observability, and security.

While Pingora may not directly compete with these service meshes in terms of their comprehensive feature sets, it could potentially disrupt the reverse proxy landscape. Service mesh adopters might consider leveraging Pingora’s customizable architecture and Rust-based foundation for building their custom proxies or integrating them into their existing service mesh solutions.

The Possibility of a “Vanilla” Pingora Proxy

Given Pingora’s extensive customization capabilities, some speculate that a “vanilla” version of Pingora, pre-configured with common proxy settings, might emerge in the future. This could potentially appeal to users who desire an out-of-the-box solution while still benefiting from Pingora’s performance and security advantages.

What is Memos?

Memos is an open-source, privacy-first, and lightweight note-taking application service that allows you to easily capture and share your thoughts.

Memos features:Open-source and free foreverSelf-hosting with Docker in secondsPure text with Markdown supportCustomize and share notes effortlesslyRESTful API for third-party integrationSelf-Hosting Memos with Docker and MySQL Database

You can self-host Memos quickly using Docker Compose with a MySQL database.

Prerequisites: Docker and Docker Compose installed

You have two options to choose MySQL or MariaDB as a Database both are stable versions and MariaDB consumes less memory than MySQL.

Memos with MySQL 8.0version: “3.0”

services:

mysql:
image: mysql:8.0
environment:
TZ: Asia/Kolkata
MYSQL_ROOT_PASSWORD: memos
MYSQL_DATABASE: memos-db
MYSQL_USER: memos
MYSQL_PASSWORD: memos
volumes:
– mysql_data:/var/lib/mysql
healthcheck:
test: [“CMD”, “mysqladmin”, “ping”, “-h”, “localhost”]
timeout: 20s
retries: 10
restart: always

memos:
image: neosmemo/memos:stable
container_name: memos
environment:
MEMOS_DRIVER: mysql
MEMOS_DSN: memos:memos@tcp(mysql:3306)/memos-db
depends_on:
mysql:
condition: service_healthy
volumes:
– ~/.memos/:/var/opt/memos
ports:
– “5230:5230”
restart: always

volumes:
mysql_data:

Memos with MySQL Database Docker Compose

ORMemos with MariaDB 11.0version: “3.0”
services:
mariadb:
image: mariadb:11.0
environment:
TZ: Asia/Kolkata
MYSQL_ROOT_PASSWORD: memos
MYSQL_DATABASE: memos-db
MYSQL_USER: memos
MYSQL_PASSWORD: memos
volumes:
– mariadb_data:/var/lib/mysql
healthcheck:
test: [“CMD”, “healthcheck.sh”, “–connect”, “–innodb_initialized”]
start_period: 10s
interval: 10s
timeout: 5s
retries: 3
restart: always

memos:
image: neosmemo/memos:stable
container_name: memos
environment:
MEMOS_DRIVER: mysql
MEMOS_DSN: memos:memos@tcp(mariadb:3306)/memos-db
depends_on:
mariadb:
condition: service_healthy
volumes:
– ~/.memos/:/var/opt/memos
ports:
– “5230:5230”
restart: always

volumes:
mariadb_data:

Memos with MariaDB Database Docker Compose

Create a new file named docker-compose.yml and copy the above content.This sets up a MariaDB 11.0 database service and the Memos app linked to it.Run docker-compose up -d to start the services in detached mode.Memos will be available at http://localhost:5230.The configurations are:mysql service runs MySQL 8.0 with a database named memos-db.memos service runs the latest Memos images, and links to the mysql/mariadb service.MEMOS_DRIVER=mysql tells Memos to use the MySQL database driver.MEMOS_DSN contains the database connection details.The ~/.memos the directory is mounted for data persistence.

You can customize the MySQL password, database name, and other settings by updating the environment variables.

Configuring S3 Compatible Storage

Memos support integrating with S3-compatible object storage like Amazon S3, Cloudflare R2, DigitalOcean Spaces, etc

To use AWS S3/ Cloudflare’s R2 as object storageSettings > StorageCreate a S3/Cloudflare R2 bucketGet the API token with object read/write permissionsIn Memos Admin Settings > Storage, create a new storageEnter details like Name, Endpoint, Region, Access Key, Secret Key, Bucket name and Public URL (For Cloudflare R2 set Region = auto)Save and select this storageFor Cloudflare R2 set Region = auto

With this setup, you can self-host the privacy-focused Memos note app using Docker Compose with a MySQL database, while integrating scalable S3 or R2 storage for persisting data.

A French startup, Mistral AI has released two impressive large language models (LLMs) – Mistral 7B and Mixtral 8x7B. These models push the boundaries of performance and introduce a better architectural innovation aimed at optimizing inference speed and computational efficiency.

Mistral 7B: Small yet Mighty

Mistral 7B is a 7.3 billion parameter transformer model that punches above its weight class. Despite its relatively modest size, it outperforms the 13 billion parameters Llama 2 model across all benchmarks. It even surpasses the larger 34 billion parameter Llama 1 model on reasoning, mathematics, and code generation tasks.

Two foundations of Mistral 7B’s efficiency:

Grouped Query Attention (GQA) Sliding Window Attention (SWA)

GQA significantly accelerates inference speed and reduces memory requirements during decoding by sharing keys and values across multiple queries within each transformer layer.

SWA, on the other hand, enables the model to handle longer input sequences at a lower computational cost by introducing a configurable “attention window” that limits the number of tokens the model attends to at any given time.

Name
Number of parameters
Number of active parameters
Min. GPU RAM for inference (GB)

Mistral-7B-v0.2
7.3B
7.3B
16

Mistral-8X7B-v0.1
46.7B
12.9B
100

Mixtral 8x7B: A Sparse Mixture-of-Experts Marvel

While Mistral 7B impresses with its efficiency and performance, Mistral AI took things to the next level with the release of Mixtral 8x7B, a 46.7 billion parameter sparse mixture-of-experts (MoE) model. Despite its massive size, Mixtral 8x7B leverages sparse activation, resulting in only 12.9 billion active parameters per token during inference.

Image Credit: Mistral.ai

The key innovation behind Mixtral 8x7B is its MoE architecture. Within each transformer layer, the model has eight expert feed-forward networks (FFNs). For every token, a router mechanism selectively activates just two of these expert FFNs to process that token. This sparsity technique allows the model to harness a vast parameter count while controlling computational costs and latency.

According to Mistral AI’s benchmarks, Mixtral 8x7B outperforms or matches the large language models like Llama 2 70B and GPT-3.5 across most multiple tasks, including reasoning, mathematics, code generation, and multilingual benchmarks. Additionally, it provides 6x faster inference than Llama 2 70B, thanks to its sparse architecture.

Both Mistral 7B and Mixtral 8x7B are good at code generation tasks like HumanEval and MBPP, with Mixtral 8x7B having a slight edge and it’s better. Mixtral 8x7B also supports multiple languages, including English, French, German, Italian, and Spanish, making them valuable assets for multilingual applications.

On the MMLU benchmark, which evaluates a model’s reasoning and comprehension abilities, Mistral 7B performs equivalently to a hypothetical Llama 2 model over three times its size.

LLMs Benchmark Comparison Table

Model
Average MCQs
Reasoning
Python coding
Future Capabilities
Grade school math
Math Problems

Claude 3 Opus
84.83%
86.80%
95.40%
84.90%
86.80%
95.00%

Gemini 1.5 Pro
80.08%
81.90%
92.50%
71.90%
84%
91.70%

Gemini Ultra
79.52%
83.70%
87.80%
74.40%
83.60%
94.40%

GPT-4
79.45%
86.40%
95.30%
67%
83.10%
92%

Claude 3 Sonnet
76.55%
79.00%
89.00%
73.00%
82.90%
92.30%

Claude 3 Haiku
73.08%
75.20%
85.90%
75.90%
73.70%
88.90%

Gemini Pro
68.28%
71.80%
84.70%
67.70%
75%
77.90%

Palm 2-L
65.82%
78.40%
86.80%
37.60%
77.70%
80%

GPT-3.5
65.46%
70%
85.50%
48.10%
66.60%
57.10%

Mixtral 8x7B
59.79%
70.60%
84.40%
40.20%
60.76%
74.40%

Llama 2 – 70B
51.55%
69.90%
87%
30.50%
51.20%
56.80%

Gemma 7B
50.60%
64.30%
81.2%
32.3%
55.10%
46.40%

Falcon 180B
42.62%
70.60%
87.50%
35.40%
37.10%
19.60%

Llama 13B
37.63%
54.80%
80.7%
18.3%
39.40%
28.70%

Llama 7B
30.84%
45.30%
77.22%
12.8%
32.6%
14.6%

Grok 1
–
73.00%
–
63%
–
62.90%

Qwen 14B
–
66.30%
–
32%
53.40%
61.30%

Mistral Large
–
81.2%
89.2%
45.1%
–
81%

This model comparison table was last updated in March 2024. Source

When it comes to fine-tuning for specific use cases, Mistral AI provides “Instruct” versions of both models, which have been optimized through supervised fine-tuning and direct preference optimization (DPO) for careful instruction following.

Deployment and Accessibility

Mistral AI has made both Mistral 7B and Mixtral 8x7B available under the permissive Apache 2.0 license, allowing developers and researchers to use these models without restrictions. The weights for these models can be downloaded from Mistral AI’s CDN, and the company provides detailed instructions for running the models locally, on cloud platforms like AWS, GCP, and Azure, or through services like HuggingFace.

LLMs Cost and Context Window Comparison Table

Models
Context Window
Input Cost / 1M tokens
Output Cost / 1M tokens

Gemini 1.5 Pro
128K
N/A
N/A

Mistral Medium
32K
$2.7
$8.1

Claude 3 Opus
200K
$15.00
$75.00

GPT-4
8K
$30.00
$60.00

Mistral Small
16K
$2.00
$6.00

GPT-4 Turbo
128K
$10.00
$30.00

Claude 2.1
200K
$8.00
$24.00

Claude 2
100K
$8.00
$24.00

Mistral Large
32K
$8.00
$24.00

Claude Instant
100K
$0.80
$2.40

GPT-3.5 Turbo Instruct
4K
$1.50
$2.00

Claude 3 Sonnet
200K
$3.00
$15.00

GPT-4-32k
32K
$60.00
$120.00

GPT-3.5 Turbo
16K
$0.50
$1.50

Claude 3 Haiku
200K
$0.25
$1.25

Gemini Pro
32K
$0.125
$0.375

Grok 1
64K
N/A
N/A

This cost and context window comparison table was last updated in March 2024. Source

For those looking for a fully managed solution, Mistral AI offers access to these models through their platform, including a beta endpoint powered by Mixtral 8x7B.

Conclusion

Mistral AI’s language models, Mistral 7B and Mixtral 8x7B, are truly innovative in terms of architectures, exceptional performance, and computational efficiency, these models are built to drive a wide range of applications, from code generation and multilingual tasks to reasoning and instruction.

Sharing links is an integral part of our daily online communication. However, dealing with long, complex URLs can be a hassle, making remembering and sharing links efficiently difficult.

What is Slash?Slash Link Shortener Dashboard

Slash is an open-source, self-hosted link shortener that simplifies the managing and sharing of links. Slash allows you to create customizable, shortened URLs (called “shortcuts”) for any website or online resource. With Slash, you can say goodbye to the chaos of managing lengthy links and embrace a more organized and streamlined approach to sharing information online.

One of the great things about Slash is that it can be self-hosted using Docker. By self-hosting Slash, you have complete control over your data.

Features of Slash:Custom Shortcuts: Transform any URL into a concise, memorable shortcut for easy sharing and access.Tag Organization: Categorize your shortcuts using tags for efficient sorting and retrieval.Team Sharing: Collaborate by sharing shortcuts with your team members.Link Analytics: Track link traffic and sources to understand usage.Browser Extension: Access shortcuts directly from your browser’s address bar on Chrome & Firefox.Collections: Group related shortcuts into collections for better organization.

Prerequisites:

Method 1: Docker Run CLI

The docker run command is used to create and start a new Docker container. To deploy Slash, run:

docker run -d –name slash -p 5231:5231 -v ~/.slash/:/var/opt/slash yourselfhosted/slash:latest

Let’s break down what this command does:

docker run tells Docker to create and start a new container-d runs the container in detached mode (in the background)–name slash gives the container the name “slash” for easy reference-p 5231:5231 maps the container’s port 5231 to the host’s port 5231, allowing access to Slash from your browser-v ~/.slash/:/var/opt/slash creates a volume to store Slash’s persistent data on your host machineyourselfhosted/slash:latest specifies the Docker image to use (the latest version of Slash)

After running this command, your Slash instance will be accessible at http://your-server-ip:5231.

Method 2: Docker Compose

Docker Compose is a tool that simplifies defining and running multi-container Docker applications. It uses a YAML file to configure the application’s services.

Create a new file named docker-compose.yml and paste the contents of the Docker Compose file provided below.version: ‘3’

services:
slash:
image: yourselfhosted/slash:latest
container_name: slash
ports:
– 5231:5231
volumes:
– slash:/var/opt/slash
restart: unless-stopped

volumes:
slash:

docker-compose.yml

Start Slash using the Docker Compose command:docker compose up -d

This command will pull the required Docker images and start the Slash container in the background.

After running this command, your Slash container will be accessible at http://your-server-ip:5231

Slash is ready & allows you to create, manage, and share shortened URLs without relying on third-party services or compromising your data privacy.

Benefits of Self-Hosting Slash Link Shortener

By self-hosting you gain several advantages:

Data Privacy: Keep your data and links secure within your infrastructure, ensuring complete control over your information.Customization: Tailor Slash to your specific needs, such as branding, integrations, or additional features.Cost-Effective: Eliminate recurring subscription fees associated with third-party link-shortening services.Scalability: Scale your Slash instance according to your requirements, ensuring optimal performance as your link management needs to grow.

Slash offers a seamless solution for managing and sharing links, empowering individuals and teams to streamline their digital workflows.