www.imarketing.

courses

CALL NOTES —
Your Entry Gate To The Open Source
AI
Hey everyone!

Thank you for the wonderful call — great to be back with you!

Shareworthy links:
● Rabbit r1 AI companion device
● Open Source LLM for iPhone / Android (in the comments)
● Huggingface
○ Huggingface Spaces
○ Hugginface Autotrain
○ Hugginface Chat
● LMStudio
● Ollama

Introduction to Open-Source AI:

Considering many of you have likely interacted with tools like Stable Diffusion, you're not
new to this field. But have you noticed how it stands up to competitors like Midjourney or
DALL-E? While the latest from Midjourney is indeed impressive, the advancements in Stable
Diffusion are equally noteworthy. It's a shining example of how open source is making waves
in the AI world.

Some images generated with Stable Diffusion:

 www.imarketing.courses

Now, let's talk about the current landscape of open-source AI.

It's no secret that these models, including the ones we frequently use, have their drawbacks.
They may not be as fast, knowledgeable, or accurate as proprietary models like ChatGPT.
And yes, they demand your own hardware, which can be quite resource-intensive. Plus,
they're not always user-friendly, often requiring a bit more technical know-how to get the
most out of them.

But here's the exciting part: these downsides are quickly becoming a thing of the past.
The gap between open-source and proprietary models is closing at an astonishing pace.
This evolution is something to keep an eye on, as it's transforming how we interact with AI.

The real game-changer with open source, and what personally excites me, is its
freedom and flexibility. Open source isn't just free in terms of cost; it's free from restrictions
and censorship. You know those frustrating moments when you're interacting with ChatGPT,
and it gives you a flat-out "I'm sorry, I can't do that" response? With open-source AI, those
moments are far less common. You won't feel like you're being babysat by big tech, and
that's a huge plus.
 www.imarketing.courses

Moreover, open-source models often excel in specific tasks due to their optimization.

This specialization means that for certain applications, they might just outperform their more
generalized counterparts. And let's not forget about privacy. With open-source AI, your data
remains yours. There's a sense of security in knowing that what you input stays private.

So, we're witnessing rapid advancements that are not only bridging the gap with proprietary
models but also offering a level of freedom and specialization that's truly valuable.

Getting into it
The world of open-source AI, particularly large language models (LLMs), is expanding
rapidly, as evidenced by the impressive number of models available on platforms like
Hugging Face.

With over 457,000 models, of which more than 325,000 are LLMs like ChatGPT, the
diversity and accessibility of AI technology are more pronounced than ever.

These LLMs are generally categorized into two groups: proprietary (closed source) and
open-source.

A key distinction between them lies in their size and complexity.

Proprietary models are usually larger, often encompassing trillions of parameters. The exact
scale of these models is not always public knowledge, but it's clear that they are significantly
more extensive than their open-source counterparts.

On the other hand, open-source models, which are more commonly used by individuals on
personal computers, are smaller in scale.

The majority of these models have parameter sizes around 3 billion (3B), 7 billion (7B), or 13
billion (13B). Despite their smaller size, it's crucial to note that they are not necessarily
inferior in performance. In fact, as I mentioned earlier, these open-source models are rapidly
catching up in terms of capabilities and applications.

As we look forward to the future of AI, it's evident that the landscape is rapidly evolving.

The increasing accessibility and improvement of open-source models, alongside the

continued development of proprietary models, are shaping a dynamic and diverse field. This
expansion not only highlights the technological advancements but also emphasizes the
growing importance of AI in various sectors and its potential to revolutionize how we interact
with technology.

Predictions for 2024

Here are some bold predictions for the year 2024 in the realm of open-source AI:
 www.imarketing.courses

1. Surge in Open-Source Models: 2024 is poised to be a landmark year for

open-source AI. Expect a significant influx of new and innovative open-source
models.

2. Rise of Multimodal AI: Multimodal AI, which integrates multiple types of data like
text, images, and sound, will gain substantial traction.

3. Small Models, Big Impact: Smaller models with around 1B parameters will start
outperforming current larger models (e.g., 70B parameters). This leap will be driven
by enhanced data curation, as well as more efficient training and compression
methods.

4. Local Deployment of AI Models: The feasibility of deploying open-source models

on personal devices, including computers and iPhones, will become a reality. These
models will rival the capabilities of current models like ChatGPT without the need for
APIs or online services.

5. Apple's Entry into AI: Apple is expected to join the AI race, potentially introducing
an LLM-powered Siri at WWDC24, according to rumors.

6. Improved Data Quality: A tenfold improvement in data quality is anticipated, which

will significantly enhance AI performance.

7. Open Source Outperforms Private Models: The performance of open-source

models is predicted to surpass that of even the best private models. This shift will be
attributed to the vast talent pool in the open-source community and support from
major companies like Facebook, AWS, and IBM.

8. OpenAI's Market Position: OpenAI might face a decline in enterprise value unless it
addresses issues related to API call limitations, rate limits, and token costs, which
are not sustainable for startups on a large scale.

These predictions paint a picture of a rapidly evolving AI landscape, where open-source

models not only compete with but potentially surpass their proprietary counterparts.

The involvement of big tech companies and the democratization of AI through open-source
platforms could lead to more accessible, efficient, and powerful AI tools for a wide range of
applications.

Huggingface
Hugging Face is essentially the go-to platform for the AI community, often likened to "GitHub
for AI." It's a comprehensive hub where individuals and organizations can build, train, and
deploy machine learning solutions.

Here's a deeper look into what Hugging Face offers:

 www.imarketing.courses

1. AI and ML Hub: Hugging Face serves as a central repository for open-source AI and
ML tools. It's a place for collaboration, sharing, and contributing to a wide array of AI,
ML datasets, and models. This aspect makes it an invaluable resource for anyone
interested in the field.

2. Exploration and Demos: The platform offers a vast array of demos and
opportunities for exploration. It's a treasure trove for those looking to see the practical
applications of various AI models.

3. Programming-Centric: While Hugging Face is an incredible resource, it's important

to note that it leans more towards those with programming skills. It's not necessarily
the easiest platform for beginners or non-programmers.

4. Spaces: One of the coolest features of Hugging Face is 'Spaces.' This allows users
to create web-based demos of their machine learning models, making AI more
accessible and interactive.

5. Extensive Learning Resources: For those who want to dive deep into AI and
machine learning, Hugging Face offers a wealth of tutorials. Their resources are
comprehensive and cater to a range of skill levels.

6. YouTube Channel and Courses: To supplement the online resources, Hugging

Face also maintains an excellent YouTube channel. They provide a full course that is
perfect for those who are serious about deepening their knowledge and skills in AI
and machine learning.

7. Community and Chat Features: The platform fosters a strong sense of community
among AI enthusiasts and professionals. It offers chat features that allow for
networking, discussion, and collaboration, further enriching the learning and
development experience.

In essence, Hugging Face stands as a pivotal platform for those interested in the future of AI
and machine learning, offering tools, resources, and a community for anyone keen on
exploring and contributing to this dynamic field.

LMStudio
LM Studio is an innovative desktop application designed to make working with open-source
AI models, particularly Large Language Models (LLMs), more accessible and manageable
on local devices.

It's available for various platforms including Apple, Windows, and Linux, ensuring a wide
range of users can benefit from its features. LM Studio allows you to download and run any
ggml-compatible model from Hugging Face, offering a simple and powerful model
configuration and inferencing UI.

The application leverages your GPU when possible, enhancing performance and efficiency.
 www.imarketing.courses

One of the key features of LM Studio is its ease of installation and use.

This is the easiest way to download & launch models from Huggingface (contains all of
them).

For example, search for something like TheBloke/dolphin-2.2.1-mistral-7B-GGUF and

download the Q5_K_M version (it’s also an uncensored model — it can tell you anything —
so have fun with it).

OR Download anything that you find interesting in the new & noteworthy on the front page of
LM studio.

LMStudio is designed to be accessible even for those with limited technical background,
making it a great tool for a broader audience interested in exploring AI and machine learning.
The platform provides a user-friendly interface that simplifies the process of exploring and
selecting from a variety of models available on Hugging Face, with detailed information for
each model to aid in informed decision-making.

For efficient model management, LM Studio allows users to handle different model versions,
including quantized versions, tailored to the user's system specifications.

This feature is particularly useful for optimizing the performance of these models on your
device, depending on your system's capabilities in terms of RAM or video RAM.

The software also provides an interactive chat interface, enabling users to directly interact
with the models and adjust various model parameters for a more controlled experience.

Additionally, LM Studio is beneficial for developers looking to integrate large language

models into their applications, as it offers a local server feature compatible with OpenAI's
API, facilitating easy integration.

The tool also ensures efficient storage management, allowing users to add or remove
models as needed to manage storage space effectively.

So, LM Studio stands out as a powerful and user-friendly tool for anyone interested in
exploring and utilizing large language models, suitable for both personal experimentation
and professional application development. It makes access to AI technology easy, and
enables users to leverage the power of large language models on their local machines.

Pinokio
Pinokio is an AI platform that offers a unique approach to managing and automating a wide
range of AI applications, distinct from LM Studio's focus on large language models.
 www.imarketing.courses

It functions like a browser, allowing easy installation, running, and control of various AI tools
with a simple click. This makes it user-friendly and eliminates the need for complex setup
processes.

Pinokio is versatile, supporting diverse applications beyond LLMs, like audio editing, voice
cloning, and creative projects in image and music. It facilitates automation through scripting
and encourages community-driven script sharing, enhancing collaboration and innovation.

Available on multiple operating systems, Pinokio stands out for its accessibility and creative
potential, making it a valuable resource for artists, designers, and creators exploring AI
applications.

Ollama
Ollama is a cutting-edge platform designed for running open-source large language models
(LLMs) locally on your machine. It streamlines the process of using LLMs like Llama 2 and
Code Llama, making it simpler for users to set up and run these models without relying on
cloud-based solutions.

This local approach offers benefits in terms of speed, cost-effectiveness, and privacy, as
data processing happens on your machine, keeping sensitive information secure.

Ollama's setup process is straightforward (for devs: especially with its availability as an
official Docker image. This facilitates easy packaging and distribution of applications in
containers, enhancing its accessibility across different operating systems).

Although primarily available for macOS, it is also compatible with Linux, and support for
Windows is on its way.

Users appreciate Ollama for its ability to run multiple models in parallel and the convenience
of setting up a dedicated machine to run the model locally, making it accessible to everyone
in a local area network.

This feature is particularly useful in a home or office setup, where a single computer or GPU
can serve multiple users.

A powerful use of Ollama is extracting information from a knowledge base, like Obsidian,
and it also includes integrations for applications such as Discord AI bots. The community
around Ollama is active, contributing various integrations and applications, enhancing its
utility and scope.

For model customization, Ollama allows users to create and tailor models to their specific
needs.

1. Initial Installation:
 www.imarketing.courses

○ Ensure you have the appropriate operating system, as Ollama currently

supports macOS and Linux, with Windows support coming soon.

○ (for devs) Install Docker if you haven't already, since Ollama is available as an
official Docker image.

2. Running Ollama:

○ Open your command line interface (CLI) (terminal or cmd).

○ To start Ollama, type ollama serve.

○ To run a specific model like Llama 2, type ollama run llama2 (this will also
install the model)

○ Ollama allows running multiple models in parallel.

3. Exploring Community Integrations:

○ Check out various integrations provided by the community on GitHub. This

includes various applications like Discord AI bots or extracting information
from knowledge bases like Obsidian.

4. (advanced) Setting Up a Dedicated Local Server:

○ You can set up a dedicated computer to run the model, which everyone on
your local network can access. This could be a simple setup with a computer
or a GPU in a closet running Ollama.

5. Creating and Customizing Your Own Model:

○ Use a text editor to create an empty file. For example, create a file named
'vader' (in the folder your CLI is interacting with)

○ Then write in the file.

For instance:

FROM llama2-uncensored
PARAMETER temperature 1
SYSTEM “””
You are Darth Vader from Star Wars. Answer as Darth
Vader only and give guidance about the Empire and
order in the galaxy
“””

6. Running Your Custom Model:

○ In your CLI, type ollama create vader -f ./vader to create the model.

○ Then, run your custom model by typing ollama run vader to interact with this
model.
 www.imarketing.courses

Mixtral 8x7b
This is the new impressive model I mentioned that outperforms GPT 3.5.

The Mixtral 8x7B model is a significant advancement in the field of large language models,
known for its sparse architecture and impressive performance. It utilizes a sparse mixture of
experts (SMoE), which allows for efficient allocation of computational resources by focusing
on the most relevant parameters for a given task. This approach makes the model not only
powerful but also resource-efficient, capable of running on less hardware than expected for
its size.

The core of the Mixtral 8x7B model is a transformer-based architecture, augmented by the
Mixture of Experts (MoE) paradigm to enhance its processing capabilities. It consists of 8
experts, each with 7 billion parameters, contributing to a total of 56 billion parameters. This
design optimizes efficiency by dynamically allocating computational resources where they
are most needed.

Mixtral 8x7B has been benchmarked against other industry standards like Meta Llama 2 and
OpenAI GPT-3.5, demonstrating superior performance in many aspects. It's particularly
adept at tasks across various languages, showcasing its proficiency in global communication
contexts. The model's architecture supports a 32k token context window, which is crucial for
maintaining coherence over longer text passages.

Regarding implementation, deploying Mixtral 8x7B requires careful consideration of

resources and infrastructure. Cloud-based solutions with GPU support are often
recommended for their scalability. The model's integration with open-source tools like
TensorFlow, PyTorch, and Hugging Face Transformers allows for flexibility and
customization. Users can tailor the model for specific languages and domains, making it
suitable for a broad spectrum of AI applications.
 www.imarketing.courses

Running Mixtral 8x7B on an Apple Studio M2 MAX, which is what I have for example, I
experience slower performance due to the model's large size and computational
requirements. The model is optimized for powerful machines, ideally with more than 32GB of
RAM.

LMStudio can recommend a suitable model version based on your computer's specs, such
as the Q5_K_M quantization of Mixtral 8x7B. However, it's important to note that the full
potential of Mixtral 8x7B might be best realized on systems with more robust hardware
capabilities.

You can find it on LMStudio by searching for TheBloke/dolphin-2.5-mixtral-8x7b-GGUF

Or install it on ollama with ollama run dolphin-mixtral:latest

Fine-tuning your own LLM

Fine-tuning your own large language model (LLM) using Hugging Face's AutoTrain involves
several steps:

1. Create a New Space on Hugging Face: Start by setting up a new space in your
Hugging Face account.

2. Choose Docker and AutoTrain: Select the Docker and AutoTrain options for your
project setup.

3. Select a Model: You can choose from various models. For instance,
TheBloke/Llama-2-7B-Chat-GPTQ — or choose what works for you.

4. Consider the Cost: Be aware that fine-tuning your model can be expensive,
potentially around $1,200 for the model previously mentioned.

5. Prepare Your Data: Fine-tuning requires data in CSV format. The data format varies
based on the trainer type you choose (SFT/Generic, Reward, or DPO Trainer). Each
format requires different column arrangements like text, rejected_text, and prompt.

6. Upload and Train: Once your data is prepared, upload it to AutoTrain and start the
training process.

While fine-tuning your own LLM using Hugging Face's AutoTrain is certainly doable, it might
not always be the most practical option, especially considering the cost and rapid
advancements in the field.

Fine-tuning a model can be expensive, and there's a risk that your model could quickly
become outdated as new, more advanced models are released. Instead, it may be more
effective to find a specialized model that is closely aligned with your specific task and
experiment with custom instructions.
 www.imarketing.courses

This approach can provide a balance between customization and cost-effectiveness,

allowing you to leverage the latest developments in AI without the need for extensive
investment in model training.

For detailed guidance and options, Hugging Face's AutoTrain documentation is a valuable
resource.