19. REFERENCES¶
I want to thank to contributors of the awesome sources which were referred during development of this project and writing this documentation. You can find these sources below, also in between the lines in code and documentation.
Websites¶
- Wikipedia
- PyTorch Documentation
- Several documents, articles, and code samples: In the code and documentation of this project, you can find several code or document links that were cited.
- Unicode.org website
- Codepoints.net website
- Unicode.org - Emoji Sequence Text Files
- Github Supported Emoji Sequence List JSON
- Emoji dissector
Papers¶
- The Llama 3 Herd of Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Llama: Open and Efficient Foundation Language Models
- Attention Is All You Need
- Grouped Multi-Query Attention
- RoFormer: Enhanced Transformer with Rotary Position Embedding: Paper | Papers with Code | LabML Annotated Implementation
- Pre-RMSNorm (Root Mean Square Layer Normalization)
- Autoregressive Model - ScienceDirect
Github Projects¶
Large Language Models¶
- Meta Llama website
- Original Llama 3.1 Python package repository of Meta
- Original Llama Toolchain Python repository of Meta
- Georgi Gerganov's llama.cpp
- HuggingFace model: meta-llama/Meta-Llama-3.1-8B-Instruct
Inspired Python Projects¶
Inspired Go Projects¶
Wikipedia and Concept Definitions¶
Computing Concepts¶
- Memory Mapping
- BFloat16 (Brain Floating Point)
- The bfloat16 numerical format
- Floating-point arithmetic - Wikipedia
- Inline functions
- Endianness (Wikipedia)
- What is Endianness?
Machine Learning Concepts¶
Unicode Concepts¶
- American Standard Code for Information Interchange
- Code page
- Unicode
- Unicode Consortium - Unicode, Inc.
- Code points
- Rune
- UTF-8, UTF-16, UTF-32
- Zero-width joiner
Package Documentations¶
Go Packages¶
Inspired PyTorch Functions¶
- torch.Tensor.stride
- torch.polar
- torch.view_as_complex
- torch.nn.Softmax
- torch.nn.SiLU
- torch.mean
- torch.rsqrt
Articles¶
Computing Concepts¶
- To Bfloat or not to Bfloat? That is the Question!
- How to Convert a Number from Decimal to IEEE 754 Floating Point Representation
- C Contiguous Arrays
Go Language¶
- Garbage Collector of Go
- Is it possible to inline function, containing loop in Golang?
- How To Use Contexts in Go
- How to wait for all goroutines to finish in Golang
- Using WaitGroup in Golang
- Writing a Stream API in Go.
- Strings, bytes, runes and characters in Go
Machine Learning Concepts¶
- Diving into the Python Pickle format
- Explain Pytorch Tensor.stride and Tensor.storage with code examples
- What's Grouped-Query attention (GQA)? a paper from Google Research
- A Guide on Word Embeddings in NLP
- Word Embeddings in Natural Language Processing(NLP)
- A guide to prompting Llama 2
- Matrix Multiplication
- A Gentle Introduction to Positional Encoding in Transformer Models, Part 1
Videos¶
Large Language Models¶
- Youtube - Andrej Karpathy - Let's build GPT: from scratch, in code, spelled out.
- Youtube - Andrej Karpathy - 1hr Talk - Intro to Large Language Models
- Youtube - Umar Jamil - Llama explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU
- Youtube - DeepLearning Hero - RoPE (Rotary positional embeddings) explained: The positional workhorse of modern LLMs
- Youtube - Serrano.Academy - The Attention Mechanism in Large Language Models
- Youtube - Serrano.Academy - The math behind Attention: Keys, Queries, and Values matrices
- Youtube - Serrano.Academy - What are Transformer Models and how do they work?
- Youtube - StatQuest with Josh Starmer - Decoder-Only Transformers, ChatGPTs specific Transformer, Clearly Explained!!!
Unicode Concepts¶
- Youtube - Intermation - Computer Organization and Design Fundamentals - Ep 020: Unicode Code Points and UTF-8 Encoding
- Youtube - Intermation - Computer Organization and Design Fundamentals - Ep 021: UTF-8 Encoding Examples
- Youtube - Studying With Alex - Unicode, in friendly terms: ASCII, UTF-8, code points, character encodings, and more
Other¶
- Modified "Bolt, bolts, construction icon" by Rafiico Creative Studio Co., Ltd. CC BY