DEV Community: Junior Nascimento

Building My Own JSON Parser in Rust

Junior Nascimento — Wed, 19 Jun 2024 02:08:51 +0000

Introduction

I recently took on a challenge to build my own JSON parser in Rust. This project was a fantastic opportunity to dive deep into parsing techniques, which are crucial for everything from simple data formats to building compilers. You can find the full details of the challenge here and the source code in my GitHub repository.

The Challenge

The challenge was structured to incrementally build a JSON parser, starting with simple JSON objects and progressively handling more complex structures. Here's a step-by-step breakdown of the process.

Step Zero: Thinking about the problem and data structure

Being honest I never really implement anything like a parser, my first idea was to count the curly braces and see if they are even (what passes the frist checks for this code challenge). But when I need to parse key-values I started to strugle. Then I took two steps backs and search about parser and remember that compiles do use tokens and parser in order to work so I started in creating a structre to handle all possible tokens in a json, and rust type system really helped me:

    enum Token {
    CurlyOpen,
    CurlyClose,
SquareOpen,
    SquareClose,
    Colon,
    Comma,
    String(String),
    Number(f64),
    Bool(bool),
    Null,
}

This struct contains all the possible tokens that we will need, I took then from the json.org website, that have a really great visual representation of how parsing a json file works.

Step One: Tokenizing the input

The frist step was to get each character of the file and iterate them to create a array of tokens to be fed in a parser. Most of the tokens are straightfoward like curly braces, braces, colons and commas, but values like string, numbers, boolean and nulls required a little more work, I will not show each one because is mostly iterate throug chars making sure that the value is expected, like in string that started with " and finishes with ", numbers can not have trailing zeros and so on. But I had some troubles that I found during testing that I find interesting and will metion latter.

fn tokenize(input: String) -> Result<Vec<Token>> {
    // some checks and initialization ...
    while let Some(&ch) = chars.peek() {
        match ch {
            '{' => {
                tokens.push(Token::CurlyOpen);
                chars.next();
            }
            '}' => {
                tokens.push(Token::CurlyClose);
                chars.next();
            }
            '[' => {
                tokens.push(Token::SquareOpen);
                chars.next();
            }
            ']' => {
                tokens.push(Token::SquareClose);
                chars.next();
            }
            ':' => {
                tokens.push(Token::Colon);
                chars.next();
            }
            ',' => {
                tokens.push(Token::Comma);
                chars.next();
            }
            '"' => {
                tokens.push(Token::String(tokenize_string(&mut chars)?));
            }
            '0'..='9' | '-' => {
                tokens.push(Token::Number(tokenize_number(&mut chars)?));
            }
            't' | 'f' => {
                tokens.push(Token::Bool(tokenize_bool(&mut chars)));
            }
            'n' => {
                tokenize_null(&mut chars);
                tokens.push(Token::Null);
            }
            _ if ch.is_whitespace() => {
                chars.next();
            }
            _ => return Err(anyhow!("Unexpected character: {}", ch)),
        }
    }

    Ok(tokens)
}

Step Two: Parsing the tokens

For parsing I had to use a struct to hold the values, using the json.org example I got all the possible values and created the following struct:

pub enum JsonValue {
    Object(Vec<(String, JsonValue)>),
    Array(Vec<JsonValue>),
    String(String),
    Number(f64),
    Bool(bool),
    Null,
}

With this struct I created a function that parse the a Token array. When testing I noticed that json.org has tests for depth. I didn't know that json had a maximum depth, this is in place to prevent some attacks, we can learn more in depth about this searching, this is pretty known actually and I'm sure someone can explain way better than me so let's keep going.

The function parse_tokens make sure that is only one root token and we made the check if is a object or an array also, since json only can have arrays or object as root nodes.
Here I already parse the object before checking, this is not ideal but is good enouth for the frist time so I didn't bother.

fn parse_tokens(tokens: Vec<Token>) -> Result<JsonValue> {
    let mut iter = tokens.iter().peekable();
    let value = parse_value(&mut iter, 0)?;

    // Check if there are any remaining tokens after the top-level value
    if iter.peek().is_some() {
        return Err(anyhow!("Extra tokens after top-level value"));
    }

    match value {
        JsonValue::Object(_) | JsonValue::Array(_) => Ok(value),
        _ => Err(anyhow!(
            "A JSON payload should be an object or array, not a string."
        )),
    }
}

The magic happens in the parse_value function:

fn parse_value<'a, I>(tokens: &mut std::iter::Peekable<I>, depth: usize) -> Result<JsonValue>
where
    I: Iterator<Item = &'a Token>,
{
    if depth > MAX_DEPTH {
        return Err(anyhow!("Exceeded maximum nesting depth"));
    }

    match tokens.peek() {
        Some(Token::CurlyOpen) => parse_object(tokens, depth),
        Some(Token::SquareOpen) => parse_array(tokens, depth),
        Some(Token::String(_)) => {
            if let Some(Token::String(s)) = tokens.next() {
                Ok(JsonValue::String(s.clone()))
            } else {
                Err(anyhow!("Expected a string"))
            }
        }
        Some(Token::Number(_)) => {
            if let Some(Token::Number(n)) = tokens.next() {
                Ok(JsonValue::Number(*n))
            } else {
                Err(anyhow!("Expected a number"))
            }
        }
        Some(Token::Bool(_)) => {
            if let Some(Token::Bool(b)) = tokens.next() {
                Ok(JsonValue::Bool(*b))
            } else {
                Err(anyhow!("Expected a boolean"))
            }
        }
        Some(Token::Null) => {
            tokens.next(); // Consume the Null token
            Ok(JsonValue::Null)
        }
        _ => Err(anyhow!("Unexpected token")),
    }
}

this basicly take care of objects, arrays, strings, numbers, booleans and null values, converting the tokens to an actuall value in rust.

Step Three: Parsing Nested Objects and Arrays

The next step was to handle JSON objects containing other objects and arrays. This required implementing recursive parsing functions, keeping in mind the depth:

fn parse_object<'a, I>(tokens: &mut std::iter::Peekable<I>, depth: usize) -> Result<JsonValue>
where
    I: Iterator<Item = &'a Token>,
{
    let mut object = Vec::new();
    tokens.next(); // Consume the '{'
    loop {
        match tokens.peek() {
            Some(Token::CurlyClose) => {
                tokens.next(); // Consume the '}'
                break;
            }
            Some(Token::String(_)) => {
                if let Some(Token::String(key)) = tokens.next() {
                    if let Some(Token::Colon) = tokens.next() {
                        let value = parse_value(tokens, depth + 1)?;
                        object.push((key.clone(), value));
                        match tokens.peek() {
                            Some(Token::Comma) => {
                                tokens.next(); // Consume the ','
                                if let Some(Token::CurlyClose) = tokens.peek() {
                                    return Err(anyhow!("Trailing comma in object"));
                                }
                            }
                            Some(Token::CurlyClose) => {
                                tokens.next(); // Consume the '}'
                                break;
                            }
                            _ => return Err(anyhow!("Expected ',' or '}' after object value")),
                        }
                    } else {
                        return Err(anyhow!("Expected ':' after key in object"));
                    }
                }
            }
            _ => return Err(anyhow!("Expected string key or '}' in object")),
        }
    }
    Ok(JsonValue::Object(object))
}

fn parse_array<'a, I>(tokens: &mut std::iter::Peekable<I>, depth: usize) -> Result<JsonValue>
where
    I: Iterator<Item = &'a Token>,
{
    let mut array = Vec::new();
    tokens.next(); // Consume the '['
    loop {
        match tokens.peek() {
            Some(Token::SquareClose) => {
                tokens.next(); // Consume the ']'
                break;
            }
            Some(_) => {
                let value = parse_value(tokens, depth + 1)?;
                array.push(value);
                match tokens.peek() {
                    Some(Token::Comma) => {
                        tokens.next(); // Consume the ','
                        if let Some(Token::SquareClose) = tokens.peek() {
                            return Err(anyhow!("Trailing comma in array"));
                        }
                    }
                    Some(Token::SquareClose) => {
                        tokens.next(); // Consume the ']'
                        break;
                    }
                    _ => return Err(anyhow!("Expected ',' or ']'")),
                };
            }
            _ => return Err(anyhow!("Expected value or ']'")),
        };
    }
    Ok(JsonValue::Array(array))
}

Step Four: Testing and Validation

Finally, I added my own tests to ensure the parser handles both valid and invalid JSON correctly. I also used the JSON test suite from json.org to validate the parser:

#[cfg(test)]
mod tests {
    use crate::parser::{parse_json, JsonValue};

    #[test]
    fn test_invalid_path() {
        let path = String::from("invalid/path");
        let result = parse_json(path);
        assert!(result.is_err());
        if let Err(e) = result {
            assert_eq!(e.to_string(), "Failed to read File");
        }
    }

    #[test]
    fn test_step1_valid() {
        let path = String::from("./tests/step1/valid.json");
        let result = parse_json(path).expect("Error parsing JSON");
        assert_eq!(result, JsonValue::Object(vec![]));
    }

    // Additional tests for other steps...
}

Step Five: Testing and fixing...

When testing I found two problems that needed I little work to work, strings and numbers.

My problem with strings was enconded characters, the frist way that I was checking for strings was to look for the end '"' putting each character in between into the strings, but I found an test that had:

{
    "key": "\""
}

My parser broke, but when making an special case for this I found that I was not at all accouting for escaped chars so I rewrite the function to account for those:

fn tokenize_string(chars: &mut std::iter::Peekable<std::str::Chars>) -> Result<String> {
    let mut result = String::new();
    chars.next(); // Skip opening (") quote

    while let Some(&ch) = chars.peek() {
        match ch {
            '\\' => {
                chars.next(); // Skip the backslash
                if let Some(&escaped_char) = chars.peek() {
                    match escaped_char {
                        '"' => result.push('"'),
                        '\\' => result.push('\\'),
                        '/' => result.push('/'),
                        'b' => result.push('\x08'), // Backspace rust don't like \b in char
                        'f' => result.push('\x0C'), // Form feed rust don't like \f in char
                        'n' => result.push('\n'),
                        'r' => result.push('\r'),
                        't' => result.push('\t'),
                        'u' => {
                            let unicode_sequence = tokenize_unicode_sequence(chars)?;
                            result += &unicode_sequence;
                        }
                        _ => return Err(anyhow!("Invalid escape sequence: \\{}", escaped_char)),
                    }
                    chars.next(); // Skip the escaped character
                } else {
                    return Err(anyhow!("Unexpected end of input after escape character"));
                }
            }
            '"' => {
                chars.next(); // Skip closing (") quote
                break; // Closing quote found
            }
            _ if ch.is_whitespace() && ch != ' ' => {
                return Err(anyhow!(
                    "Invalid unescaped whitespace character in string: {}",
                    ch
                ));
            }
            _ => {
                result.push(ch);
                chars.next();
            }
        }
    }

    Ok(result)
}

for some reason rust didn't like that I put '\b' and '\f' into the push function so I used the ascii values instead. This also acount for whitespaces in the strings, the only whitespace allowed is ' ' space itself, tabs and others must be escaped in json.

I also found that I was not parsing unicode sequeces and they are a little different:

fn tokenize_unicode_sequence(chars: &mut std::iter::Peekable<std::str::Chars>) -> Result<String> {
    let mut result = String::new();

    chars.next(); // Skip 'u'
    let mut unicode_sequence = String::new();
    for _ in 0..4 {
        if let Some(&hex_digit) = chars.peek() {
            if hex_digit.is_ascii_hexdigit() {
                unicode_sequence.push(hex_digit);
                chars.next();
            } else {
                return Err(anyhow!("Invalid Unicode escape sequence"));
            }
        } else {
            return Err(anyhow!(
                "Unexpected end of input in Unicode escape sequence"
            ));
        }
    }

    if let Ok(unicode_char) =
        u16::from_str_radix(&unicode_sequence, 16).map(|u| char::from_u32(u as u32))
    {
        if let Some(c) = unicode_char {
            result.push(c);
        } else {
            return Err(anyhow!("Invalid Unicode character"));
        }
    } else {
        return Err(anyhow!("Invalid Unicode escape sequence"));
    }

    Ok(result)
}

This took care of the errors that I had with strings, now we go for numbers. Frist of all, I totally forgot e AND E can be in numbers, they represent scientific notation (e=10^n) like 2e5 is the same as 2 * 10^5. Also json don't allow leading zeros in numbers so I accounted for that. This got a simple function a lot complex but the end result (not at all perfect) was not too bad:

fn tokenize_number(chars: &mut std::iter::Peekable<std::str::Chars>) -> Result<f64, anyhow::Error> {
    let mut result = String::new();
    let mut is_first_char = true;
    let mut has_dot = false;

    while let Some(&ch) = chars.peek() {
        match ch {
            '0'..='9' => {
                if is_first_char && ch == '0' {
                    chars.next(); // Consume the '0'
                    if let Some(&next_ch) = chars.peek() {
                        match next_ch {
                            '.' => {
                                // Handle 0.x numbers
                                result.push('0');
                            }
                            '0'..='9' => return Err(anyhow!("Invalid number with leading zero")),
                            _ => {
                                result.push('0'); // Just 0
                                break;
                            }
                        }
                    } else {
                        result.push('0'); // Just 0
                        break;
                    }
                } else {
                    result.push(ch);
                    chars.next();
                }
            }
            '.' => {
                if has_dot {
                    return Err(anyhow!("Multiple decimal points in number"));
                }
                result.push(ch);
                chars.next();
                has_dot = true;
            }
            '-' | '+' if is_first_char => {
                result.push(ch);
                chars.next();
            }
            'e' | 'E' => {
                result.push(ch);
                chars.next();
                // After 'e' or 'E', we should expect a digit or a sign
                if let Some(&next_ch) = chars.peek() {
                    if next_ch == '-' || next_ch == '+' || next_ch.is_digit(10) {
                        result.push(next_ch);
                        chars.next();
                    } else {
                        return Err(anyhow!("Invalid character after exponent"));
                    }
                } else {
                    return Err(anyhow!("Exponent without digits"));
                }
            }
            _ => break,
        }
        is_first_char = false;
    }

    if let Some(ch) = result.chars().last() {
        if matches!(ch, 'e' | 'E' | '.' | '-' | '+') {
            return Err(anyhow!("Invalid number"));
        }
    }

    match result.to_lowercase().parse() {
        Ok(number) => Ok(number),
        Err(..) => Err(anyhow!("Number is invalid")),
    }
}

Conclusion

Building a JSON parser from scratch was a challenging but rewarding experience. It provided valuable insights into parsing techniques and Rust programming. You can check out the complete source code and try the parser yourself by visiting my GitHub repository.

Building my Own wc Tool in Rust

Junior Nascimento — Tue, 18 Jun 2024 22:51:13 +0000

Introduction

In this post, I'll walk you through my journey of building a custom version of the Unix wc (word count) tool, which I named ccwc (Coding Challenges Word Count). This project was inspired by a coding challenge designed to teach the Unix Philosophy of creating simple, composable command-line tools. You can find more details and source code in my GitHub repository.

The Challenge

The challenge was to build a wc tool that can:

Count the number of bytes, characters, lines, and words in a file.
Handle command-line options to specify what to count.
Read from standard input if no file is specified.

The Unix Philosophy encourages writing simple programs that do one thing well and can be combined to perform complex tasks. Following this principle, I broke down the challenge into several steps, each adding a new feature to the tool.

Step-by-Step Development

Step Zero: Data Structure

One of my favorite things in Rust has to be the type system, thats why I created a struct to represent the stats we might have to get from the input

struct Stats {
    pub bytes: usize,
    pub chars: usize,
    pub lines: usize,
    pub max_line_lenght: usize,
    pub words: usize,
    pub path: String,
}

Step One: Counting Bytes

The first task was to implement the -c option to count the number of bytes in a file. I did this using a 'constructor'

impl Stats {
    pub fn new(chars: Vec<u8>, path: String) -> Self {
        let mut stats = Stats {
            bytes: chars.len(),
            chars: 0,
            lines: 0,
            max_line_length: 0,
            words: 0,
            path,
        };
        // Additional processing...
        stats
    }
}

This function initializes a Stats struct with the number of bytes based on the length of the input vector.

Step Two: Counting Lines

Next, I added support for the -l option to count the number of lines. This required iterating through the characters and incrementing the line count whenever a newline character (\n) was encountered:

for c in chars {
    if c == '\n' {
        stats.lines += 1;
    }
}

Step Three: Counting Words

The -w option was a bit more complex, as it required detecting word boundaries. I used a simple state machine to keep track of whether the current character is part of a word:

let mut in_word = false;
for c in chars {
    if !c.is_whitespace() {
        in_word = true;
    } else if in_word {
        stats.words += 1;
        in_word = false;
    }
}

Step Four: Counting Characters

The -m option counts the number of characters. This is straightforward unless the locale supports multibyte characters. For simplicity, I treated each byte as a character in this implementation.

Step Five: Default Behavior

I'm handling flags in a very straightforward:

let args = env::args().skip(1); // Skips the frist argument in the command (most likely the binary name i.e: ccwd ....)
    let mut flags: Vec<char> = vec![];
    let mut files_paths: Vec<String> = vec![];
    let avaliable_flags = ['c', 'm', 'l', 'L', 'w']; // All avaliable flags

    for arg in args {
        if arg.starts_with('-') { // if the argument start if a - it must be a flag or multiple flags
            for flag in arg.chars().skip(1) { // skiping the -
                flags.push(flag); 
            }
        } else { // otherwise is treated like a file path
            files_paths.push(arg);
        }
    }

    let invalid_flag = flags.iter().find(|flag| !avaliable_flags.contains(flag));

If no options are provided, the tool should count bytes, lines, and words by default:

if flags.is_empty() {
    flags.push('c');
    flags.push('l');
    flags.push('w');
}

Final Step: Reading from Standard Input

The final feature was to allow the tool to read from standard input if no file is specified:

let number_of_files = files_paths.len();

if number_of_files < 1 {
    let mut buffer = Vec::new();
    match io::stdin().read_to_end(&mut buffer) {
        Ok(_) => {
            stats.push(Stats::new(buffer, "".to_string())); 
        }, 
        Err(err) => { panic!("Error reading from stdin: {err}")}
    }
}

Conclusion

Building ccwc was a rewarding experience that really helped me learning rust. By breaking the project into manageable steps, I was able to incrementally add features and ensure each part worked correctly before moving on.

Feel free to check out the complete source code on my GitHub repository. I hope this post inspires you to tackle similar challenges and build your own command-line tools.

Take care folks!

My First Zig API

Junior Nascimento — Thu, 07 Mar 2024 13:50:56 +0000

There's this event, which is conceived by Zanfranceschi. Essentially, Zan gave us a specification that we followed to implement an API. For this first quarter edition of 2024, we have a kind of "Bank" where some pre-registered clients check their statements and make transactions. If you want to read more details, I invite you to take a look at the repository of the event, where you can find the specifications along with the rest of the participation instructions and related information.

With the API created, the idea is to use a load balancer to manage two instances of the API. This edition focuses on data consistency. We used docker-compose to deploy images for the instances, the load balancer, and also a database. As of the writing of this article (07/03/2024), we still don't have what will be used to obtain the results from each API. However, for now, we're using Gatling to conduct stress tests and some validations, and we're using p75 to get an idea of how the API behaves (if you're not familiar with p75, take a look here).

The first thing I usually do to solve this kind of thing is to create the API in memory for some initial tests, mainly for validation purposes and to check if everything is okay and according to the specification. At this point, I had already decided to write the API in a language I didn't have much experience with, such as Go, Rust, or Zig. But looking at the other submissions, there were already many APIs in Go and Rust, so I tried to diversify and chose Zig. Considering that I already had experience in Rust and Go, it would be interesting to write my first program in Zig.

Zig is a relatively new language, so not all of its content and documentation are as elaborate. I relied extensively on reading and understanding source code. Despite everything, Zig has a Std lib that is quite comprehensive. That's where I started my adventure. I tried to use std.http, and I didn't have a very good experience with it. The API isn't very beginner-friendly, so I tried to look for a framework. The first framework I tried to use was Zap, which is much easier to use than std.http. However, I missed some features, such as the utilization of router parameters, but nothing that prevented me from using it. I wrote a small URL parser with std.Tokenizer (as I mentioned, the std lib is quite comprehensive), and I managed to make the first functional version of the API with in-memory data.

But then the self-inflicted issues started: memory leaks, deadlocks, and inconsistencies because I wasn't using locks or anything in the requests. Even though this had nothing to do with the framework, I ended up looking for another framework and stumbled upon http.zig/httpz, which the developer karlseguin also had a library to use with Postgres, which was the database I already planned to use.

The migration was quite smooth, mainly because there wasn't really a migration; I decided to start from scratch, as you can see in this commit. But that wasn't a bad thing; with the knowledge I had gained using Zap, I was much faster and managed to implement everything back in record time.

The connection with the database was also very smooth; I owe a lot to karlseguin for making everything easier with his libraries. I didn't have any problems integrating the API with the database; the only part where I had some difficulty was using the record type from Postgres, which was returned by a stored procedure I was using in the database. Since the API had a method to receive raw data, I "solved" this problem by writing a small parser that used std.mem.readInt to convert the bytes into integers, as that was all I needed. But since I wasn't sure if this was a good idea, I created an issue in the pg.zig repo to ask, and within hours, the creator of the library made a PR that made using Postgres records trivial (as always, karlseguin saved me).

After solving these problems, I created a docker-compose along with a database and an instance of Nginx and started performing stress tests.

Not surprisingly, I had some more memory leaks, which was to be expected in my first experience; it's also been a few years since I programmed in C++ and C, so I was rusty. I managed to avoid the leak and a deadlock by deallocating a result from a database query that I had forgotten about and also deallocating an array list (similar to the vector in C++).

With these changes, I managed to achieve incredible results, in my opinion.

4ms p75, much better than I anticipated.

I just want to thank Zanfranceschi, Karlseguin, and the Zig community on Discord, where some people answered some of my doubts. Writing this API in Zig was a great experience, and I really loved the language, and I hope to create many cool things with it in the future!

How Advent of Code helped me to like writing code again

Junior Nascimento — Sun, 03 Dec 2023 00:34:50 +0000

First, some context. I've been programming for almost ten years. I first started learning Pascal with a Brazilian IDE called Pascalzim back in 2014. After my first attempts at code, I fell in love with it; I could write code and solve coding problems in an entire day. When I started college in 2017, I joined the competitive programming club (not exactly a club but something similar). I felt like my purpose was to write these codes that can solve problems and see my name in the rankings. Competing and learning from smarter people than me was a joy.

But as the years passed, I began to work as a full-stack developer at a kind of big company. The first months were great; I was getting paid to do what I loved.

But the days passed, and more and more I was not doing side projects, I was not learning anything new, I was not solving any problems.

I was just adjusting things in the product, discussing story points with my team, and working in a tech that was becoming more obsolete every second.

For some months now, I've been unable to work on new projects that I like. Every tiny thing blocked me. Not in a way like, I can't do that, but more like “Eh, I could do that, but it's too much trouble.”

But 01 December I gather some friends and asked: “Have you guys know about the advent of code?”, none of them had, so I explained and they like the idea.

We started on day one of course, and although the first part of the issue as straight forward.
The second part really got us, and I had to go back to my C days writing code to solve competitive programming challenges. That took my joy of writing code back.

I don't know if much of you guys feel like me, that you don't really enjoy writing code or thinking, but AoC is helping me. You cloud try!

How I, an C++ developer, began to learn rust

Junior Nascimento — Tue, 07 Mar 2023 05:30:04 +0000

Hi, I am Júnior Nascimento. I've coding for a little bit than 1 decade now. I started learning Pascal language then migrate to Java, and finally to C and then C++.

Been using C++ for most of my journey I mostly do simple projects and competitive programming, nothing crazy, just for fun.

Then after a couple semesters into college, I learned Javascript because I wanted a job to become financial stable. My life was pretty much programming Javascript, using some frameworks to learn and do some interviews and Python for my academic needs.

Feedforward, a couple of years I landed a job using TypeScript and .NET core.

Sometimes I was doing some side projects using SolidJS and Tailwind. But a few weeks ago I wanted to do something more "low level" I was tired of using Typescript and doing something for the web. I was tired of making APIs with .NET, I wanted some sort of challenge.

I wanted to learn rust for some time now, but never really sit down to do some coding in rust.

Having implemented Neural Networks and Fluid Dynamics in C++ before, I thought to rewrite these projects in rust to become familiar with the language. But then I remember these book I saved almost a year ago about making a Raytracer in a weekend, so I just jumped into it.

The book shows a step-by-step guide on how to implement a raytracer in C++ and how the physics works and how we are going to translate to code.

But maybe you are asking yourself: What is a raytracer?

A raytracer is a computer program or algorithm that creates realistic 2D images of a 3D virtual environment by simulating the behavior of light. It does this by tracing the path of light rays as they interact with objects in the scene and determining how they are reflected, refracted, and absorbed.

To make an image, a raytracer casts a virtual camera into the 3D environment and traces rays of light from the camera through each pixel of the image plane. It then calculates the color and intensity of the light that should be visible at that point based on the properties of the objects and lighting in the scene.

Raytracing can create very realistic images with accurate lighting, shadows, reflections, and refraction, but it can be demanding on computer resources, especially for complex scenes with many objects and light sources. This technique is used in fields such as computer graphics, animation and visual effects for film were highly realistic images are needed.

With this of the way, let's go to the fun part.

I really jumped into this with no background in rust whatsoever, but I have a lot of experience with C++ so understanding the examples in the book was easy.

For the most part the functions are one to one conversion, not surprises here because we are calculating rays and making some linear algebra, is math, so every language should be able to do this with little to no difference.

C++ classes and Rust structs are very similar, so I had absolute no problem in translating and learning the syntax of Rust.

But for this implementation the author of the book choose to make a Hittable abstract class to represent every possible object that light can hit, and in Rust we don't have the concept of classes, so I was search on how to do something similar and learned about Traits.

Traits define shared behavior or functionality across multiple types without requiring a common inheritance hierarchy. They are similar to interfaces and specify a set of methods that types implementing the trait must provide.

For example, this is a Hittable trait:

pub trait Hittable : Sync {
  fn hit(&self, ray: &Ray, t_min: f32, t_max: f32) -> Option<HitRecord>;
}

This trait means that a struct should implement the function hit to being able to be of type Hittable.
The hit function receive a light ray and some limit (to facilitate computation)

Other thing rust implement is lifecycles, we have, but C++ just assumes you know what you are doing and don't check anything.
See this struct for instance:

pub struct HitRecord<'a> {
  pub point : Vector3<f32>,
  pub t: f32,
  pub normal: Vector3<f32>,
  pub material: &'a dyn Material,
}

I confess that when I was implementing this part, I was furious because I cloud not use a Trait as a type without marking as a dynamic and thus needing a lifecycle.

This struct represents where a ray of light hit something. And we need the material where the ray hit to calculate reflection, refraction and the color of the ray in the camera.

As we have more than one material the material field has to be a trait, in order to use a trait as a filed we have to mark the field as dynamic and thus have a lifecycle indicator. The <'a> basically means that the material is live as long what the Hitrecordobject.

I also learn about the Box "type" in rust, it's a smart pointer that allocate memory on the heap and to own and manage the lifetime of the data.

I used to make a list of hittable objects:

pub struct HittableList {
  pub objects: Vec<Box<dyn Hittable>>,
}

This is an array of points for hittable objects, in this project I only implemented spheres but any object that implement the Hittable trait cloud be in this list.

The last thing that I absolutely love about rust was Option and how using tuples are so easy.

In C++ the author (and I also in most of the cases I can remember) uses pointers to change variables in functions, to not copy and return a boolean value to check if the result was a sucess, in Rust I was able to eliminate this using the Option type.

The Option type means that a function can return any type you want, or None

For example:

fn refract(v: &Vector3<f32>, n: &Vector3<f32>, etai_over_etat: f32) -> Option<Vector3<f32>> {
  let uv = v.normalize(); 
  let dt = uv.dot(&n); // Angle between the ray and the normal
  let discriminant = 1.0 - etai_over_etat.powi(2) * (1.0 - dt.powi(2)); // Using law of cosines and the Snell's law
  if discriminant > 0.0 {
      // Can refract
      let refracted = etai_over_etat * (uv - n * dt) - n * discriminant.sqrt();
      Some(refracted)
  } else {
      // Must reflect
      None
  }

Ignore the math and focus in the part that if the light reflect we don't refract, so we return None, then when we use this function we only have to check if we have something to refract:

 if let Some(refracted) = refract(&ray.direction(), &outward_normal, etai_over_etat) {
      let reflect_prob = schlick(cosine, self.ir);
      if rand::thread_rng().gen::<f32>() >= reflect_prob {
        let scattered  = Ray::new(record.p, refracted);
        return Some((scattered, attenuation));
      }
    }

This check if we refract then based on a schlick function scatter the ray.

At the final, I was able to render this image:

I really enjoyed the experience and fell in love with rust, i planing doing some other projects with!

If anyone wants to see the whole project, you can check the project repository in GitHub!

GraphQL API with GraphQL Yoga

Junior Nascimento — Wed, 13 Jul 2022 19:23:27 +0000

Graphql Yoga is a batteries included GraphQL Server, and is very easy to getting started with it!

If you are not familiar with GraphQL, you should checkout the official documentation.

But in a nutshell:
"GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools"

The main goal of GraphQL is to fetch that data you need and only it, using the runtime it also possible to fetch multiple resources in one request so compared to REST API you can achive more results with less code.
Other main point in GraphQL is that you define your data using a powerful type system, that is consistent and shared with the clients, so everything is transparent.

First of all we need to create a new node project and configure it.
So make an directory, in my case is called graphql-yoga-api

npm init -y

(Caution to not make you project name graphql-yoga because it will conflict with some of the packages we will install)

After that i like to use typescript with es-lint so lets install them:

npm i typescript eslint ts-node ts-node-dev cross-env -D

And use the cli to configure:

npx tsc --init
npx eslint --init

Finally we can update package.json and adding some scripts:

"scripts": {
  "dev": "cross-env NODE_ENV=development ts-node-dev --exit-child --respawn src/index.ts",
  "start": "ts-node src/index.ts"
}

No need to worry to much about these commands, but the dev command utiliza cross-env to set the environment variables then call ts-node-dev with respawn on to keep running the project when we are making changes. The start commands simples run typescript directly, if you want you can also compile the project an run with node buld/index.js

Right now its a good time to define the project structure, in this project we will be using this one:

project
├── src
│   └─── index.ts
│
├── ....
└─── tsconfig.json

The project entry point will be src/index.ts

Now we have to install the GraphQL dependencies:

npm i graphql @graphql-tools/schema @graphql-yoga/node

First we will define the GraphQL type in index.ts:

const typeDefs = /* GraphQL */ `
  type Query {
    hello: String!
  }
`;

and defining a function to retrieve the data:

const resolvers = {
  Query: {
    hello: () => 'Hello World!',
  },
};

Right now we have everything we need for GraphQL , so lets combine the types and resolvers and start the server:

const schema = makeExecutableSchema({typeDefs, resolvers});

async function main() {
  const server = createServer({schema});
  server.start();
}

At this point your index.ts must be like this

import {createServer} from '@graphql-yoga/node';
import {makeExecutableSchema} from '@graphql-tools/schema';

const typeDefs = /* GraphQL */ `
  type Query {
    hello: String!
  }
`;

const resolvers = {
  Query: {
    hello: () => 'Hello World!',
  },
};

const schema = makeExecutableSchema({typeDefs, resolvers});

async function main() {
  const server = createServer({schema});
  server.start();
}

main();

When you run the project you are going to see a message like this in the console:

[INFO] 08:55:30 ts-node-dev ver. 2.0.0 (using ts-node ver. 10.8.2, typescript ver. 4.7.4)
💡   🧘 Yoga -   Running GraphQL Server at http://127.0.0.1:4000/graphql

If you go to the link, you will see a page with GraphiQL a tool to test and debug the API. To run what we just created type this in the left container, then press the big play button:

query {
  hello
}

once you make the request you should see something like this in the right container:

{
  "data": {
    "hello": "Hello World!"
  }
}

Is done! See, that's easy, now you know the basics of an GraphQL server and can modify the types and resolvers to see what we can achieve with this minimal setup, GraphQL is an amazing tool and have a excellent ecosystem, you should try it!

I will keeping posting about GraphQL so if you want to see more follow me!