Sudarsan's Blog

About Me

Wed, 02 Oct 2024 00:00:00 +0000

Computer programmer who enjoys Go, Rust, Systems and Network Programming and Distributed Systems problems. I’m currently obsessed with eBPF, Wireguard and Wireshark.

Previously I helped build Cloudflare Tunnels and Cloudflare Warp-Connector (on the Data on-ramping side).

Getting Cloudflare Tunnels to connect to the Cloudflare Edge with QUIC

Wed, 20 Oct 2021 00:00:00 +0000

This was a project I personally drove and delivered on whilst working on Cloudflare. During that time, I ran into a really interesting problem getting it to work on Cloudflare’s complex distributed edge. I try and walk through that debugging story here with some gentle introductions by usage to tcpdump and strace.

The full blog post written by me is here

Cloning boxed traits in Rust

Sat, 06 Jun 2020 00:00:00 +0000

I enjoy coding with Rust’s composition system. It’s nice to also have generics. It makes adding and incorporating changes rather nice. I recently ran into a problem when I wanted to copy a group of boxed traits.

Lets start from the beginning. Imagine we have an http client that makes requests give a URL. Like so…

use reqwest::{Client, Method, RequestBuilder};

struct Cl {
    client: Client,
}

impl Cl {
    fn new() -> Self {
        Cl {
            client: Client::new(),
        }
    }

    async fn make_request(&self, url: &str) -> Result<String, Error> {
        let req = self.client.request(Method::GET, url);
        let res = req.send().await?;
        let res_str = res.text().await?;
        Ok(res_str)
    }
}

type Error = Box<dyn std::error::Error + Send + Sync>;

#[tokio::main]
async fn main() -> Result<(), Error> {
    let c = Cl::new();
    let resp = c.make_request("http://google.com").await?;
    println!("{}", resp);
    Ok(())
}

Now, this just works. What if we used an endpoint that needed authentication instead of google.com? Assuming this service used basic authentication, we’d do something like this.

use reqwest::{Client, Method, RequestBuilder};

struct Cl {
    client: Client,
}

impl Cl {
    fn new() -> Self {
        Cl {
            client: Client::new(),
        }
    }

    async fn make_request(
        &self,
        url: &str,
        user_name: &str,
        password: &str,
    ) -> Result<String, Error> {
        let req = self.client.request(Method::GET, url);
        let res = req
            .basic_auth(user_name.to_string(), Some(password.to_string()))
            .send()
            .await?;
        let res_str = res.text().await?;
        Ok(res_str)
    }
}

type Error = Box<dyn std::error::Error + Send + Sync>;

#[tokio::main]
async fn main() -> Result<(), Error> {
    let c = Cl::new();
    let resp = c
        .make_request("http://site-with-basic-auth.com", "username", "password")
        .await?;
    println!("{}", resp);
    Ok(())
}

But what if we wanted to call an endpoint that authenticates with a bearer token?

use reqwest::{Client, Method, RequestBuilder};

struct Cl {
    client: Client,
}

impl Cl {
    fn new() -> Self {
        Cl {
            client: Client::new(),
        }
    }

    async fn make_request(&self, url: &str, token_value: &str) -> Result<String, Error> {
        let req = self.client.request(Method::GET, url);
        let res = req
            .header(reqwest::header::AUTHORIZATION, token_value.clone())
            .send()
            .await?;
        let res_str = res.text().await?;
        Ok(res_str)
    }
}

type Error = Box<dyn std::error::Error + Send + Sync>;

#[tokio::main]
async fn main() -> Result<(), Error> {
    let c = Cl::new();
    let resp = c
        .make_request("http://site-that-needs-bearer-token", "Bearer token")
        .await?;
    println!("{}", resp);
    Ok(())
}

Now, we don’t want make_request to keep changing or override it. Enter traits. We could simply replace the myriad input variables with a trait that looks like this.

trait Authorizer {
    fn authorize_request(&self, req: RequestBuilder) -> RequestBuilder;
}

and now we re-write make_request to look like this

use reqwest::{Client, Method, RequestBuilder};

trait Authorizer {
    fn authorize_request(&self, req: RequestBuilder) -> RequestBuilder;
}

struct Cl {
    client: Client,
    auth: Box<dyn Authorizer>,
}

impl Cl {
    fn new(auth: Box<dyn Authorizer>) -> Self {
        Cl {
            client: Client::new(),
            auth: auth,
        }
    }

    async fn make_request(&self, url: &str) -> Result<String, Error> {
        let req = self.client.request(Method::GET, url);
        let req = self.auth.authorize_request(req);
        let res = req.send().await?;
        let res_str = res.text().await?;
        Ok(res_str)
    }
}

Note: I could have simply have included the trait as a field to make_request like so

    async fn make_request(&self, url: &str, &dyn Authorizer) -> Result<String, Error> {

This makes authorizer be dynamic for different urls. I changed this out a bit to suit my demonstration.

However, imagine I had a use case that required clients made with base endpoints for different APIs. This lets the caller implement Authorizer. Nothing new so far. Now the caller can implement variations of Basic and Bearer Auth.

use reqwest::{Client, Method, RequestBuilder};

trait Authorizer {
    fn authorize_request(&self, req: RequestBuilder) -> RequestBuilder;
}

struct Cl {
    client: Client,
    auth: Box<dyn Authorizer>,
}

impl Cl {
    fn new(auth: Box<dyn Authorizer>) -> Self {
        Cl {
            client: Client::new(),
            auth: auth,
        }
    }

    async fn make_request(&self, url: &str) -> Result<String, Error> {
        let req = self.client.request(Method::GET, url);
        let req = self.auth.authorize_request(req);
        let res = req.send().await?;
        let res_str = res.text().await?;
        Ok(res_str)
    }
}

type Error = Box<dyn std::error::Error + Send + Sync>;

#[tokio::main]
async fn main() -> Result<(), Error> {
    let bearer_client = Cl::new(Box::new(BearerTokenAuthorizer::new("key", "value")));
    let resp = bearer_client
        .make_request("http://site-that-needs-bearer-token")
        .await?;
    println!("{}", resp);

    let basic_auth_client = Cl::new(Box::new(BasicAuth::new("user", "password")));
    let resp = basic_auth_client
        .make_request("http://site-that-needs-basic-auth")
        .await?;
    println!("{}", resp);
    Ok(())
}

pub struct BasicAuth {
    user: String,
    api_token: String,
}

impl Authorizer for BasicAuth {
    fn authorize_request(&self, req: RequestBuilder) -> RequestBuilder {
        req.basic_auth(self.user.to_owned(), Some(self.api_token.to_owned()))
    }
}

impl BasicAuth {
    fn new(user: &str, value: &str) -> Self {
        BasicAuth {
            user: user.to_string(),
            api_token: value.to_string(),
        }
    }
}

pub struct BearerTokenAuthorizer {
    token_key: String,
    token_value: String,
}

impl Authorizer for BearerTokenAuthorizer {
    fn authorize_request(&self, req: RequestBuilder) -> RequestBuilder {
        req.header(reqwest::header::AUTHORIZATION, self.token_value.clone())
    }
}

impl BearerTokenAuthorizer {
    fn new(key: &str, value: &str) -> Self {
        let value = format!("Bearer {}", value.to_string());
        BearerTokenAuthorizer {
            token_key: key.to_string(),
            token_value: value,
        }
    }
}

So far, so good right?

So for my usecase… I wanted to clone the Client around. This meant deriving a trait. As simple as

#[derive(Clone)]
struct Cl {
    client: Client,
    auth: Box<dyn Authorizer>,
}

Lets try to build it.

 copy-box-traits % cargo build
   Compiling copy-box-traits v0.1.0 (/Users/sudarsan/dev/rust/copy-box-traits)
error[E0277]: the trait bound `dyn Authorizer: std::clone::Clone` is not satisfied
  --> src/main.rs:10:5
   |
10 |     auth: Box<dyn Authorizer>,
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::clone::Clone` is not implemented for `dyn Authorizer`
   |

OOPS.

So now I had to implement a custom clone for my Boxed trait.

This didn’t seem as straightforward as I thought it would be. I started with…

impl Clone for Box<dyn Authorizer> {
    fn clone(&self) -> Self {
        Box::new(*self.clone())
    }
}

Now… Rust would complain saying…

the size for values of type `dyn Authorizer` cannot be known at compilation time

I clearly can’t dereference the box and put it back in after cloning it.

So the right way to do it, is to implement a generic authorizer cloner trait CloneAuthorizer for all types . Now all we have to do is add this as a built in trait to the trait. This is because only built in traits can be used as closure or object bounds.

trait Authorizer: CloneAuthorizer {
    fn authorize_request(&self, req: RequestBuilder) -> RequestBuilder;
}

impl Clone for Box<dyn Authorizer> {
    fn clone(&self) -> Self {
        self.clone_authorizer()
    }
}

trait CloneAuthorizer {
    fn clone_authorizer(&self) -> Box<dyn Authorizer>;
}

impl<T> CloneAuthorizer for T
where
    T: Authorizer + Clone + 'static,
{
    fn clone_authorizer(&self) -> Box<dyn Authorizer> {
        Box::new(self.clone())
    }
}

This makes the Box<dyn Authorizer> or any struct with this trait clone-able.

The full code can be found at https://github.com/sudarshan-reddy/copy-box-traits

Resume

Sat, 06 Jun 2020 00:00:00 +0000

A little code generation in Go in 2020.

Sat, 14 Mar 2020 00:00:00 +0000

Perl gives me headaches. But I’m fond of one of Larry Wall’s three virtues: Laziness. This is a story of how sometimes laziness lets you do fun things.

Distributed systems and microservices are as ubiquitous as the coronavirus in 2020. This results in a requirement for a lot of integration. And a nice way to maintain the glue code is to use SDKs rather than have multiple services write the transport glue themselves. Like so…

It was all nice and fun until we had to start writing an SDK for each new microservice.

And each new endpoint.

And each new parameter addition.

Generating these parts would make new additions trivial, keep the SDK codebases homogenous and eliminate the occurence of manual errors.

So the SDK has two components. The component that is manually built and the parts that can be generated. Here is a layout of the project.

---
 - client
 - api.go
 - methods.go
 - go.mod
 - go.sum

client is the http.Client embedded with some retry logic etc. It has two methods that are important. NewRequest which creates a wrapped http request and Send() which is sort of like client.Do().

api.go is a collection of all the methods attached to an interface that the sdk exposes.

methods.go is what we want to generate. It will be a collection of implementations of helpers that call the api and parse and respond readable Go structures. Like so:

https://awesomewebsite/v1/getdetails

func (c *client) GetDetails() (*Details, error) {
	op := client.Operation{
		HTTPMethod: http.MethodGet,
		HTTPPath:   endpoint + "/v1/getdetails",
	}

	var response *Details
	req, err := r.cl.NewRequest(op, &response, nil)
	if err != nil {
		return response, nil
	}
	if err := req.Send(); err != nil {
		return response, nil
	}

	return response, nil
}

If you notice, most of the code is tedious housekeeping that gets repeated. So getrandomstuff becomes

https://awesomewebsite/v1/getrandomstuff

func (c *client) GetRandomStuff() (*RandomStuff, error) {
	op := client.Operation{
		HTTPMethod: http.MethodGet,
		HTTPPath:   endpoint + "/v1/getrandomstuff",
	}

	var response *RandomStuff
	req, err := r.cl.NewRequest(op, &response, nil)
	if err != nil {
		return response, nil
	}
	if err := req.Send(); err != nil {
		return response, nil
	}

	return response, nil
}

Step in code generation. Now what I wanted to do was to use a specification that the user is able to add a configuration that the generator captures and translates into something like the snippets about. Lets define a set of rules to outline this.

type Generator interface {
	Generate(Reader, Writer) error
}

// Reader is a general wrapper on io.Reader to read an input to obtain
// the apis.
type Reader interface {
	ReadAPI(io.Reader) ([]API, error)
}

// Writer is a general wrapper on io.Writer to write the processed file.
type Writer interface {
	WriteAPI(io.Writer, []API) error
}

So, we need something to Read from and somewhere to Write.

Reader

Multiple options presented themselves.

It could be an API specification: But everyone hates swagger so much. It seemed to be punishment for the developers to write the yaml with all of swagger’s quirks.

Writing Swagger yamls are painful and so there is no example.

It could be a JSON/XML based configuration system: Still needed the developer to work on a different “language/syntax” compared to what the rest of the codebase was.

{
    "methods": [
        {
            "name": "GetDetails",
            "endpoint": "v1/getdetails",
            "verb": "Get",
            "inputParams": [],
            "outputParams": [
                {"Details": {}}
            ]
        },
        {
            "name": "GetRandomStuff",
            "endpoint": "v1/getrandomstuff",
            "verb": "Get",
            "inputParams": [],
            "outputParams": [
                {"RandomStuff": {}}
            ]
        }
    ]
}

It could be an interface also written in Go: The user will have to add a method to an interface and define the structures. The downside of this is the http endpoint and verb have to be annoted.

type Client interface{
    // Get v1/getdetails
    GetDetails() (*Details, error) 

    // Get v1/getrandomstuff
    GetRandomStuff() (*RandomStuff, error) 
}

type Details struct{}

type RandomStuff struct{}

Out of these three options, the one I preferred was adding to the interface. It seemed intuitive and granted control to a Go developer. It also presented the biggest challenge of the three. Reading an AST. We could obviously just read the file like a plaintext but that seems very hacky and tightly coupled to a pre-conceived notion of how we expect the signatures to be written. Granted, we still expect annotations to be in a certain way but its much more malleable.

AST - Quick primer

ASTs are very common in compiled languages. An AST is an intermediate form of representation of code that the compiler can read and optimize. If you think about it, what we are doing is what a compiler would do. The only difference here being us translating go code to more go code. This brings us to…

Compilers - Very very quick primer

Most compilers break down into three primary stages: Parsing, Transformation and Code Generation.

Parsing: Parsing is generally broken down into Lexical analysis and synctactic analysis. A lexer generally is responsible for this tokenization process. Tokens are tiny little objects that describe an isolated piece of the syntax. They could be numbers, operators or symbols amongst other things. Syntactic analysis is what we are interested in. It takes the tokens and builds an intermediate representation called the Abstract Syntax Tree. Luckily for us, go allows us to use their tools: go/ast, go/parser, go/token.

I’m not the biggest fan of the internal package in go but code generators seem to be the perfect candidate to put in here.

I created two packages internal/generator/ and internal/cmd to add the generator code.

The parsing part of the AST was already done by Go. All I had to do was to call the already existing functions like so…

    fs := token.NewFileSet()
	p, err := parser.ParseFile(fs, "", r, parser.ParseComments)
	if err != nil {
		return nil, err
    }

Transformation: Transformation is manipulating the AST to make changes to it to enable the final process of…

I wanted to transform the AST into a data model that would be easy for the code generation to happen. I laid out a structure like this because I was dealing with APIs and their methods.

// API is a set of indicatives that we will loop through
// to generate code.
type API struct {
	InterfaceName string
	Methods       []Method
}

// Method represents a set of metadata levels details that belong
// to an api method.
type Method struct {
	Endpoint     string
	MethodName   string
	Path         string
	Verb         string
	Body         string
	InputParams  []Parameter
	OutputParams []Parameter
}

Now that I had the AST parsed thanks to ParseFile, I had to loop through it to obtain:

All the interface declarations.
All the methods attached to the interface declarations.
The input and output parameters of these methods.
The code annotation talking about the http verb and endpoint.

Decls is all the declarations. We loop through that and assert for interfaces.

...
	// loop through all the top level declarations of the AST.
	for _, decl := range p.Decls {
		// We don't care about functional declarations, so lets pick
		// only the generic ones.
		decl, ok := decl.(*ast.GenDecl)
		// We don't really need to process token.TYPE.
		if !ok || decl.Tok != token.TYPE {
			continue
		}

		var api API
		for _, spec := range decl.Specs {
			//We only want to read through type declarations.
			typeSpec, ok := spec.(*ast.TypeSpec)
			if !ok {
				continue
			}

			api.InterfaceName = typeSpec.Name.String()

			// Again, we are only interested in Interface definitions.
			interfaceType, ok := typeSpec.Type.(*ast.InterfaceType)
			if !ok {
				continue
			}
...

Now that we have the interfaces we extract the other stuff we mentioned above from 2-4.

    func getMethods(fields []*ast.Field) []Method {
    	var methods = make([]Method, 0)
    	for _, method := range fields {
    		if len(method.Names) == 0 {
    			continue
    		}
    
    		var ignore bool
    		var matches [][]string
    		for _, d := range method.Doc.List {
    			comment := extractComment(d.Text)
    			if comment == "Ignore" {
    				ignore = true
    			} else {
    				matches = verbEndpointRegex.FindAllStringSubmatch(comment, -1)
    			}
    		}
    		if ignore {
    			continue
    		}
    
    		fn, ok := method.Type.(*ast.FuncType)
    		if !ok {
    			continue
    		}
    
    		var methodData Method
    		methodData.Verb = matches[0][0]
    		methodData.Path = matches[1][0]
    		methodData.MethodName = method.Names[0].String()
    		methodData.InputParams = getFields(fn.Params.List)
    		methodData.OutputParams = getFields(fn.Results.List)
    
    		methods = append(methods, methodData)
    	}
    	return methods
    }
    
    func extractComment(comment string) string {
    	slashRemoved := strings.Trim(comment, "//")
    	return strings.Trim(slashRemoved, " ")
    }
    
    func getFields(fields []*ast.Field) []Parameter {
    	params := make([]Parameter, 0)
    	for _, field := range fields {
    		var name string
    		if len(field.Names) > 0 {
    			name = field.Names[0].Name
    		}
    		switch t := field.Type.(type) {
    		case *ast.SelectorExpr:
    			p := Parameter{
    				Name: name,
    				Type: selectorToString(t),
    			}
    			params = append(params, p)
    		case *ast.StarExpr:
    			p := Parameter{
    				Name: name,
    				Type: starToString(t),
    			}
    			params = append(params, p)
    		case *ast.Ident:
    			p := Parameter{
    				Name: name,
    				Type: t.Name,
    			}
    			params = append(params, p)
    		}
    
    	}
    	return params
    }
    
    func selectorToString(t *ast.SelectorExpr) string {
    	var param bytes.Buffer
    	if ident, ok := t.X.(*ast.Ident); ok {
    		param.WriteString(ident.Name)
    	}
    	param.WriteString(".")
    	param.WriteString(t.Sel.Name)
    	return param.String()
    }
    
    func starToString(t *ast.StarExpr) string {
    	var param bytes.Buffer
    	param.WriteString("*")
    	if ident, ok := t.X.(*ast.Ident); ok {
    		param.WriteString(ident.Name)
    	}
    	return param.String()
    }

Code Generation: Is basically generating runnable, optimized code in the format the compiler wants and for that we have the …

Writer

The writer reads the transformation we applied (The API struct) and uses go’s interesting template library to inject the values.

The template library to achieve our example looks like so

const fnTmpl = `
		{{$i := separator ", "}}
		{{$o := separator ", "}}
		{{$length := len .OutputParams}}
		func (c *client) {{.MethodName}} ({{range .InputParams}}{{call $i}}{{.Name}} {{.Type}}{{end}}) ( 
		{{range .OutputParams}} {{call $o}} {{.Name}} {{.Type}} {{end}}){
			op := client.Operation{
				HTTPMethod: http.Method{{.Verb}},
				HTTPPath:   endpoint + "/{{.Path}}",
			}

			{{.Body}}

			var response {{(index .OutputParams 0).Type}}
			req, err := c.cl.NewRequest(op, &response, nil)
			if err != nil {
				{{if eq $length 1}} return nil {{ else }} return response, nil {{ end }} 
			}
			if err := req.Send(); err != nil {
				{{if eq $length 1}} return nil {{ else }} return response, nil {{ end }} 
			}

			{{if eq $length 1}} return nil {{ else }} return response, nil {{ end }} 
		}
`

func separator(s string) func() string {
	i := -1
	return func() string {
		i++
		if i == 0 {
			return ""
		}
		return s
	}
}

The injection looks like this.

for _, api := range apis {
	for _, method := range api.Methods {
		t := template.Must(template.New("func").
			Funcs(template.FuncMap{"separator": separator}).
			Parse(fnTmpl))
		t.Execute(&buf, method)
	}
}

And there you have it. The generation parts. To get this to work with go generate all you have to do is add this to the root file.

//go:generate go run internal/cmd/genmethods/generator.go

How I got a 100 Dollar toy drone to follow me around.

Mon, 10 Feb 2020 00:00:00 +0000

It’s that time of the year again when I decide to add to my blog. It was my birthday last week and I decided to buy myself a small DJI drone to hack with. I wanted to get something lightweight, cheap and easily replaced. The DJI Tello so far has turned out to be delightfully sufficient for my criteria.

The video is here.

The experiment I set out to do is to have the Tello latch on to a face and follow it. To do this, I needed to be able to:

- Control the drone's movements.
- Be able to get drone flight data/visual. 
- Have the drone recognize a face. 
- Tell the drone to follow the face.

Controlling the drone:

So, with no information about the drone, I naively assumed I’d have to push some code/instructions into the drone like an arduino or a Lego mindstorms unit. However, the android app that came with the tello required the phone to connect to the drone on wifi.

This got me thinking. There had to be a server of sorts with a protocol that the android app was using to communicate to the drone. All I had to do was to write a a client that spoke that protocol and I didn’t have to burn any software into the drone.

I found this excellent article that a lot of other hackers had ~~died~~ worked super hard to put together. So, all I needed was to write a client that could send these UDP packets.

To my most pleasant surprise, I found that gobot had already done the excellent work of implementing this protocol as an SDK here. Their documentation listed examples of starting and controlling the drone and the much cooler one of displaying the drone output on mplayer.

The code below (taken from gobot) would have your tello take off and land.

package main

import (
	"fmt"
	"time"

	"gobot.io/x/gobot"
	"gobot.io/x/gobot/platforms/dji/tello"
)

func main() {
	drone := tello.NewDriver("8888")

	work := func() {
		drone.TakeOff()

		gobot.After(5*time.Second, func() {
			drone.Land()
		})
	}

	robot := gobot.NewRobot("tello",
		[]gobot.Connection{},
		[]gobot.Device{drone},
		work,
	)

	robot.Start()
}

Getting the flight visual:

My second problem was also easily solved. Gobot sufficiently lets you stream the drone camera UDP output to mplayer (or as we will in the future, ffmpeg pipes).

For my experiment, I used the ffmpeg encoder to a stdoutpipe.

	ffmpeg := exec.Command("ffmpeg", "-hwaccel", "auto", "-hwaccel_device", "opencl", "-i", "pipe:0",
		"-pix_fmt", "bgr24", "-s", strconv.Itoa(frameX)+"x"+strconv.Itoa(frameY), "-f", "rawvideo", "pipe:1")
	ffmpegIn, err := ffmpeg.StdinPipe()
	if err != nil {
		return nil, err
	}
	ffmpegOut, err := ffmpeg.StdoutPipe()
	if err != nil {
		return nil, err
	}

I then read this repeatedly

for {
		outData := make([]byte, frameSize)
		if _, err := io.ReadFull(f.ffmpegOut, outData); err != nil {
			return err
	}
...

Have the drone recognize a face:

I did a little bit of caffe work to train a CNN to recognize faces. The GoCV bindings to opencv comes with an xml cascade file that does this sufficiently well but I liked my model better. Plugging it into GoCV is as simple as

	net := gocv.ReadNetFromCaffe(f.protoPath, f.modelPath)

The neural network can then run inferences here:

		detBlob := net.Forward("detection_out")

...
        confidence := detections.GetFloatAt(r, 2)
		if confidence < 0.5 {
			continue
		}

Tell the drone to follow the face:

This was the most fun part. My logic was simple. I created bounds for the drone camera. My idea was to have the drone turn in the direction of the human face bounding box every time it left it’s line of sight. I also added in a distance vector to have the drone maintain a safe distance from the human face.

if f.detectSize {
			f.detectSize = false
			refDistance = dist(left, top, right, bottom)
		}

		distance := dist(left, top, right, bottom)

		if right < W/2 {
			f.drone.CounterClockwise(50)
		} else if left > W/2 {
			f.drone.Clockwise(50)
		} else {
			f.drone.Clockwise(0)
		}

		if top < H/10 {
			f.drone.Up(25)
		} else if bottom > H-H/10 {
			f.drone.Down(25)
		} else {
			f.drone.Up(0)
		}

		if distance < refDistance-distTolerance {
			f.drone.Forward(20)
		} else if distance > refDistance+distTolerance {
			f.drone.Backward(20)
		} else {
			f.drone.Forward(0)
		}

Note:

If you watch the video closely, you’ll notice that my ~~beard is very patchy~~
boundary values are so small that the drone sometimes “loses” my face. This is an improvement I will be working on.

Source code:

All the code is available at https://github.com/sudarshan-reddy/dogbot

Why types are elegant

Mon, 07 Jan 2019 00:00:00 +0000

I’ve been working with Go(for work) and Rust(for fun) lately. I got pulled into the two languages for different reasons (easy concurrency and memory safety respectively) and ended up staying for a totally different reason altogether. Their type systems.

Composition instead of inheritance feels elegant, right and very readable. I’m going to talk about something that happened today when I was very thankful for composition and typing itself.

A colleague and I ran into a problem that quickly devolved into an argument and as it is with professional software engineers, wanton namecalling. For work, I’d written a common request making library. It wasn’t anything special. It would branch off of a http.Client and make requests and responses and parse them to specific interfaced outputs that we used for our internal client sdks.

This library took a logger interface. One that could potentially log debug messages. To keep it simple, we initially implemented it as with a single Printf method like so.

    type Logger interface {
    	Printf(format string, args ...interface{})
    }

The logger was hardly used. The errors were all propagated back from the library and didn’t need to be logged at all. The logger was primary used to run httputil.DumpRequest in a debug like functionality.

If you read the source code for DumpRequest, you’ll notice that it is an allocation heavy operation that consumes the request body and reassigns it. All in all, it isn’t something you want to be logging in production level request making code.

Our solution was simple. We’d make passing the logger interface optional. If the logger interface wasn’t nil, we’d call Print.


func (cl *CustomClient) Send(){
    ...
    httpReq, err := http.NewRequest(http.MethodGet, "https://something.com", populateBody())
    ...
    
    if cl.Logger != nil{
        reqDump, err := httputil.DumpRequest(r.req, r.body)
        ...
        cl.Logger.Printf("%s", string(reqDump))
    }

}

This solved our problem. We could use the expensive logging in a debug setting but essentially disable it in production by simply passing the logger as nil.

However…

Assume we wanted to, in the future, add a token bucket based request rate limiter. This rate limiter would come with the added feature of logging a warning every time it began throttling requests.

The interface above could change to something that looked like this.

type LevelLogger interface {
	Info(args ...interface{})
	Warn(args ...interface{})
	Debug(args ...interface{})
	Infof(format string, args ...interface{})
	Warnf(format string, args ...interface{})
	Debugf(format string, args ...interface{})
}

We could have the client implement multiple levels and even set the loglevel. (this example logger follows github.com/sirupsen/logrus ’s intuitive structure for illustration purposes)

This however puts us in a bit of a bother because our earlier implementation doesn’t account for this. True, the client could stop calling the debug log itself by a simple implementation that could look like this.

type structuredLogger struct{
    level int
}

func (s *structuredLogger) Debug(args ...interface{}){
    if s.level > 1 {
        return
    }
}

The printf code earlier I showed you would now look like this.


func (cl *CustomClient) Send(){
    ...
    httpReq, err := http.NewRequest(http.MethodGet, "https://something.com", populateBody())
    ...

    if cl.Logger != nil{
        reqDump, err := httputil.DumpRequest(r.req, r.body)
        ...

        // The structuredLogger implementation could stop calling this
        cl.Logger.Debugf("%s", string(reqDump))
    }

}

The obvious issue we ran into here was while the debug log wouldn’t get printed, the DumpRequest function would still be called. We debated a bit and screamed at each other angrily in German and Tamil.

And finally it dawned to me, as all things do, when I was in the bathroom. I washed my hands(I promise) and got typing. I’d been working a lot with Rust traits recently and while this was definitely a Go thing as well, the incessant typing of systems practise definitely helped.

The solution was still to use the logger, but make the dumped Request a type that would compute only if called inside log.Debugf taking advantage of Go’s fmt.Stringer.

type requestDump struct {
    req  *http.Request
    body bool
}

func (r *requestDump) String() string {
    if reqDump, err := httputil.DumpRequest(r.req, r.body); err == nil {
        return string(reqDump)
    }
    return ""
}


...

func (r *Request) Send() error{
        if r.Logger != nil {
            r.Logger.Debugf("%s \n", &requestDump{req: r.HTTPRequest, body: true})
        }
}

This works perfectly for our usecase. Not only does the allocation heavy dumpRequest only take place during debug loglevel, it’s isolated to this specific call of debug and only computes if it definitely is going get printed by the client’s implementation fo the Debugf itself.

This is probably a simple thing about Go that most of you already do but made me feel very happy about type systems coming out.

Let me know what you guys think in the comments.

Training a simple gender classifier with Python and Predicting with Go

Wed, 08 Aug 2018 00:00:00 +0000

Ever since Tensorflow released Bindings for Go, I’ve been itching to give it a go. The ease of deployability with Go and microservice friendliness and even simple http performance improvements make it really handly to build a working prediction application with Go.

The immediate and apparent downside for anyone who’s tried to train a model is how unintuitive scoping is with Tensorflow for Go.Python’s a lot easier to train models with for a newcomer because of a lot of things:

    + The excellent Keras Library and API.
    + Most tutorials and lessons are done with Python.
    + Numpy is wonderfully intuitive.
    + Fantastic image processing support.
    + Generators are fun.

However, once the model is trained, making inference/applying feedforward does not need too much Tensorflow scoping wizardry.

Go is an obvious choice for deployment/inference because:

    + Fantastic for writing microservices.
    + Nice for people who prefer composition over inheritance for app building.
    + Benchmarks show Go is much faster than Python. While this does not matter 
      when training because GPU compute doesn't require too much parallelism/concurrency.
    + Go with Chi seems to trump Python with Flask and even Python with Twisted 
      by a considerable margin
    + Easy concurrency primitives add versatility to leverage the model.

If you want to skip the ramble and get your hands dirty, you can use telemus as a Quick Start.

Training a gender classifier in Python

For my proof of concept, I decided to a do a simple real world classifier that would take in a picture and classify if it was male or female. To this end, we will be fine-tuning a ResNet50 model replacing the final layer with a Softmax Classifier.

Data

For the data, I will be using the wonderful CelebA dataset that labels 200,000 odd celebrities with over 40 attributes (Male being one of them).

Now, I plan to source mine more data of my own for the future and therefore split the images into folders by class. If you intend to do this, the rudimentary script below helps.

    import csv
    import shutil
    import os
    
    src_path = '/path/to/celebA/data'
    dst_path = '/path/to/train/directory'
    
    def prepare_celebA():
        reader = csv.DictReader(open('list_attr_celeba.txt'), delimiter=' ')
        for row in reader:
            img_name = row['image']
            src = os.path.join(src_path, img_name)
            if row['Male'] == '1':
                dst = os.path.join(dst_path, 'male', img_name)
                shutil.copyfile(src, dst)
            if row['Male'] == '-1':
                dst = os.path.join(dst_path, 'female', img_name)
                shutil.copyfile(src, dst)
    
    if __name__ == '__main__' :
        prepare_celebA()

The script above can be repurposed slightly for validation data as well.

Pre-requisites to be able to deploy in Go

While I like Keras and use Keras to run my models, the Go application requires a number of things to be met.

    + The model to be transferred has to be a tensorflow graph.
    + The tensorflow graph has to be saved with a tag.
    + The input layer and inference layer have to be named.

Training

With the pre-requisites above in mind, we start a tensorflow session and add it to a Keras backend first thing in our code.

    import tensorflow as tf
    from keras import backend as K

    sess = tf.Session()
    K.set_session(sess)

I use two generators to read my prepared celeb data. The load_celeb_a code is listed below.

The images I use are 224 * 224 and I only have an Nvidia GTX 1080 so I tend to keep the train_batch_size to only as much as my GPU can handle.

    from keras.preprocessing.image import ImageDataGenerator
    
    train_directory = ""
    valid_directory =""
    train_batch_size = "50"
    valid_batch_size = "50"
    
    def load_celeb_data(img_rows=224, img_cols=224):
        train_batches = ImageDataGenerator(
        rotation_range=40,
        width_shift_range=0.2,
        height_shift_range=0.2,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True,
        fill_mode='nearest'
        ).flow_from_directory(train_directory,target_size=(img_rows,img_cols),
                              shuffle=True,
                              batch_size=train_batch_size, classes=('male',
                                                                    'female'))
        valid_batches = ImageDataGenerator().flow_from_directory(valid_directory,target_size=(img_rows,img_cols),
                              shuffle=True,
                              batch_size=valid_batch_size,classes=('male','female'))
    
        return train_batches, valid_batches

Finally, finetune the model.

 model = keras.applications.resnet50.ResNet50()
    classes = train_gen.class_indices
    
    model.layers.pop()

    for layer in model.layers:
        layer.trainable = False
    for layer in model.layers[:30]:
        layer.trainable = True
    last = model.layers[-1].output

    x = Dense(len(classes), activation="softmax", name="inferenceLayer")(last)

    finetuned_model = Model(model.input, x)

    finetuned_model.compile(optimizer=Adam(lr=1e-04), loss='categorical_crossentropy', metrics=['accuracy'])

The most relevant change here for us is this line right here.

    x = Dense(len(classes), activation="softmax", name="inferenceLayer")(last)

I’ve gone ahead and named the inference layer (No points for originality there).

We use the ResNet50 model where the input layer is already named. Its called input_1.

Once the model is fit, make sure to save the graph with some tag.

    builder = tf.saved_model.builder.SavedModelBuilder("forGo")
    builder.add_meta_graph_and_variables(sess, ["tags"])
    builder.save()

This is going to get the model saved to a folder named forGo.

You can find the full code here.

Prediction

Time to move to the prediction part. We will build a simple command line app that takes an argument of an image and predicts its output. Copy the forGo directory to a common location. I like to put it within the Go project directory because its not too large.

The Go program is laughably simple once we’ve got the others set up.

All we have to do is to read the image and convert it to a *Tensor data type.

Here’s the gist.

    imageFile, err := os.Open(imgName)
	if err != nil {
		log.Fatal(err)
	}
	var imgBuffer bytes.Buffer
	io.Copy(&imgBuffer, imageFile)
	img, err := readImage(&imgBuffer, "jpg")
	if err != nil {
		log.Fatal("error reading image: ", err)
	}

	result, err := model.Session.Run(
		map[tf.Output]*tf.Tensor{
			model.Graph.Operation("input_1").Output(0): img,
		},
		[]tf.Output{
			model.Graph.Operation("inferenceLayer/Softmax").Output(0),
		},
		nil,
	)

	if err != nil {
		log.Fatal(err)
	}

	if preds, ok := result[0].Value().([][]float32); ok {
		fmt.Println(preds)
		if preds[0][0] > preds[0][1] {
			fmt.Println("male")
		} else {
			fmt.Println("female")
		}
	}

You can find the full code here

Env variables

Mon, 04 Dec 2017 00:00:00 +0000

Ever wake up on the symmetrical side of the bed? Like at the middle? Yep. That�s when you decide that you are never going to (ever) code in constants in your application. So how do you get them? Configuration files? JSONs? Yamls? Environment variables are more elegant and easier with a docker-compose no?

12 factor apps are what cool kids are doing these days (read three years ago). And it makes sense that you put your configurable variables into the environment and do a getenv. But what happens when you have twelve twelve factor apps with twelve different sets of environment configurations?

If you put these into a script/environmental file like below and source them, then you probably have the right idea.

    export CONSTANT_1 = stuff
    export CONSTANT_2 = morestuff
    export CONSTANT_3 = youguesseditmorestuff

But why not harness the incredible power of zsh and chpwd? The gist is, chpwd is a hook that fires everytime your (yep) pwd changes. Just make sure your constant settings (like above) go into a variable.env file and source it at chpwd. Just put this in your .zshrc.

function chpwd(){
    if [ -r $PWD/<environmentvariablefilename.env> ] ; then
        source $PWD/<environmentvariablefilename.env>
    fi
}

So, if there’s a file to be sourced, we source it. Saved me a couple of seconds. Have fun.