Ramkumar K R

A silly mistake

Sat, 28 Dec 2019 22:45:00 +0000

I was working on an assignment whose problem statement was like this -

You are writing a program that guides a robot over terrain from a starting location to a destination. The program gets a two-dimensional matrix where the indexes define the x and y coordinates. The elements represent the elevation of the terrain. I was to use different search algorithms such as A*, BFS, and UCS.

I finished my assignment well on time and was also happy with the design. I used the strategy pattern and would choose the relevant algorithm based on the input. I spent days designing the right heuristic for A* search. Though I invested a lot of time and effort into the assignment, I got a meager 22%. It turns out that I read the input incorrectly for reading the elevation (swapped places of X and Y). Here’s the two letter change which would fix my code -

-        landing.setZ(terrainMap.get(landing.getX()).get(landing.getY()));
+        landing.setZ(terrainMap.get(landing.getY()).get(landing.getX()));

Although I was initially frustrated with myself, I realized the consequences of code which has on people. For me, it was almost harmless, since I lost a few marks. However, code can also affect the lives of a lot of people. This assignment taught me more about the effects of code that have on the world than about the search algorithms. I feel that it is important for every person who writes code to understand its implications on the world and proceed with caution.

Another aspect is about testing. Although it is not possible to test every possible scenario for all non-trivial code, it shows that there is a need for improving my testing. In my next assignment, I stuck to Test-driven development (TDD) which took a bit of extra time but ensured that I covered almost all major scenarios.

I end this blog post by a slightly modified quote of Martin golding -

Always code as if the person who ends up using or maintaining your code is a violent psychopath who knows where you live

Optimizing spend on AWS

Tue, 20 Aug 2019 00:23:00 +0000

Optimizing spend on AWS

A large chunk of the internet today is currently hosted on AWS. However, without proper measures, the cost of running infrastructure on AWS can increase dramatically. Taking on the responsibility of DevOps at my company, (which should ideally be called as DevSecFinOps) I was tasked with bringing down our AWS spend. After 2 months, and a region migration, we brought down the cost of our bill down by 50%. This post will talk about the various steps I took in optimizing the spend on AWS.

Where am I spending money?

We must first identify the areas where money is being spent on AWS. AWS provides three tools of various levels of granularity to identify expensive resources.

AWS monthly bill

AWS provides a monthly bill which provides a region and service level visibility. For example, the bill can say that $100 was spent on m5.large instances in ap-south-1 region for the month.

Cost explorer

AWS Cost explorer goes one level deeper by providing a day level visibility. The cost explorer can be very useful to perform preliminary analysis on the cost. For example, the cost explorer can say that an additional m5.large instance runs in the ap-south-1 region every monday.

AWS Cost and usage report

The cost and usage report provides the highest level of visibility with a hourly account of expenses. However, it is also very bulky and requires proper tagging of resources and other tools such as athena or elasticsearch to analyze the data.

Tagging your resources

Tagging resources is one of the most important steps to be done for cost optimization. Providing proper tags can be highly effective in identifying the areas of expenses in your infrastructure. To paraphrase Edna mode, “Done properly, Tagging Resources can be a heroic act…Done properly”

Usually there can be about 3-4 tags common for all resources and other application specific tags. A few common tags which I use are Application, Stack, Team, Owner, Environment etc.,

Indexing the report to elasticsearch

The Cost and usage report usually is huge and cannot be analysed manually. One way to analyse the data is to index the report to elasticsearch. With a Visualization tool such as kibana or grafana, it is possible to build dashboards and visualize your spends. I indexed the report to elasticsearch using logstash.

Another way to analyze the report is to use Amazon Athena to perform relational database like queries on files in S3.

Some opensource repos which do these are

Elasticity

The infra requirements change in my company from time to time. For example, we may require more instances during office hours or during the sale. Instead of running instances to handle the maximum load all the time, we can scale up or down the instances as required. The following document by AWS provides much more detail - https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html#autoscaling-benefits-example

Along with the number of instances, the configuration of each instance is also a contributing factor for both performance and cost. Having the right instance to run the right application requires a profound understanding of the application as well as a scientific way to measure the performance of the application under various conditions.

Design for cost optimization

While designing applications, we must always consider the cost factor and take decisions which can optimize our spend in the long run. For example, if you have multiple services which require load balancing, using a single ALB with host or path based routing is more cost efficient compared to having separate load balancers for each service. In order to achieve this, the applications must be designed such a way that by adding host or path based routing should not cause side-effects.

Services

Here, I go through 2-3 commonly used services and list some common tips on optimizing cost

EC2

Running hours - It is possible to reduce the cost by turning off the instances when not in use. For instance, dev and test environment instances can be turned off on nights, weekends and holidays.
Right Sizing - Having the right instance size can play a crucial role in obtaining better performance and lower our costs.
Spot instances - AWS provides instances at ridiculously high discounts based on availability of extra capacity called “Spot instances”. However, these instances are available based on demand and have a termination window of 2 minutes. It is a great way to run dev and test workloads or even containers in production. Note that applications such as databases may not play well with these instances.
EBS Volumes - AWS provides different types of volumes to cater various use cases. In case you are using io1 volumes, it would be cheaper and performant to switch to a gp2 volume of a bigger size. To illustrate - A io1 volume of size 100 GiB and 1000 IOPS costs about $78 per month in the us-east-1 region. However, a gp2 volume of 400 GiB and 1200 IOPS costs about $37 per month in the same region.
EBS Snapshots - EBS snapshots are incremental and are internally linked with the sections which have not changed (Refer to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSSnapshots.html#how_snapshots_work for more details). Over a period of time, snapshots can be stale since the data in the volume might have changed completely and it may not be useful to have them anymore. In such a case, deleting these snapshots can reduce storage costs. More information is available in here - https://docs.aws.amazon.com/whitepapers/latest/cost-optimization-storage-optimization/optimizing-amazon-ebs-storage.html#delete-stale-amazon-ebs-snapshots
Reserved instances - There are different types of Reserved instances (RIs) available with AWS. Please consider buying an RI if an instance meets the following criteria
- The instance runs more than 75% of the time per month.
- The application run by this instance has a predictable workload and the requirements can be pre-determined.
- There is no further optimization which can be done for the instance.

S3

Storage classes - S3 provides various storage classes to serve different use cases. The lifetime of the object stored in S3 is an important factor for the cost. For example, if an object is being stored for 90 days, the infrequent access storage class may be cheaper than Glacier.

Data transfer

Communication between internal applications - If there are multiple applications talking to each other within a VPC, Data transfer costs can be minimized by not using Elastic IPs or public endpoints. You can have a private route53 hosted zone accessible inside the VPC instead of referring to private IPs.

Checklists

Checklists serve as a way to systematically optimize your resources. For example, a checklist for EC2 instances can be built and verified on each instance. An action can be taken to optimize whenever the criterial in the checklist is not met. These checklists differ for each company. You can find the checklists I used as a sample below -

EC2

1. Can I stop this instance at night and weekends?
	a. Yes - stop the instance at night and weekends. This reduces cost by 50%.
2. Can I use a linux instance instead of windows?
	a. Yes - migrate to using linux. This reduces cost by 23%.
3. Is my maximum CPU utilization < 40% over 30 days?
4. Is my maximum RAM utilization < 40% over 30 days?
	a. If both 3 and 4 are yes, downgrade the instance to 1 level lower (example: m5.xlarge to m5.large)
	b. If only 4 is yes, use a c series instance with same number of cores (since the application is compute intensive)(example: m5.xlarge to c5.xlarge)
	c. If only 3 is yes, use a r series instance with same memory (since the application is memory intensive)(example: m5.xlarge to r5.large)
5. Is my application fault tolerant and does not depend on local file storage (except writing logs. Example - all dockerized applications)?
	a. If yes, then please run spot instances. I would recommend a 50-50 split between on-demand/RI and spot instances
6. Will my instance keep running 24x7 for more than a year?
	a. Then please purchase an RI (Reserved Instance). Decide whether to buy for 1 or 3 years and also to buy No-upfront, partial upfront and full-upfront
7. Am I using the latest generation of EC2 instances?
	a. If no, then please take a downtime to upgrade the instances to the latest generation. Prepare a plan for the migration. This reduces cost by 10-15%.

S3

1. Do we need the S3 bucket and its data?
  a. If no, then please delete it
2. Do we need all the data in Standard Storage?
	a. If no, then consider uploading new objects in a different storage class such as IA or Glacier. Please refer to Log retention policy to decide on the right storage class

RDS

1. Can the RDS instance be shut down at night and weekends?
	a. If yes, then shut down the RDS instance at night and weekends. This will reduce cost by 50%
2. Is the RDS instance Multi-AZ and running non-production environments only?
	a. If yes, then turn off Multi-AZ for non-prod environments
3. Is the maximum CPU utilization < 30% over the last 6 months?
	a. If yes, then try to downgrade the instance by 1 level
4. Is the storage space used < 20% over the last 6 months?
	a. If yes, then try to reduce the storage space allocated for the RDS instance
5. Are there databases which are not being used since last 1 year?
	a. If yes, then discuss if they are still required. If not, then remove it.
6. Are there manual database snapshots which are not required?
	a. Then delete it.

Continuous monitoring

Cost optimization is not a one time exercise. It has to be constantly monitored and the infra has to be reviewed regularly to avoid any leaks. For example, every new instance brought up can be run through the checklist to ensure that we effectively utilize the services provided by AWS.

I hope this post was useful in providing some insights on optimizing spend on AWS.

A Jugaad

Sun, 08 Jan 2017 00:00:00 +0000

I was given the task of writing a few scripts using maxscript to automate some render based tasks.

Maxscript

Maxscript is a scripting language for 3DS Max. It helps you to automate tasks and provide UI for them.

The challenge

One of the scripts I have to write involved in writing a post render script using Autodesk Backburner to save multiple render channels to a path given using user input (choosen using the standard file dialog). The post render script contained saving all render channels using vfbcontrol() However, the method requires the path of the rendered output to be passed as an argument. The problems are

The post render script call for backburner requires the absolute path to the script file.
There can be multiple renders happening at a point in time.

The Jugaad

Write a pre-render script which generates the post render script with the file path (One more level and it will become inception).
Store the post render script in the same directory as the render output

The jugaad was done to meet some pressing deadlines. If you think there is a better approach, I would be really thankful if you can tell me about it.

Running Rspec in bitbucket pipelines

Sat, 03 Sep 2016 00:00:00 +0000

Recently, I integrated bitbucket pipelines into the development workflow of my organization. We use the rails framework and rspec to run our tests.

Bitbucket pipelines

This is a new CI system from bitbucket based on docker. A YAML file which is very similar to the docker compose file is required to run the build. You can know more about bitbucket pipelines from here.

Steps to start using bitbucket pipelines

Get an invite

Since pipelines are still in beta, you need an invite to start using bibtbucket pipelines. You can request for an invite from bitbucket from the pipelines features page.

Understanding the structure of the YAML file

Before diving into the specifics, you may need to understand the structure of the bitbucket-pipelines.yml file. Bitbucket provides an excellent explanation for this. It can be accessed here

Docker image

You need a docker image to start with. I used the offical rails 4.2.4 docker image for this. (note that this build is deprecated and you may have to use the ruby image for newer updates.) Private docker images are also supported. The advantage of using a private docker image is that you can install all your dependencies upfront and reduce your build time considerably. You can get more info about using private docker images in here

Installing dependencies

My organization had a bunch of private dependencies. Since I do not use a private docker image, I had to install them during the build. In order to access these private repositories, I created a new pair of RSA keys and cloned them using SSH.

Things to note

The SSH keys should not have a passphrase. You may have to generate keys without a passphrase. The command I used was ssh-keygen -N '' -t rsa -f ~/.ssh/id_rsa
You may need to add a list of known hosts. Example - echo "domain ssh-rsa <some key>" >> ~/.ssh/known_hosts

The SSH key can be configured as a secret environment variable. You can also encode the key using base64 and decode it as a step in the build. More details can be found in this question.

Sending Notifications

Currently sending notifications is not supported by pipelines. You can instead make a CURL call to a service such as slack which provides webhooks for sending notifications.

curl -X POST -H 'Content-type: application/json' \
--data '{"text":"Build Successful"}' \
 "http://www.example.com"

That’s it!. Once you can access your repositories and install your dependencies, you can just run `bundle` and then run the test command. Since I use rspec here, I just run `bundle exec rspec`

Speeding up Rspec with factory_girl

Sat, 25 Jun 2016 17:48:02 +0000

I use rspec and factory_girl in my organization for writing unit tests. However, it used to be very very slow. I took some steps to speed it up.

Using `build_stubbed` instead of `build` or `create` for creating objects using factories as much as possible

Using build_stubbed builds factories and stubs all associations for the object. There were a lot of specs which could use the build_stubbed method instead of create or build for creation/building of objects from factories. This resulted in a non-requirement of a database connection and neccesity to create associated objects which sped up the tests by about 55-60%. For more details you can refer to the link – https://robots.thoughtbot.com/use-factory-girls-build-stubbed-for-a-faster-test

Using an in-memory database

Database was a considerable bottleneck for the performance of our tests. This is a common scenario which happens either due to a slow disk or due to a large number/complexity (or both) of queries being executed.

To solve this problem, I used the memory_test_fix gem which uses the sqlite3 database adapter and uses your memory as the database. This sped up the tests by about 18-20%.

Github link – https://github.com/mvz/memory_test_fix

Running multiple tests at once

Tests run only in a single process and results in under utilizing the processor (which is usually dual or quad core). Running multiple processes to run tests in parallel resulted in a much shorter duration than running them sequentially.

I used the parallel_tests gem and ran tests with 4 processes. This resulted in reducing the duration by approximately 2.5-3 times. (This should have ideally been 4 times but since I had a lot of applications running the performance obtained is less).

Github Link – https://github.com/grosser/parallel_tests

Results

I had a test suite containing 100 unit tests. build_stubbed was used for only 4 tests (I should have done this for atleast 40 and ran the tests). Making improvements one by one on the same suite reduced the overall execution time from 20 minutes to 6 minutes. Here’s a Before-After table containing execution times (rounded to the minute).

Improvement Done	Before	After
Using `build_stubbed` for 4 out of 100 tests	20 minutes	19 minutes
Using an in-memory database	19 minutes	16 minutes
Using `parallel_tests` gem with 4 processes	16 minutes	6 minutes

Extension development in firefox and chrome

Sat, 09 Apr 2016 11:17:31 +0000

I developed an extension in both chrome and firefox which provides a new tab page resembling a speed dial. I’ll just highlight the advantages and disadvantages of development in both the browsers.

Chrome

Advantages

Chrome has a very good documentation and tutorials having a step by step explanation for a few examples as well.
Since the APIs were based on javascript, no more additional learning was required to start.
Testing or Trying out the extension in chrome on Mac or Linux is pretty simple as well.
1. Go to chrome://extensions and enable developer preview
2. Click on the Load unpacked extension and select the folder having the extension code
3. Your extension is now loaded to the browser
Google Analytics works very well with the extension
Publishing updates to your extension is hassle free and all your users are updated to the latest version within a few minutes
Ability to have the extensions paid and can be a source of income.
An extension written for chrome will work in chromium and is most likely to work in opera , vivaldi and other chromium based browsers as well.

Disadvantages

An amount of $5.00 needs to be paid to google for publishing your first app or extension on the chrome web store.
Trying out the extension in a developer mode in Windows with a stable version of chrome can be very annoying since chrome asks you to disable the extension every time the browser is opened.
Stable versions of chrome on Windows disables all the extensions which are not from the chrome web store. These extensions cannot be enabled again either manually or using an API. You can read more about it in here.
There is a cap of 20 apps/extensions/themes (combined) which can be published to the chrome web store. If you want to publish more, you need to get an approval from google for this.

Firefox

Advantages

No payment required to publish the addon
Very easy to translate a chrome extension to a firefox addon. Only the manifest file format needs to change
Testing, building and trying out the extension in any operating system is possible
Extensive libraries for customization options which can change the entire look and feel of the browser.
No cap on the number of extensions you can release to the add on store
Ability to distribute the extension outside the addon store

Disadvantages

Each review takes about a week to complete
Multiple libraries can be used to implement a single functionality. For example, the new tab page can be overriden either by XUL or by using the new tab override API provided by firefox
Poor or no documentation for the libraries
Accessing external scripts from the extension is not allowed. This would require you to have all the library scripts ( such as jquery) as a part of your add on therby increasing the addon size
Dwindling user base…..The number of users for firefox is going down in the past few months. This forces me to think if the effort required for writing an extension is worthy or not
No Pricing – You can request for users to donate some amount for the development and maintainence of the addon. This has discouraged many companies to create addons for firefox

Footnotes

All the points are classified under advantage/disadvantage are purely in my opinion
In case you want to try out my extension, you can find the links below
Chrome – https://chrome.google.com/webstore/detail/new-tab/dbnbjnjckidjkjdocfflalcgmlhkcfee
Firefox – https://addons.mozilla.org/en-US/firefox/addon/yet-another-new-tab-page/?src=search

Ramkumar K R

A silly mistake

Optimizing spend on AWS

Optimizing spend on AWS

Where am I spending money?

AWS monthly bill

Cost explorer

AWS Cost and usage report

Tagging your resources

Indexing the report to elasticsearch

Elasticity

Design for cost optimization

Services

EC2

S3

Data transfer

Checklists

EC2

S3

RDS

Continuous monitoring

A Jugaad

Maxscript

The challenge

The Jugaad

Running Rspec in bitbucket pipelines

Bitbucket pipelines

Steps to start using bitbucket pipelines

Get an invite

Understanding the structure of the YAML file

Docker image

Installing dependencies

Sending Notifications

That’s it!. Once you can access your repositories and install your dependencies, you can just run bundle and then run the test command. Since I use rspec here, I just run bundle exec rspec

Speeding up Rspec with factory_girl

Using build_stubbed instead of build or create for creating objects using factories as much as possible

Using an in-memory database

Running multiple tests at once

Results

Extension development in firefox and chrome

Chrome

Advantages

Disadvantages

Firefox

Advantages

Disadvantages

Footnotes

That’s it!. Once you can access your repositories and install your dependencies, you can just run `bundle` and then run the test command. Since I use rspec here, I just run `bundle exec rspec`

Using `build_stubbed` instead of `build` or `create` for creating objects using factories as much as possible