![\"\"](\"https:\/\/thinkingtester.com\/wp-content\/uploads\/2021\/02\/laptop-5906264_640.png\")
<\/figure><\/div>\n\n\n\nReliability engineering<\/strong> focuses on the ability of applications to be as available as possible. It aims to offer a good user experience, even in the following situations: Do you know how your application will behave in those scenarios? If not, it sounds like it’s a good time to test those scenarios! There are two ways to test:<\/p>\n\n\n\n Bring the service down<\/strong> Change your connection strings<\/strong> NOTE that it is a bad idea to test either of the above strategies in Production, at least until your application is VERY resilient. You will want to do this testing in your test environment. <\/p>\n\n\n\n Once you have discovered what happens when your application or its dependencies has an outage, it’s time to make your app as resilient as possible. Here are seven strategies for doing that:<\/p>\n\n\n\n No application can run perfectly 100% of the time; servers are imperfect and our apps are almost always dependent upon outside forces. But it’s important to know exactly what will happen when services are down and to figure out the best ways to respond to those issues before<\/em> they happen. As testers, we can encourage our team to participate in this process. <\/p>\n","protected":false},"excerpt":{"rendered":" Imagine that you are working on a team that is creating a new feature that allows users to submit and watch videos. The application uses a third party- we’ll call it Encodurz- that encodes the videos that are uploaded to your application. You work hard to test the feature; making sure that the UI is […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"hide_page_title":"","footnotes":""},"categories":[],"tags":[],"class_list":["post-522","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/thinkingtester.com\/wp-json\/wp\/v2\/posts\/522","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/thinkingtester.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/thinkingtester.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/thinkingtester.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/thinkingtester.com\/wp-json\/wp\/v2\/comments?post=522"}],"version-history":[{"count":8,"href":"https:\/\/thinkingtester.com\/wp-json\/wp\/v2\/posts\/522\/revisions"}],"predecessor-version":[{"id":531,"href":"https:\/\/thinkingtester.com\/wp-json\/wp\/v2\/posts\/522\/revisions\/531"}],"wp:attachment":[{"href":"https:\/\/thinkingtester.com\/wp-json\/wp\/v2\/media?parent=522"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/thinkingtester.com\/wp-json\/wp\/v2\/categories?post=522"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/thinkingtester.com\/wp-json\/wp\/v2\/tags?post=522"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
* The server goes down
* The database is unavailable
* An API that your application relies on is unavailable
* A third-party provider that your application depends on is unavailable<\/p>\n\n\n\n
You can bring a server down by unplugging it, but chances are your server is not nearby for you to do that. But you can also bring a server, a webservice, or a database down by shutting it down using scripted commands. If you don’t have permission to do that, you can find someone in DevOps at your company who has the correct permissions. <\/p>\n\n\n\n
It’s really easy to simulate an outage of a server, a database, an API or any other service your application depends on simply by changing the way your application connects to it. For example, if your app connects to your company database with a username and password, all you need to do is simply send in a bad password. Or you could change the URI needed for the connection so that it’s incorrect.<\/p>\n\n\n\n
This method puts logic in the code that tries connecting to a resource a few times, and when it is unable to connect, switches over to a different resource. For example, if your application usually points to Server A, and you have a backup server called Server B, when the circuit-breaker is tripped the connection changes over to Server B.<\/li>
Sometimes a third-party app will fail temporarily for an unknown reason. You don’t want your request to the app to fail and never try again. So you can build in some retries; perhaps if the request fails, you wait 30 seconds and try again, if it fails again you wait 60 seconds and try again, and so on. You don’t want to retry indefinitely, but instead set some sort of time limit so if the request still hasn’t succeeded when the limit is met, an error is returned.<\/li>
It’s a great idea to have some kind of caching service that will be able to serve up data if a request fails for some reason. When the request fails, your application just grabs the slightly-stale cached data and returns that instead.<\/li>
If your application detects that there’s a problem writing to a data source, you can configure it to go into a read-only mode so that your users can at least see their data. You should set a message to display when this is the case to explain to users why they can’t update their data at the moment. <\/li>
It is so annoying to get a cryptic error like “Error: T-128556” when using an application. That’s not helpful at all! Instead, provide your users with as much detail as you can about what’s wrong. In the example at the beginning of this post, there could have been a message that read “Sorry, we are having an issue connecting to our video encoding software at the moment. Please try again in a few minutes.”<\/li>
If your application goes down completely or is very degraded, it’s a great idea to have a status page (hosted on a different<\/em> server) that provides a way to communicate to your end users what’s happening. You could include a timestamp with your time zone, a list of the features that are affected, and a message about what’s going wrong. Then you can keep the status page updated at regular intervals until the problem is fixed. <\/li>
If the outage was very brief or only affected a few users, you might not need to make the post-mortem public. But if it was a big outage, it’s a great idea to communicate to customers what happened, why it happened, and how you are going to prevent it in the future. See Slack’s message about their January 4th outage<\/a> for a really good example of this.
<\/li><\/ol>\n\n\n\n