Test Engineering Notes

On choosing and changing languages for test automation

2022-10-30T06:00:00+00:00

Photo by Javier Allegue Barros on Unsplash

You are at the beginning of your testing career… You see that in many job descriptions, they ask about test automation skills. Or you heard that by working in automation, you can earn more money (which depends). Which programming language to choose? Which language and tech stack is best of the best? Is there a single choice for it?

It can happen to anybody. You can be frustrated in your current job. You can feel that you are doing the simple things, and your progression is stopped. You also desire more salary or more significant responsibilities. So the call or message notification on Linkedin about a new and fancy job opportunity is coming just in time. But the issue here is that this new job is in a completely different tech stack or programming language. What can you do with it?

Today I want to share my thoughts on choosing the first programming language. Also - we will think together whether it is worth changing between language and tech stack - and how to make the such process beneficial for you and your manager.

My story

I had various jobs and titles in my 10+ years of IT industry journey. I was a Test Automation Specialist, a QA & Testing Analyst, an SDET, a QA Automation Engineer, and even a Software Engineer.

In fact, it is not about how fancy is your position’s name. It is all about your responsibilities and your influence on the team or department level.

Over the years, I’ve changed automation languages several times:

at my first job, I wrote UI autotests in C# + Specflow;
then - wrote API tests on Groovy DSL (SoapUI) and simply on “pure” Java;
then it was the Ui web test automation with C# again;
then it was a big project, and tests on Scala.
Then it was Java, Spring Boot and software engineering job.
Right now, I am writing my tests and tools mainly in Scala.

Questions to consider before choosing a language

Image from Reddit

Before choosing the programming language for automation, you need to ask yourself a set of questions.

Technological stack and architecture.
Testing processes. Are there testing processes on the project? How do you plan to use automation testing? Who will be responsible for such tests - testers, developers, or a separate team of engineers?
Test automation strategy. Is there an ultimate goal of automation? What types of automation tests do you plan to write? When will such tests be executed? How do you use the test reports?
Programming and automation experience. Do you have programming experience? Did you write automation tests at all?

How many vacancies are currently on the market for this language
How deeply do you plan to learn this language? What is your technical background?
Is the language and the infrastructure around it mature enough? Do stable frameworks and tools exist that will allow you to quickly write the necessary solutions?
How do you plan to improve your career by learning a language?

On environment

Your work environment matters. Depending on the team, its skills, and you personally - the choices for language can be different. Let me show you a few cases and possible solutions for dealing with automation.

One tester. A small team. No prior automation experience.

Options:

Invite a short-term automation consultant. They will help develop an automation strategy and initial Proof-of-Concept for automation tests.
Choose the same programming language for automation that the developers in your team use. They will be able to advise and help you with the tests. The downside of this approach is that developers can write complex programming languages that are not very efficient for writing simple end-to-end tests. (Or course, you also can write Selenium tests in Haskell.)
Find a mentor (inside your company or elsewhere) to help and guide you in learning automation.

One tester. A small team. Some automation experience.

Options:

Choose the language you know best. Also, consider how many automation tools and libraries exist for the language. It may happen that many tools will have to be written: “from scratch.”
Alternatively, you can choose the language and stack identical to what the developers use. Learning a new programming language to create several UI autotests is relatively easy (if you already have some background).

A team of testers. No automation experience. You’re the lead.

Options:

Select an individual tester from the team and train him or her in test automation.
Hire a dedicated automation engineer who will later train all testers in automation.
Train developers to implement automation tests by themselves. (Difficulty level: HARD).

On choosing languages

Image from Reddit

There are many programming languages. Some languages are harder to learn. But such languages can give a boost to writing tools.

If you like the world of front-end - then your choice is Javascript or Typescript and its army of frameworks. This world of tools and libraries is a very dynamic one. You will always have a job!
If you’re more into network stuff, hacking, data science, or other back-end stuff (or just like simple and friendly code) - look towards Python.
For those who want enterprise-level stability, consider Java or C#.
Those who want to work with blockchain - can learn smart-contract languages like Solidity, Plutus, and other platform-specific ones. You need to learn Haskell, Rust, and C++ for more low-level things in the blockchain world.

About new languages

If you choose a programming language that is new or not a mainstream, then be prepared to learn it deeply and then look for a job for a longer time. As a side effect, the salaries may be slightly higher. But it may NOT be.

Additionally, if the language is new, you will need to write a lot of the functionality of the frameworks of other languages by yourself. On the one hand, it’s an opportunity to improve your technical skills. On the other hand, it requires time and effort (which may not be available to you).

On changing languages and tech stacks

Image from Reddit

If you are a Junior or Middle automation engineer. It is relatively simple to change the programming language and stacks at this level. If the new project has an established framework and/or senior engineer to mentor you - you can switch without fear. You will need to learn a lot - but a good mentor can make this learning process as quick and painless as possible.
If you are a Senior engineer. If you join the new company as the first engineer in the team, you will need time to learn new things and write proofs-of-concept that might only work on the first try. On the other hand - if you join a well-established team, your onboarding process will be more straightforward.
Switching between statically typed languages (Java, C#, and C-like languages in general) is quite simple. It will be a little more challenging to transition from statically typed to dynamically typed languages, such as JS, Python, Groovy, Ruby, etc. (or vice versa). Plan more time for learning new concepts.
Few words about functional programming languages (Scala, Closure, Haskell, and others). It is possible to write in OOP style on some of them (in particular on Scala). For tests, in principle, it will be enough. But functional programming will still need to be mastered to get the most out of the language. And it takes even more time than the languages from the previous point. Plus, you can only write in a functional style in languages like Haskell or Closure.
About working at FAANG and other western startups. They all hire engineers - the people who can master one or another language in a short period (not instantly, of course) and write production or test code with it. But do not worry. In most cases, such companies will have a lot of experienced engineers who will help with code reviews and feedback. So what matters here - is your basic technical knowledge and problem-solving experience. Programming language - is only one thing among your tools.

On skills

Programming language is only a tiny part of what a test engineer or SDET should be able to do in automated testing.

In addition to the language, there are also basic approaches to testing and automation. There is technical knowledge of various systems’ architecture and how to test them.

There are many additional tools for writing and running tests - build tools, automation frameworks, various libraries for logging, parallel runs, reporting, and CICD tools. And also - design patterns and the ability to write clean and understandable code.

Most of the automation skills do not depend on the programming language. You can find analogs in the selected technology stack for most of the tools. If such tools do not exist in the chosen technology, why not build it yourself?

The main job of an engineer is to solve a business problem most efficiently. Effectiveness here lies primarily in how fast, scalable, and understandable the tool or approach is to other engineers and managers.

Conclusion

Image from Reddit

Choosing a programming language for automation - is a hard thing. Especially if you are a single tester in a small team. Each language has its pros and cons. Each language brings an ecosystem with it. But asking good questions BEFORE making such a choice - can decrease the amount of “headache” in the future.

Changing languages and stacks later in your career also have their own implications. You can be a generalist and scratch the surface of many languages. Or you can be a specialist in one language - but do technically complex things and optimizations. It’s up to you to decide which engineer you wanna be.

In the end, I want to state a few general things that you should never forget when you work with test automation:

A few stable and fast tests are better than a “super framework.” If you plan to create only a Smoke suite from a pair of end-to-end tests “for every day, “ you can do it with almost any tool. Just make sure that the tests are fast and stable. Such tests will bring benefits to the whole team.
Automation can take many forms. It can be a UI or API test for the most critical features, deployment scripts, or database migration. Small script that automate daily routine tasks can bring more value to the team than a lot of simple UI tests.
Automation strategy is the first step. If you plan to create automation tests at different levels, think about test coverage, launch on various platforms, then first create and communicate the automation strategy to management. Without a clear understanding of the “end” point, you will not be able to tell whether you are moving in the right direction.
Learn programming and the technology stack. Writing a large project for automation requires the same knowledge and programming skills as writing feature code. Therefore, constantly improve your level of expertise.

On test engineer’s maturity

2022-09-25T06:00:00+00:00

Photo by Lance Grandahl on Unsplash

Questions

From time to time, similar questions arise in various tester chats and forums:

Do I know enough as a junior or trainee?
What should the middle engineer know?
When to understand that you are a senior?
How to get the position of lead or manager?

I have also asked myself and other people such questions. But unfortunately, he did not receive comprehensive answers.
After all, there is no single correct answer to such questions. Everything depends on the context: you, the company, and the market.

Test Engineer Maturity

Instead of looking for complex answers, I suggest paying attention to what questions we ask ourselves at each stage of our career development. Each company has its own stages of career development, but I see the following: Junior => Middle => Senior => Lead => Manager => Consultant => “Guru”.

Junior

How to test this feature or application?
Which test design technique is best?
What is Git, and why is it needed?
Which programming language to choose for automation (to get bigger salary)?
What is the best testing tool?

Middle

How to properly configure CICD pipeline?
How to fix all flaky tests?
What are design patterns, and how to apply them?
Why do we need to write clean code and follow coding conventions?
How to run tests in multiple threads somewhere on Selenoid?

Senior

How to improve the testability of a particular feature or service?
What are the problems with the system architecture? What will happen if “this individual component” on the diagram stops working correctly?
How much business value do self-tests bring?
How to encourage the entire team, from developers to managers, to use test results?
Which tests are better to do ‘manually’ rather than to spend time on automation?
How quickly and safely does this or that component work? But how does one component’s bad performance affect the system’s performance as a whole?

Lead

How to properly organize testing processes in a team?
How to ensure that the team achieves business goals?
How to create a culture of shared responsibility for quality in the team?
How to collect metrics, work with priorities, and communicate with different teams?

Manager

How does testing help minimize business risks?
How to motivate test engineers to grow and develop?
How to create testing vision for the company?
How to set up processes and engineering culture at the company level?
How to communicate test results to senior management (in their language)?

Consultant

How to create a better testing strategy for a specific company in a particular context?
How to convince the client that this tool or approach is better?
How to quickly make a Proof-of-Concept of an automation solution and get feedback?
How do you train engineers or managers on a chosen concept, approach, or tool?

“Guru”

Why do we test?
How does testing differ from verification?
How can we tell if our expected results are correct?
How do we know what we know?
How do we learn?
Are there universal approaches and heuristics for testing?
How to measure or change people’s perception of quality?

Conclusions

Of course, if you are at the same stage, no one forbids you to ask any other questions from different levels.

But the effectiveness of your work and the answers to questions will directly depend on your skill level and the scope of work.

And what questions do YOU ask yourself on the project?

Testability and how to improve it

2022-08-14T06:00:00+00:00

Photo by Jeswin Thomas on Unsplash

Why is it testability?

At first glance, the typical test engineer already has enough things to do:

Work out the requirements.
Create plans and strategies.
Write test cases.
Report and re-check bugs.
TEST!

Somewhere between these things, you still have to find time for automation. And you still need to make a demo for the customer.

Why even bother with that testability?

Isn’t that the task of whoever writes this code? Is it possible for a test engineer to influence the team (especially the architects and developers) - and make the software a little more open and easy for testing?

Let’s try to figure this out a little together.

Testability: officially and not so much

What is the definition of testability given by industry standards?

According to ISO/IEC 25010:2011: Testability - degree of effectiveness and efficiency with which test criteria can be established for a system, product, or component, and tests can be performed to determine whether those criteria have been met.

According to ISTQB terminology: Testability - The capability of the software product to enable modified software to be tested.

In simple terms, testability is how convenient and easy it is to test an individual component and the entire system.

Testability depends on multiple factors:

How easy is it to control the internal state of the components during testing?
Is it hard to observe the system and its components during tests?
Is it possible to test components in isolation?
How fully documented is the component?
Is it doable to write automated tests for it?

Testability at the code level

To efficiently test classes at the module level, you need to be able to create Mock objects for all class dependencies.

If the developer applies the Dependency Inversion principle, there will be significantly fewer testing problems. Ideally, you need to pass all dependencies through the class constructor.

Also, don’t forget about observability and controllability, even at the class level. The developer (or test engineer) must be able to see the state of critical parts of the object at any time. It would be much better to add different methods to get this information directly to the test class.

But still - this aspect is entirely under the responsibility of the developer.

Testability at the architecture level

A practical tip at the architecture level would be to separate infrastructure and domain code. By domain code, we understand all the code responsible for business logic. Infrastructure code works with external dependencies - databases, message queues, third-party services, etc.

To separate one type of code from another, you can design components using the Ports and Adapters (Hexagonal) pattern. Alistair Cockburn proposed this model in 2005.

Maurício Aniche gives a more practical example of such a model in the book Effective Software Testing:

Testability through the eyes of testers

Okay, but how can a particular test engineer affect testability?

Unique properties of elements

If you’ve written more than one UI test, you can tell a lot about the pain of “long and fragile” locators. Changing the elements on the page is one of the main reasons for the instability of such tests.

As a test engineer, you can do two things in this case to improve testability:

Agree with the team (or architects) to add unique properties for each new critical element;
Agree with the developers that you will add unique properties to elements whenever it’s needed (for legacy components);

Having an special property like “automation-id” or “data-id” for a web element - saves a time and nerves for a test engineer.

Components in isolation

Please keep track of how convenient it is to test the component in isolation. Ideally, a developer or tester should isolate a single part (e.g., a single microservice) and test only the business logic.

Is it easy to replace the actual database with an in-memory one?
Is it convenient to fill the database with the necessary test data for the correct component operation?
Is it easy to block external requests to third-party services?
Is it hard to send or read messages in the Message Queue for a single service?

You can always work together with a developer to build such tools for the components of your system. In many cases these tools can be find as libraries. If not - it is a good point to create it and spread as inner source in the organization.

API for system management

It is good to analyze how you test (from time to time). Some components in the system may work using probabilities or specific third-party data. The tests for such components will be unstable. But such systems should be tested and, ideally - automated as well.

In search of the answers, we can think of it as a system of saves in a computer game. Imagine that you need to test how the “boss” behavior at the end of the level. You can verify it by playing the whole level from the start till the end. If we talk about one or two minutes - it is ok. But if each level in the game takes even ten minutes or more - such tests quickly become unrealistically long.

A better way will be to use magic save, reach the end of the level and test exact behavior.

The solution is to make a separate test API that will allow you to bring the system to one or another state. Using such API will make tests faster and increase the stability and predictability of the system under test (at a certain level).

That is the reason why many games have a long list of cheat codes.

You can think about such a “cheat” system in the context of your application. These can be special commands you need to enter or separate ways of creating data that the system will process as a test (even in production). Such data will then be easier to filter and analyze.

And how do YOU improve testability on your project?

The most complicated automated tests

2022-07-17T06:00:00+00:00

Photo by GR Stocks on Unsplash

Are UI tests that easy to write and maintain? Are unit and integration tests really the most complicated tests.

This blog post will explain my thoughts on which tests are complex and why. If you agree or disagree with these thoughts, let’s discuss them in the comments.

What started it all?

At the start of my career in test automation, I focused entirely on UI tests. Then, after some time, I got to know the zen of the API level and started writing such tests wherever possible (wherever that API existed). It was a time of SOAP and REST HTTP testing.

But during the first years, no matter how many automation tests I wrote, I always thought my tests were too easy. Real “hardcore” developers write complex and scary unit and integration tests. To write those unit tests, you must know the programming language and frameworks very profoundly.

These thoughts were in my head until I decided to look at frameworks for unit testing and unit tests in one of the projects at work.

Unit tests

On the one hand, it seems that unit tests are difficult to write. But it becomes much easier after you understand the very essence of such tests and master at least the basic capabilities of any test framework (such as JUnit), mainly when you write features and tests yourself.

We concentrate on class level (or several closely related classes) in unit testing. The main idea is to test business logic here. Then you can mock all external calls using a mocking framework such as Mockito. Simple test data is created in the form of stubs or fakes.

When you figure out how to write mocks and tests correctly, the speed of creating new tests depends only on the complexity of the business logic of the class itself. As for supporting such tests, it is also not tricky: such tests are closely related to the feature code, so they need to be changed often.

Implementation complexity of unit tests is easy as well as the maintenance cost. The only question is how testable the feature code is.

The critical thing about unit tests is that the developer fully controls the environment, test data, and behavior of dependencies.

Integration tests

Integration testing is one of the most controversial themes among engineers. Every person can understand the integration level on their own.

By integration tests, I will understand the testing of individual components in isolation from the entire system. For example, when we have many microservices on the backend, the integration test can be both at the service level (we check the business logic of the service as a whole, and at a more granular level - we verify the operation of the service with the database).

In this case, we mock requests to other services or third-party components using Wiremock. Other dependencies can be run locally as Docker containers (testcontainers) or in the form of in-memory databases.

Such tests show a more realistic picture of the operation of part of the system and its components.

The implementation complexity of such tests is average. The main problem here is the correct configuration of dependencies for the service.

The maintenance is also not very hard. If you have a proper design for your test and feature code, the interfaces between components will be more stable than in unit tests. So the change of database or messaging broker should not have a high impact on the core logic of the system or service.

As for the control, the developer has limited control over the dependencies. It is possible to configure the database or message queue. You can also set some test data as pre- or post-conditions. But you can’t be 100% sure that the DB implementation is correct.

End-to-End tests

By end-to-end tests, I mean both UI and API tests for the system. The main thing in such tests is that we work with the system as a user. You can call such tests black-box if you want.

Once you master the Selenium Webdriver’s API and know how to use Page Object (and other) patterns, implementing end-to-end tests becomes very easy. Of course, there can be exceptions: very complex and tricky logic, exotic browsers, or non-standard APIs. Creating new UI tests is trivial for a developer (compared to unit and integration tests). That is why everyone believes that end-to-end tests are the easiest.

But the thing is that automated testing is not only about implementing new tests. It is also the maintenance of such tests. It is debugging and fixing. Here lies the biggest pitfall (almost an iceberg) in the perception of end-to-end testing.

A LOT of external and internal factors affect the performance and stability of such tests:

databases can perform slow replication and return inconsistent data
hundreds or thousands of backend microservices can stop working at any time (causing cascading problems)
critical services may receive data from the message broker (e.g., Kafka) with a delay
third-party services can also work unstable or be completely unavailable
the system may be under the influence of an external DDoS attack
load balancers and caches can also be unstable
the web or mobile applications can work differently (depending on the phone model or the browser version)
requests from the browser in the tests may be processed more slowly than from the actual user
the test framework can be erratic (especially if you run many tests in parallel)
connections to containers in Selenoid can also disappear for some reason
connection to device farm can also be a point of instability

Considering all these hidden moving parts, the business, management, and even fellow testing team still expect almost 100% stability of UI tests.

But the reality is harsh. When any UI test fails - it needs to be isolated, analyzed, and carefully fixed. If you have a dedicated automation team, those few brave engineers need not only understand at which point of the test the issue occurred. They need to isolate the problem and report it to the particular development team for a fix.

Sometimes, the fix will be on the UI test or framework level. Sometimes it is a change in requirements or the urgent need to update the test data.

But in many cases, the fix also can’t be done immediately, especially if the root cause is unclear and need time for investigation. The development team has its priorities and tasks. Ignorance of such failing tests and issues increases the risk of having a bug in production. But anyway - it is a question of how effectively to use end-to-end testing on the project.

As a test automation engineer, you don’t have control of the system and its dependencies. Here is your browser - and that’s it!

So, which automation tests are the most complicated?

I don’t say that some tests are redundant and do not bring value. I just want to state that after an initial learning effort, the implementation of unit and integration tests becomes a relatively easy task. That’s why we need to have more tests at these levels.

UI tests can become a nightmare if they are flaky and slow. But it can slow the whole development and release process if the automation team is separate and there is no dedication to end-to-end tests from each development group.

End-to-end tests can bring value or be the most complicated automated tests to maintain. It can be a huge source of frustration for automation engineers.

Do you expect stable UI tests for your automation team?

Under The Hood: How Git works?

2022-06-26T06:00:00+00:00

Photo by Sebastian Herrmann on Unsplash

Finding a test engineer who would not have heard of version control systems is difficult nowadays. Some have worked (and work with quite “old”) - such as Subversion, Mercurial HG, etc.

But the vast majority of modern projects today use Git. If you need to download a “fresh” version of the product code or automated tests, go to the Terminal (or GUI tool) and perform a git fetch or git pull.

Learning the basic Git commands is not that hard. It is a bit more difficult to understand what merge and rebase are (and their difference).

But few engineers look deeper - how does Git work “under the hood”? Of course, you don’t have to know this to use Git. But I was still interested to understand. Plus, if you will be asked about Git internals at the interview - you will already know it! (But why ask such a question???).

Welcome to the world of graphs and hash functions. You can learn about another application of hash functions - Merkle trees - in my previous post.

A bit of Git theory

Let me remind you what a typical workflow with Git looks like:

Make a clone of a remote repository: git clone https://somerepo
Make a branch for your changes (fixes, tests, etc.): git checkout -b fix/autotests
Write everything you want (Git will notice it but wait for your actions): git status returns “changes not staged for commit.”
Add files to staged area using: git add
Create a new commit: git commit -m "Fix autotests"
Send prepared commit to a remote repository: git push origin fix/autotests
Pass the code review and merge the changes into the main branch: git rebase or git merge
Send changes to the main branch from the local repository to the remote one: git push origin main/master
Remove the branch, if necessary: git branch -D fix/autotests

The structure of Git

There are three types of objects that Git takes care of:

Commit - contains the author of the commit, date, message, and pointer to the tree of changes. Also, each commit has one (or more) parent commit;
Tree - contains a pointer to other trees and specific blob objects;
Blob - specific data (source code files, pictures, videos, etc.)

So what is the structure of the Git repository? This is a data structure called a graph. Or, more specifically - a directed acyclic graph.

Where can all this information be found? When you clone a repository or create a new one, all this data will be stored in a hidden .git directory in the project.

What are tags and branches? These are just pointers to a commit object.

What happens during git init?

Let’s create a new repository in the directory - testrepo. To do this, go to the folder and run the command git init.

As a result, we get a lot of exciting things in the .git directory:

Making a first commit

Create a new README.md file with any content and add it to the first git add. Now, .git looks like this:

We can see that Git created some new folders and files: 3a folder with a strange file inside.

It results from the SHA-1 hash function running on our README.md file. (You can read about the hash functions in one of my previous blog posts)

Git has a separate team to help us with research: git cat-file.

-t argument indicates the type of object
-p argument shows the contents of the object

Note: You may specify not the full hash of the object - but only the first six characters (Keep in mind that you must also set the subfolder characters - in our case it is 3a. And then the first four characters of the object)

In our case, the results will be as follows:

When we added the README.md file to the staging area, Git also created a blob object under the hood.

Next, create a new commit with the file and look at the contents of the .git directory again.

As you can see, now we have not one new object but three. One we have already seen (starts with 3ae2e2). What about others?

232c6a is a tree object. It points to a blob object (we also see the hash of this object and the name). 100644 is the permissions for this file.

85ac4e is a commit object. It points to an object of type tree (we see its complete hash) and indicates the author and the text of the commit.

More commits

To complete the task - add another file and correct README.md.

.git contains new objects:

First, we have a new commit object that contains a pointer to a new tree object, f71e00a8f2d3eff8348ad77149932dc11ab9e717.

In addition, in this commit we have parent 85ac4e3d6dba749723b69ce6f655c5acf5a00782

If we look at the new tree, we will see that it now points to two objects of type blob: a new file and a modified README.md.

Interestingly, the blob object for README.md already has a different hash. The reason is that we changed the file’s text - so the hash value also changed. But the old blob file also remained available (3ae2e2)!

A few words about the branches

Finally, I want to show what happens when we create new branches in Git.

Let’s run the git checkout -b fix/autotests command and look in .git.

If we look at the contents of the .git/refs/heads/fix/autotests file, we will see that it points to our second commit!

Conclusions

As you can see - Git under the hood looks not so complicated. Especially when you already know what hashing is.

Therefore, the next time you see unknown numbers and letters near commits, you will know that it is a just hash value from objects.

Blockchain for Test Engineers: Merkle Trees

2022-06-19T06:00:00+00:00

Photo by Camille Brodard on Unsplash

This blog post is a part of “Blockchain for Test Engineers” series.

In previous articles, we have already discussed hash functions and their usage. Today I propose to learn about an exciting data structure built using hashes - Merkle Tree.

Blockchain and transactions

A blockchain is a chain of data blocks connected cryptographically (each subsequent block points to the previous one).

In addition, each block stores a lot of other interesting information: the hash of the block itself, the number of transactions, the complexity of the mining algorithm, and more. But most importantly - each block contains a set of validated transactions.

But how can a user verify that a transaction is inside the block? Wasn’t it changed during the mining phase?

In the simplest case, the user must download the entire blockchain to his computer and check the required blocks. Each block takes up, on average, about 1MB. Therefore, the transaction verification process can take a long time (and requires hard disk space too).

But there is a way to check transactions quickly and with a much smaller data set than the whole block? The answer is yes - it is Merkle tree data structure.

What is Merkle Tree?

Imagine that we have eight transactions ready to be included in the block:

Apply hash functions to each transaction:

Let’s concatentate each pair of hashes and then hash the result:

We will continue the pairing and hashing processes until we get a single value:

The resulting binary tree is called a Merkel Tree. The last hash obtained is the tree’s root (Merkle Root). This root is also a part of the header of each block in the Bitcoin blockchain.

Other parts of the Merkle Tree are non-leaf nodes and leaf nodes.

What is the meaning of the Merkle Tree?

The advantage of organizing transactions in the form of Merkle Tree manifests when we check the integrity of the transaction.

In a simple case you can download all eight transactions, then generate a leaf nodes and recreate the process of creating a tree from scratch. And then - compare the resulting root of the tree with the one stored inside the block.

We can do it for a dozen transactions. But if there are thousands of transactions? Or millions, or maybe more? For example, some projects can contain billions of Merkle Tree leaves. So to re-create the tree, you need to compute a vast amount of hashes with the O(n) complexity.

Merkle Proofs allows you to mathematically check the integrity of the transaction without the need to download and recreate all transactions. You need to perform much fewer operations.

For example, we need to check transaction 2. In this case, we need to have Merkle Tree root and Merkle Proof for this transaction:

In our example, the proof contains only 3 hashes and another 4 hashes we need to calculate to get a Merke Root. Otherwise, we will need to download 8 transactions and calculate 15 hashes.

Instead of donwloading n transactions, we need to download O(log n)-sized Merkle Proof and also - Merkle Root.

In the Bitcoin blockchain, if we need to verify that transaction is indeed in the block, we need to download the block header (80 bytes instead of 1MB block) and then request Merkle Proof for a specific transaction. As a result, we can unambiguously check the integrity of the transaction with much lower resource costs.

Merkle Tree is used in Bitcoin’s “light” clients (Simple Payment Verification nodes) to quickly validate if transaction belongs to a particular block.

Where you can find Merkle Trees?

It turns out, that Merkle Trees are used in many modern systems:

Git and Mercurial version control systems - for storing versions of data)
Cassandra, DynamoDB - for detecting inconsistencies between replicas and minimize the amount of data transferred
Bitcoin, Ethereum blockchains - for validating integrity of transactions
IPFS file system

High Tech - Low Test or Problems with Modern Testing

2022-06-12T06:00:00+00:00

Photo by Leyre on Unsplash

I have been working in test automation and development for more than ten years. I’ve seen good projects, and I’ve seen awful ones.

In this blog post, I want to highlight the problems I see in the testing industry and how we all can fix it. (You may have a different experience - so let’s share it in the comments!). This is a text version of my talk that I gave at Abu Dhabi MoT meetup at the beginning of 2022.

What has changed in the industry in 10 years

Over the past ten years, we have witnessed a significant technological boost - driverless cars, artificial intelligence, AR / VR, blockchain, drones, and robots. Much testing and test automation moved from the desktop to the web and mobile devices.

From Waterfall models, many businesses went to Agile and Scrum.

From big and costly tools for writing automation tests, we have moved to small, flexible, and, most importantly - free libraries and tools. Now we have intelligent reporting systems and tests in Docker containers. Running thousands of tests in parallel in the clouds now is not so expensive.

But at the same time, many things and statements have remained unchanged.

Here are some of the endless topics that are constantly raging among testers:

“Manual testing is dead!”
“Let’s automate everything! Through UI - that’s the best!”*
“SDET is the peak of the evolution of a test engineer! I want to be like Google!”
“I can’t get a job in development, so I will work for a year or so as a tester and then try again.”
“Salaries of testers - at the bottom!”
“Test engineering is a job for low-skilled professionals!”

But why is there such a situation in the industry? Could the problems be something else? What issues do I see?

Problems with modern testing

Photo by Karla Hernandez on Unsplash

I will divide the problems into two large groups.

The pursuit of technology

Concentration only on short-term goals. Many engineers prefer only quick solutions. Such an attitude to automation tests is prevalent. The engineers quickly wrote “something” without any thoughts of maintainability and ran out to a new project.
Resume-driven development. Many testers only consider jobs and projects as a way to get a fancy new line in CV. There is nothing about deepening the skills and solving business issues with testing and automation.
The green pipelines cult. In this case, test engineers who have successfully included tests into their CICD pipelines begin to be obsessed with making tests green all the time. Such obsession often results in ignoring or even removing unstable tests (which leads to ignoring the problems behind ignored tests).
Framework-oriented automation. In some projects, engineers are so eager to write the perfect framework that they forget about the tests themselves. As a result, we have 3-6 months of development without tests!. But for the customer, the absence of tests is a direct indicator of the lack of benefits from automation.
Search for silver bullets. Some engineers create a successful framework for one project (and for one context) and then start pulling it into all the others as a “perfect solution.” It is not reusability; it only applies one hammer to all the nails and bolts.

Insufficient level of skills and knowledge

The lack of knowledge can take multiple forms.

Technical knowledge

In interviews, senior candidates often quickly answer any questions about testing. They can draw a “testing pyramid or any figure you want” for you, tell you all the test design techniques in the world, and write the perfect bug report. But candidates quickly lose confidence when you ask questions about fundamental technical aspects (such as how HTTP or networks work).

You can tell me that not every project requires specific knowledge. That’s true. But test engineers should know (or at least be aware of) the fundamental technical knowledge. Ideally, a little bit deeper than the first link in the Google search.

90% of modern systems work with the network in one way or another, send messages or requests, and use the database or distributed storage. Multiple layers of abstractions can cover it - but in a nutshell - it all works similarly.

Programming and architecture

Some test engineers still believe that learning programming is complicated and unnecessary. Let the programmers write the code!

The other part is those who have learned to code a bit but concentrate their efforts only on the UI tests. The world outside of these tests does not seem to exist.

Knowledge of system architecture and how they work is a scarce skill among test engineers. All this is considered work for experienced “bearded” architects or only senior developers with dozen years of experience.

But without knowing how the system works inside and with each other, it is easy to miss a lot of critical errors. On the other hand, the lack of technical knowledge will make it hard to describe such issues in a bug report.

I’m not saying that without knowledge of the components of the system, it is impossible to point out lousy testability. As a result, we continue to write fragile XPath locators because no one has added IDs for elements.

Knowledge of business and product

Of course, it all depends on the project. There are some projects where a testing team works as if on a factory line: get the build, test it and pass it to the next unit (without asking any questions).

Test engineers are significantly lacking in product knowledge. Few engineers work with users or with customer support. But those people can provide helpful information about the riskiest parts of the application or give feedback on the usability system.

Even fewer test and automation engineers think about how their work affects business. Do those automation tests save the company money? Or provide feedback faster than the manual testing?

Personal development of engineers

Regarding personal development, there are two significant groups of engineers exist:

“Why should I learn to program (any other skill)? That will make me a less skilled tester!”
“I allegedly want to study, but I’m waiting for my one-year feedback. My manager or test lead will come and tell me what I need to learn.”

But here lies the most critical point. ONLY YOU ARE RESPONSIBLE FOR THE PERSONAL DEVELOPMENT OF SKILLS, KNOWLEDGE, AND CAREER!

Your manager (or anyone else on the Internet) will not be able to create a map of your skills for you. Only you know your strengths and weaknesses. Only you know (or guess) what gaps you have in your knowledge. Only you can understand how to gain that knowledge and reach a new level (and get that dream boost).

The manager can only adjust your plan and indicate the company’s resources: the budget for attending conferences or other developments.

A vicious circle of uncertainty

So what questions does an average test engineer usually ask himself?

“I can’t write code - I’m not a developer …”
“Who am I to talk to an Architect about the architecture or testability of system components?”
“The DevOps team have turned off unstable tests in the pipeline - well, they know more about how it should be …”
“The manager keeps asking me: what exactly are you testing? Why is that? And that?”
“Business is just trying to cut the costs and fire the whole department of unnecessary testers ….”
“How do I even know what problems our users have?”

As a result, most test engineers fall into a circle of uncertainty.

Testers cannot correctly prove their benefits to the team, other engineers, management, and business. In such cases, it is hard for a tester to build trust and confidence inside the team;
Demotivation of test engineers is growing. The idea to switch to software development or management grew along with demotivation.
Many experienced and senior engineers leave the testing industry. Cases of successful testers are becoming fewer and fewer. Cases when test engineers influence the product and processes and make technical improvements - can be counted on your fingers. Achievements and stories at conferences come down to “well, we got this framework or this tool for test management - now it’s all beautiful.” There is a lack of people who can write new libraries and tools. There is no one to test complex and scientific software. There is no one to do research in the field of testing;
The industry continues to think that a tester is someone who “just pushes buttons and does almost nothing.” Young test engineers see the same picture but are even more demotivated;

How to change the situation for the better

Prepare a plan for your professional development. What skills to acquire, what to learn, and how to use those skills at work. If you can gain knowledge and skills in the company - that’s excellent! If not - look for another project or company. There is always a choice - and it is in your hands.
Learn to program. I’m not talking about skills at the level of a senior developer. But the ability to write a script can remove a lot of monotonous work from the deployment or configuration. Reading and understanding someone else’s code gives a better understanding of how the system works in general and which cases are not yet covered.
Deepen your knowledge of application and systems architecture. Start with the system you are testing now. Decompose it. Disassemble architectural diagrams. Think about how and where an individual component may “fall” and how the system will respond. Talk to your architect.
If you are creating scripts or automation tests, always ask yourself: who and how will use my code (tests)? Is it written clear enough? Is there a detailed report? How easy will it be for a new engineer to understand the code?
Learn how users work with your system. What is your product business model? What are the riskiest parts? This information will help you test what is vital to the user and save money for the business.

Blockchain for Test Engineers: What is blockchain?

2022-06-05T06:00:00+00:00

Photo by Robynne Hu on Unsplash

This is the sixth blog post in the “Blockchain for Test Engineers” series.

We have explored concepts from hashing, encryption, and distributed systems in the previous posts. Now it’s time to apply this knowledge and understand what blockchain is.

When I first started to google on this topic, I found many articles, videos, and even courses dedicated to the blockchain. But all these resources eventually fell into two categories: too superficial or too deep. There was nothing in between.

Today I want to briefly, technically, and with pictures, tell you what blockchain is and how it works. We will save questions of advantages and disadvantages and other deep technical details for another time.

Why do we need a blockchain if there is already a bank?

Each of us uses the services of a bank and makes transactions from a card or account. We can see the list of transactions by going to the bank’s application. (Or by requesting in paper form as a reference).

Each transaction contains information about the time and purpose of the payment. In addition, we can also see the recipient of each transaction.

Two parties can see this list of transactions - you, as a client of the bank, and the bank itself. The bank “securely” stores the information about your transactions on the bank’s servers. By receiving this list, you trust the bank that it did not inadvertently manipulate your balance.

In addition, the bank has every right to freeze your funds if it considers them suspicious.

Don’t you think that requires a lot of trust and single-handedly manages your accounts?

What is blockchain?

The original idea of blockchain emerged as an alternative to regular bank payments. (Of course, the blockchain is not only used for financial transactions - but let’s start with them).

Instead of the bank servers, we have a peer-to-peer network of thousands of computers. Each node in the network contains an entire history of transactions made by all users in the system. As transaction data is stored on multiple nodes simultaneously, nobody can “freeze” your money or delete transaction history.

The nodes in the network share information about transactions with each other. So the state of the transaction history is synchronized.

Another key difference from regular banking is that transaction history on the blockchain cannot be changed and is visible to anyone.

How does it work technically?

We can think about transaction data on each node as a form of a distributed ledger (or, in other words - an extensive distributed database of data). The ledger here means that information contains a few critical components: who send the money, to whom, and which amount.

The ledger is a set of identical blocks (that are linked cryptographically) and distributed across all nodes in a peer-to-peer network.

Each block has four fields: block size, block headers, transaction counter, and transactions.

The block headers contain:

the block version;
the hash of the previous block. The blockchain is named so - it is a chain of blocks;
the Merkle root hash of all transactions included in the block;
timestamp;
difficulty target;
nonce;

Why store the hash of the previous block?

Here is the beauty and power of the hashing. It helps to ensure the consistency of the blockchain and prevent changes.

Imagine that an attacker wants to change the content of a transaction that belongs to block one. The change in the transaction will change the hash of the transaction. Then the change of transaction hash will change the hash of block one. If an attacker wants to spread a new block across the network, he must modify the other blocks after block one. (because they are linked).

In addition - the attacker then needs to spread a new chain of blocks (modified) across all the nodes and force everybody to use an updated version of the chain. Given the number of nodes in the network, it can be a challenging and expensive task. Why expensive? An attacker needs to re-create many blocks, which takes resources and time.

Every block and every transaction are digitally signed. Any network user can verify this signature and ensure there were no fakes. So it will be suspicious if all blocks in the “attackers chain” are signed by a single user.

How user creates a new transaction

The client forms a transaction using the command line or special applications (crypto wallets). Inside a transaction, there are sender, recipient, and the transfer amount. In addition, he sets the transaction fee. (We will see different transaction models in the upcoming blog posts).

In our example, client one connects to node one and sends trx1, and client two sends trx2 to node three. The client has no control over which node it connects to and where it sends the transaction.

Each network node collects incoming transactions for some time in its local storage (MemPool). Nodes exchange incoming transactions with each other (but it is not mandatory. Nodes can be at any place in the world and have different configurations - so the view of the network can differ from node to node).

From time to time, the node forms a new block of transactions. The node signs the block. In addition, to finally create and distribute a valid block in the network, the node must participate in the Proof-Of-Work consensus.

Mining new blocks

Each node with a new potential block starts a race with the other nodes who prepared a candidate block. The ultimate goal of each miner node is to make a special kind of block hash. For example, it can be a certain amount of zeroes at the beginning of the hash value.

The miner adds a number (nonce) to the block to find such a hash. If the hash is not equal to the desired one - the miner increments the nonce and does it again. This task is simple but resource-intensive - you need to make a tremendous amount of trial and error by brute force. This process is called mining.

The first node that finds the correct nonce and sends the block to the network - receives a reward in coins plus optional fees from each transaction. The reward is not immediately accessible to the user but only after adding some amount of new blocks.

Block validation

When a node finds such a nonce, it sends a new block to other nodes in the network. The node that receives the new block validates it and all transactions inside.

What is checked inside a block:

block structure
block header hash is equal
block timestamp
block size
the first transaction is a coinbase? (Coinbase transaction transfers the reward to the miner - it is always first in the block)
all transactions within the block

If everything is in order with the block - it starts to spread over the network as a new valid block.

Block propagation

Sooner or later, nodes will distribute the new block over the network using some protocol (for example, it can be Gossip protocol).

In Bitcoin, a new block appears on the network every 10 minutes (the internal mining algorithm checks the complexity of the mining for all peers and adjusts it every 2048 blocks). But this does not mean at all that your transaction will be in the block in 10 minutes. But more on that next time.

After some time, each node in the network gets the latest mined block. In parallel, other nodes can mine the subsequent candidate blocks.

What you need to consider when testing the blockchain

the blockchain networks are huge
the blockchain networks are always in a working state
each change in the network affects many users
you can’t modify the data once saved on the blockchain
the information you save on-chain during testing is visible to anyone

I hope that now you have a general understanding of the blockchain. We will gradually delve into different parts of this enormous system and see which thing we can test in the following blog posts.

Blockchain for Test Engineers: P2P networks and Gossip protocol

2022-05-29T06:00:00+00:00

Photo by Federico Beccari on Unsplash

This is the fifth blog post in the “Blockchain for Test Engineers” series.

Blockchain is not a usual distributed system. It is a peer-to-peer network of nodes that communicates with each other.

Today we will have a brief info on what peer-to-peer network is, why it is used in building distributed systems. Additionally, we will take a look at the way how new messages are propagated in the peer-to-peer networks.

The ways how networks can be organized

Client-Server model

This is a very common way for us to organize networks. It is a very familiar model for any test engineer working on WEB or mobile applications.

This type of organization allocates one (or more) nodes that have a server role. Other nodes are clients. Clients make requests to the server using various communication protocols (HTTP, RPC, etc). The server processes the request and responds to the client.

Despite of fact, that this way of system organization is a very widespread - it has it’s own problems. The main disadvantage of the model is that the server is somehow the only point of failure. Attackers can either take over the server and access information from customers (thereby interfering with the normal operation of the application). Or - with the help of DDoS attacks - make the server inaccessible to clients.

One way to deal with a single point of failure is to use multiple servers in different parts of the world. This approach works but adds to the difficulty of replicating data between these servers. But about this - another time :).

Peer-to-Peer model

It turns out that you can avoid problems with a single point of failure, even without multiple servers.

In Peer-to-Peer (P2P) networks, each node (computer) is equal to another (logically). In fact, each node is a server and a client at the same time.

Each node can provide data at the request of others, but also make requests and communicate with other nodes.

Why it is hard to deal peer-to-peer networks:

implementation of the reliable P2P system is a complex task;
configuration can be completely different from node to node;
nodes can be online or offline at any given point of time;
it is not possible to trust the message from other nodes (In client-server model you can trust server node at some degree);
additional effort needed for communication and coordination of messages in the system;

Well-known P2P examples

As peer-to-peer technology has evolved, a lot of real-wolrd implementations have appeared.

There are several generations of Peer-to-Peer networks:

First generation. Dedicated servers are still used as databases or node coordination servers. Examples: Napster, EDonkey2000.
Second generation. There are no dedicated servers at all - all nodes are completely equal. Search and download protocols from other sites are not very effective. Example: Gnutella.
Third generation. The network is completely decentralized. Data retrieval is performed more efficiently and quickly - using an algorithm that uses distributed hash tables. Example: BitTorrent.

Communication in a peer-to-peer networks

As we learn peer-to-peer networks more, one of the most fascinating questions may appear: “How do peer-to-peer nodes know about any updates or messages?”

If we are dealing with a centralized system, the issue is solved very simply.

All nodes are connected to the server node (Node 1) and receive information from it about any state changes. It does not matter whether the nodes themselves poll the server, or the server distributes new messages themselves. But if the server node stops working for some reason, the system will no longer be able to receive new updates.

The task becomes more difficult when there is no dedicated node in the peer-to-peer system, which would be the “server” or “coordinator”. Each node is self-sufficient and equal. The only thing that can be done in such a system is to exchange messages.

The Gossip protocol helps to solve the problem when the network is decentralized and unreliable and when the nodes may also be unreliable.

How the Gossip protocol works

The Gossip protocol works in much the same way as gossip or epidemics spread in life. To do this, we use a simulator.

Imagine that we have a peer-to-peer decentralized network of 20 nodes. As soon as one node receives a new message - it must spread it to all other participants. For each node, we can choose fanout values - how many other nodes we can send messages to in one round. For example fanout = 4.

In each round, the receiving node sends it to the other four participants.

In the next round, each updated node sends message to other four randomly chosen nodes

Yes, rounds are repeated until all nodes have a message.

Theoretically, you can calculate how many rounds it takes for all nodes in the system to receive messages and be synchronized. The number of rounds depends on the number of nodes and the fanout value.

There are multiple ways how node can get information about new updates and messages:

Push model - nodes with new updates send them to other nodes
Pull model - nodes actively request updates from other nodes
Push-pull model - nodes can request for new and send updates

Pros and cons of Gossip protocol

Advantages:

Simplicity of the protocol
High fault-tolerance
High scalability O(logN)

Disadvantages:

Slow work
Delays (new messages are not processed until all rounds are over)
Hard to debug

Conclusions

Peer-to-peer networks are not a “silver bullet”. They have pros and cons. That’s why client-server approach is a widespread right now. P2P networks can be useful in some cases - e.g. in the blockchain networks.

Although the Gossip protocol is quite simple and obvious, it has many applications in real systems. In the following posts, we will see in more detail how the Gossip protocol works on the example of Bitcoin and other blockchains.

For those who want to understand the work of the Gossip protocol in more detail and theoretically - I recommend an interesting scientific work on this topic.

Blockchain for Test Engineers: Distributed Systems

2022-05-22T06:00:00+00:00

Photo by Shubham Dhage on Unsplash

This is the fourth blog post in the “Blockchain for Test Engineers” series.

In the previous posts, we talked about the tiny little pieces of blockchain knowledge: hashing, encryption and digital signatures. This time we will examine bits of the vast topic of distributed systems.

Why would I need to know about distributed systems - I’m just a tester?

If you are testing an application that is little more than a couple of simple scripts, there is a non-zero chance that you are developing (or using) distributed systems in one way or another.

Does your application have a database? It may be distributed.
Do you have microservices on the backend? It is also, in essence, a distributed system. Do load balancers regulate the incoming traffic? Balancers can also be distributed.
Components exchange messages via messaging (for example, Apache Kafka)? And that can be distributed too. Do you host several data centers in different parts of the world to give users faster answers? This is also a distributed system.

In fact, almost any large modern application is a distributed system. And consists of several smaller systems.

In my opinion, test engineers who deal with distributed systems in one way or another should at least know these three things:

how do these systems work? (easy level)
how can these systems fail? (medium level)
how can the risk of failure be reduced? (hard level)

Also - almost each blockchain system is distributed and peer-to-peer in a nutshell. So you need to know how distributed systems work and on which principles they are built.

Why do we need distributed systems at all?

Resiliency. If all calculations are carried out on only one machine, then the failure of such a machine will mean that the entire system will be inaccessible to the user.
Performance. Even the most powerful one, no single computer can handle simultaneous requests from millions and billions of users.
Handle significant computational problems. Many tasks will be impossible or highly time-consuming to compute on a single machine.

What is a Distributed System?

In simple words, a distributed system is a system that consists of many nodes (for example, computers, processes, devices) that communicate over a network and perform some form of a task together.

Leslie Lamport, a renowned researcher and author of scientific papers, gave this description: ”. . . a system in which the failure of a computer you didn’t even know existed can render your own computer unusable.”

Distributed systems are a really vast topic. A lot of researchers dedicate their lives to learning distributed systems.

As an engineer, you need to be aware of the following things about distributed systems:

what are timing models (synchronous, asynchronous, or partially synchronous)
what is the type of communication between nodes
what are failure modes (the node can stop working in case of failure or recover automatically)
what are failure detectors in the system (how nodes can discover that something wrong happened to other nodes)
what are leader elections, quorums, and consensus protocols mean (RAFT, Paxos, Multi-Paxos, etc.)
how to calculate time in distributed systems (primarily if the nodes are geographically distributed around the world)
what are the standard whitepapers and principles in distributed systems (e.g., FLP Consensus)

What are the misconceptions about distributed systems?

Developers who write code for distributed systems are far from ideal. Especially if they are new to the industry or have never dealt with massive systems. Such developers can form many false assumptions about distributed systems in their heads.

Peter Deutsch (and several other engineers) from Sun Microsystems back in the 90s formed a list of these assumptions. It can be found as “8 fallacies of distributed computing” on the Internet.

Eight misconceptions about distributed systems:

The network is reliable. In reality, packets in the network can be lost; services can stop and endlessly wait for a response.
Network latency is zero. There will always be network latency - at least, it’s limited by the speed of light. It is impossible to instantly transmit a packet over the network from one point on the Earth to another.
The bandwidth is infinite. The bandwidth is limited, but it can also be different for different nodes.
The network is secure. In fact, any messages sent over the network can be intercepted by attackers. Or hosts can be attacked or blocked.
The network topology does not change.
There is always one administrator on the network. Nodes and subnets can be managed by many various companies with different security policies.
Transport costs are zero.
The network is homogeneous. In the real world, nodes on a network can be completely different, with different characteristics.

Distributed systems from a testing point of view

As distributed systems have many aspects, let’s take one of them as an example: communication.

By communication, we mean exchanging messages between two nodes (Node 1 and Node 2) or between the client and any node in the system. The most straightforward and practical example is when a client makes a standard HTTP request to a server.

What can happen in communication between two nodes?

Node 1 sends a request and receives a response from Node 2.
Node 1 sends a request, and the request will not reach Node 2.
Node 1 sends a request, and Node 2 receives the request and tries to process it. During processing, Node 2 becomes unavailable for Node 1. (But under the hood, Node 2 can also make requests to other nodes!)
Node 1 sends a request to Node 2. Node 2 processes the request and returns a response, but Node 1 does not receive it.

If Node 1 does not receive a response from Node 2, then it will not be able to understand what is causing the problem. It could be network problems and message loss, a Node 2 problem, or a dead link.

So that Node 1 does not wait forever for a response, you can use the mechanism of waiting (Timeout) for a response. Out of timeout - Node 1 generated a communication error.

But what should be the timeout?

If you set the timeout too low, Node 1 may mistakenly think that Node 2 is unavailable. On the other hand, if the timeout is too long, the waiting time can significantly impact system performance.

You can use the ping and heartbeat mechanisms as an alternative to timeouts.

Ping. Node 1 can ping Node 2 from time to time and check if the node is alive. Even if, at some point, the node does not respond, Node 1 can continue to send such requests in the hope that Node 2 will be available sooner or later.
Heartbeat. Node 2 can send particular messages to all nodes it communicates. Thus, it notifies the network that it is up and running. If Node 1 does not receive heartbeat messages from Node 2, it marks that node as unreachable.

Both mechanisms are used in distributed systems - e.g., for communication and synchronizing microservices on the backend.

It is only one example of how we can think about distributed systems and try to develop testing scenarios for them.

Learning about distributed systems

It is up to you to dive into distributed systems in-depth or not. Suppose you test some DeFi protocols or applications that only use a specific blockchain (Ethereum or Cardano). In that case, you need to concentrate on these protocols’ technical and business aspects first. Learning about internals adds more benefits later.

But if you are a part of a core development team or building a distributed system (or blockchain) from scratch - you must know the theoretical basis.

If you are interested in distributed systems - I want to share a few resources on where to start.

Simple explanations can be found in the introductory course created by Chris Colohan. It is a good starting point for newcomers.
If you want to read a single book - it is “Designing Data-Intensive Applications” by Martin Kleppmann. I even made a review post about this book last year.. The book can be hard to read, but it is full of helpful information.
If you prefer to watch videos - here is a course on distributed systems at the University of Cambridge taught by Martin Kleppmann

What can you learn from the course?
- what distributed systems are and how nodes can communicate with each other
- canonical theoretical tasks from two generals to Byzantine ones.
- time and how to work with it in large systems: physical and logical clocks, clock synchronization, and cause-and-effect relationship of events on different nodes
- the concept of replication and quorums, as well as message broadcasting algorithms
- consensus algorithms - from the simplest to even simpler (RAFT)
- how Google Spanner works and different tools for collaboration between users
For more in-depth explanations of consensus protocols you can watch an academic course from Lindsey Kooper

Conclusions

Distributed systems are a vast but really fascinating topic to learn. As our modern systems are getting bigger and more complex, the knowledge about distributed computing and consensus algorithms can significantly benefit you.

It is a must-have knowledge if you work with blockchain systems (at least from the theoretical part). Otherwise, you will always test your app against the blockchain as a black-box.

Such testing is not a bad thing at all - but you may miss a lot of important internal issues by concentrating only on the surface.

Test Engineering Notes

On choosing and changing languages for test automation

My story

Questions to consider before choosing a language

Project-related:

Language-related:

On environment

One tester. A small team. No prior automation experience.

One tester. A small team. Some automation experience.

A team of testers. No automation experience. You’re the lead.

On choosing languages

About new languages

On changing languages and tech stacks

On skills

Conclusion

On test engineer’s maturity

Questions

Test Engineer Maturity

Junior

Middle

Senior

Lead

Manager

Consultant

“Guru”

Conclusions

And what questions do YOU ask yourself on the project?

Testability and how to improve it

Why is it testability?

Testability: officially and not so much

Testability at the code level

Testability at the architecture level

Testability through the eyes of testers

Unique properties of elements

Components in isolation

API for system management

And how do YOU improve testability on your project?

The most complicated automated tests

What started it all?

Unit tests

Integration tests

End-to-End tests

So, which automation tests are the most complicated?

Under The Hood: How Git works?

A bit of Git theory

The structure of Git

What happens during git init?

Making a first commit

More commits

A few words about the branches

Conclusions

Blockchain for Test Engineers: Merkle Trees

Blockchain and transactions

What is Merkle Tree?

What is the meaning of the Merkle Tree?

Where you can find Merkle Trees?

High Tech - Low Test or Problems with Modern Testing

What has changed in the industry in 10 years

Problems with modern testing

The pursuit of technology

Insufficient level of skills and knowledge

Technical knowledge

Programming and architecture

Knowledge of business and product

Personal development of engineers

A vicious circle of uncertainty

How to change the situation for the better

Blockchain for Test Engineers: What is blockchain?

Why do we need a blockchain if there is already a bank?

What is blockchain?

How does it work technically?

Why store the hash of the previous block?

How user creates a new transaction

Mining new blocks

Block validation

Block propagation

What you need to consider when testing the blockchain

Blockchain for Test Engineers: P2P networks and Gossip protocol

The ways how networks can be organized

Client-Server model