Black-box testing vs White-box testing

[smol-ninja]

Flow follows the white-box testing approach.

From ChatGPT

What is white-box testing?

White-box testing requires understanding the internal structure of the code. It focuses on code paths, loops, and conditions, ensuring that all parts of the code work as expected. White-box testing is often used for unit and integration testing where in-depth knowledge of the code is necessary.

Should testing of a code be dependent on the internal logic/implementation?

No, testing of a code should not depend on the internal logic or implementation. Instead, it should focus on testing the external behavior of the code—this is called black-box testing. Tests should validate that the code meets the specified requirements and produces the expected results for given inputs, regardless of how the code is implemented internally. This makes the tests resilient to changes in the code structure, as long as the external behavior remains consistent.

What is black-box testing?

Black-box testing focuses on testing the external behavior of the software without knowledge of the internal code. It checks whether the input produces the expected output without considering the actual implementation. This method is useful for functional testing and UI testing, where you are more interested in how the system behaves rather than how it is built

Currently, all Flow tests (concrete, fuzz, invariant) follow white-box testing, which means if we change an internal implementation of a function without affecting the APIs (inputs, outputs) and expected returned values, we would also be required to change a lot of tests. IMO a good testing system should test for the behavior of the system and not rely on the internal implementation of the functions.

A good approach would be to have white-box testing in concrete whereas black-box testing in fuzz and invariant, though more knowledge is needed in this domain.

So, starting this discussion to brainstorm on these lines, gather resources, understand it better and eventually refactor our tests to use a mix of black and white box techniques.

Resources

• https://chatgpt.com/share/670e4dd3-e2e8-8000-8dd2-d372b26a5761

cc @sablier-labs/engineers as it may be relevant to Solana and other repos.

[PaulRBerg]

Interesting concept.

However, I think that the line is fuzzier (pun-intended) between white-box and black-box testing than it may seem at first blush.

Presumably, black-box testing involves writing middleware that converts the internal logic into some abstracted interfaces. So then the maintenance moves to the middlewares.

I could be wrong, though. A PR that shows how black-box testing works in practice would be helpful.

[smol-ninja]

I am closing this discussion because I have realized that, to which I also agree, its hard to distinguish between the two. It can also vary on a case-by-case basis. If tests have total coverage and there are sufficient fuzz tests, all possible scenarios should be covered anyways.

[PaulRBerg]

It was good to bring this up, anyway. We learn by explaining our theories.

[IaroslavMazur]

A good way to describe the differences between White-Box and Black-box testing approaches is whether the tester is/has to be actively thinking about the actual lines of code they're testing.

In White-Box testing, you're writing the tests 1) knowing the actual implementation and, therefore, 2) being affected by the implementation when creating the tests (which, of course, may lead to missing any test cases that have, also, been forgotten to be covered by the code you're testing).

This perspective is useful, e.g., for achieving a high code coverage - and is a better fit for unit/smaller tests.

In Black-Box testing, however, the tester doesn't care about the actual implementation that's being tested. Moreover, the tester may not know the programming language in which the product they're testing has been written in. In fact, they may not even be a programmer, at all (there are tools that allow you to create automated testing suites by building the test cases/scenarios from small pieces/blocks that do simple things - like, moving the mouse pointer to a certain position, left-/right-clicking, introducing a text input, etc.).

Black-Box testing is a good fit for bigger/more complicated tests - like, Integration, Invariant or Fuzzing. The key is to think from the perspective of the end user, interacting with the product you're testing in all possible ways you/an attacker can think of - and validating/asserting the side-effects that your interactions have produced (i.e. are the results correct or not?). This kind of testing allows you to look at the product in a "zoomed out", more high-level way - and, hopefully, catch the bugs you wouldn't have caught if you continued getting "inspired" by the code you're testing when creating the tests 😉

Therefore, both of these approaches are useful and shouldn't be disregarded.

[andreivladbrg]

the tester is/has to be actively thinking about the actual lines of code they're testing.

writing the production code and testing it without thinking about the lines

[smol-ninja]

Loved the meme Andrei, but jokes aside, I think that’s not a bad way to write tests. Of course it's assuming that tester has the knowledge of specs and what to expect from the API.

For example, imagine you have made a mistake in the source code, maybe missed an input check. When you’re writing tests without thinking about each line of the source code, you wont skip testing for that missed check. However, if you’re writing tests while thinking about each line of source code, you might also miss writing a test for that missed check.

[andreivladbrg]

yeah, i agree with the idea of thinking outside the box and not writing the tests simply by iterating through each line
my point with the meme is that we will always have biases here, since we both write the prod and test code.