Startup challenges: Ensuring the quality with Appium

2 мин на чтение

Our backend team writes Behavior Driven Tests with Behave to check their integration with external clients.

But since these tests can only emulate external clients (like mobile apps), they didn’t check end-to-end integration.

We thought that having an honest end-to-end check will guarantee the correctness of the basic product scenarios.

Appium is easy to set up and use. We piloted it in conjunction with Sauce Labs for launching and recording test results.

We assumed that any team/role (backend dev, mobile dev, QA engineer, etc.) could use a universal tool like Appium. We hoped the backend team would primarily use them as they are familiar with the Python language.

Experiment results

We integrated Appium tests into CI/CD pipelines of the backend and mobile apps. We piloted the solution for half a year. Through this time, it caught few major bugs.

Tests writing experience

After setting up some test infrastructure, they were easy to write.

Here’s the excerpt from one of the tests:

@dataclass(frozen=True)
class PreviewAndShareScreen(CloseableScreen):
    share_button: TestObject

    def assert_has_share_button(self):
        self.app.wait_for(self.share_button)

    def share(self):
        if not self.app.is_ios:
            self.app.wait_and_click(self.share_button)

    @staticmethod
    def create(app: App):
        if app.is_ios:
            return PreviewAndShareScreen(
                app=app,
                share_button=ios_button_by_text('Share')
            )
        return PreviewAndShareScreen(
            app=app,
            share_button=obj_by_text_and_id('Share', 'com.anna.money.app.dev:id/buttonPrimary')
        )


class TestAccountStatement:
    @pytest.mark.regress
    @pytest.mark.statement
    def test_account_statement_share(self, registered_user_app: App):
        preview_screen = PreviewAndShareScreen.create(registered_user_app)
        system_popups = SystemPopups.create(registered_user_app)
        chat_screen = ChatScreen.create(registered_user_app)

        # wait for suggest to appear
        chat_screen.get_suggest('Create an invoice')
        chat_screen.send_text('Account statement')
        chat_screen.click_suggest('Full history')

        def check_sharing():
            if registered_user_app.is_ios:
                preview_screen.assert_has_share_button()
            else:
                preview_screen.share()
                system_popups.wait_for_share_popup()
                # cancel native share popup
                system_popups.cancel()

            preview_screen.close()

        # check PDF
        account_statement_card = T5.create(registered_user_app)
        account_statement_card.click_second_button('PDF')

        check_sharing()

        # check CSV

        account_statement_card.click_first_button('CSV')
        check_sharing()

As you can see, it required platform-specific queries. So it could take quite a while to write a new test completely since you need to check it on Android and iOS separately.

False-positive results

Since our backend utilizes a microservice architecture, it has a lot of moving parts. It means every time this and that may get broken. It resulted in unstable test results.

Though we provided quite convenient tools to track down the problem, it was quite hard to find the source of the problem. Most of the time, the tests were false-positive. It made Appium tests noisy. After some time, everybody lost attention to the alerts.

No integration in the process

We failed to integrate Appium tests in the development processes of teams.

The Backend team was more familiar with Behave, so they didn’t see a lot of benefit in using Appium. They were not ready to invest more time in Appium infrastructure, as they were fine with Behave tests.

Summary

Since we didn’t get much interest in this tool from developers, we decided to close the experiment.

Teams are more effective with platform-specific tools, like Behave, XCTest, Espresso, etc.