But since these tests can only emulate external clients (like mobile apps), they didn’t check end-to-end integration.
We thought that having an honest end-to-end check will guarantee the correctness of the basic product scenarios.
We assumed that any team/role (backend dev, mobile dev, QA engineer, etc.) could use a universal tool like Appium. We hoped the backend team would primarily use them as they are familiar with the Python language.
We integrated Appium tests into CI/CD pipelines of the backend and mobile apps. We piloted the solution for half a year. Through this time, it caught few major bugs.
Tests writing experience
After setting up some test infrastructure, they were easy to write.
Here’s the excerpt from one of the tests:
@dataclass(frozen=True) class PreviewAndShareScreen(CloseableScreen): share_button: TestObject def assert_has_share_button(self): self.app.wait_for(self.share_button) def share(self): if not self.app.is_ios: self.app.wait_and_click(self.share_button) @staticmethod def create(app: App): if app.is_ios: return PreviewAndShareScreen( app=app, share_button=ios_button_by_text('Share') ) return PreviewAndShareScreen( app=app, share_button=obj_by_text_and_id('Share', 'com.anna.money.app.dev:id/buttonPrimary') ) class TestAccountStatement: @pytest.mark.regress @pytest.mark.statement def test_account_statement_share(self, registered_user_app: App): preview_screen = PreviewAndShareScreen.create(registered_user_app) system_popups = SystemPopups.create(registered_user_app) chat_screen = ChatScreen.create(registered_user_app) # wait for suggest to appear chat_screen.get_suggest('Create an invoice') chat_screen.send_text('Account statement') chat_screen.click_suggest('Full history') def check_sharing(): if registered_user_app.is_ios: preview_screen.assert_has_share_button() else: preview_screen.share() system_popups.wait_for_share_popup() # cancel native share popup system_popups.cancel() preview_screen.close() # check PDF account_statement_card = T5.create(registered_user_app) account_statement_card.click_second_button('PDF') check_sharing() # check CSV account_statement_card.click_first_button('CSV') check_sharing()
As you can see, it required platform-specific queries. So it could take quite a while to write a new test completely since you need to check it on Android and iOS separately.
Since our backend utilizes a microservice architecture, it has a lot of moving parts. It means every time this and that may get broken. It resulted in unstable test results.
Though we provided quite convenient tools to track down the problem, it was quite hard to find the source of the problem. Most of the time, the tests were false-positive. It made Appium tests noisy. After some time, everybody lost attention to the alerts.
No integration in the process
We failed to integrate Appium tests in the development processes of teams.
The Backend team was more familiar with Behave, so they didn’t see a lot of benefit in using Appium. They were not ready to invest more time in Appium infrastructure, as they were fine with Behave tests.
Since we didn’t get much interest in this tool from developers, we decided to close the experiment.
Teams are more effective with platform-specific tools, like Behave, XCTest, Espresso, etc.