AssertionError when running e2e tests

[samuelfangjw]

Commit SHA: eb6f5db
Operating System: macOS Big Sur (11.4)
JDK Version: openjdk64-1.8.0.292
Node.js Version: v14.17.0
Google Cloud SDK Version: 296.0.0
Firefox / Chrome Version (if applicable): 89.0.2 (Firefox) / 91.0.4472.114 (Chrome)

test.properties (in e2e tests folder with comments removed)

test.app.url=http\://localhost\:4200
test.csrf.key=samplekey
test.backdoor.key=samplekey

test.selenium.browser=firefox

test.browser.closeonfailure=true
test.firefox.path=

test.geckodriver.path=/usr/local/Cellar/geckodriver/0.29.1/bin/geckodriver
test.chromedriver.path=/usr/local/bin/chromedriver

test.search.active=false

test.timeout=15

test.email=alice.tmms@gmail.tmt
test.senderemail=admin@teammates-john.appspotmail.com
test.verify.emails=true

What I was trying to do / What I need help with

I'm trying to run e2e tests on the latest master branch but have been facing this error. (full stacktrace here)

Failed suite [Failed suite [Failed suite [Failed suite [e2e-tests]]]] > e2e-tests(failed)(failed)(failed)(failed) > teammates.e2e.cases.FeedbackMcqQuestionE2ETest > baseClassSetup FAILED
    java.lang.AssertionError at FeedbackMcqQuestionE2ETest.java:25

Failed suite [Failed suite [Failed suite [Failed suite [e2e-tests]]]] > e2e-tests(failed)(failed)(failed)(failed) > teammates.e2e.cases.StudentHomePageE2ETest > baseClassSetup FAILED
    java.lang.AssertionError at StudentHomePageE2ETest.java:22

These assertions errors point to the bolded line below

    @Override
    protected void prepareTestData() {
        testData = loadDataBundle("/NameOfTest.json");
        removeAndRestoreDataBundle(testData);
        // other lines of code, differs test to test
    }

Going into the removeAndRestoreDataBundle, I get

    protected void removeAndRestoreDataBundle(DataBundle testData) {
        int retryLimit = OPERATION_RETRY_COUNT;
        boolean isOperationSuccess = doRemoveAndRestoreDataBundle(testData);
        while (!isOperationSuccess && retryLimit > 0) {
            retryLimit--;
            print("Re-trying removeAndRestoreDataBundle");
            ThreadHelper.waitFor(OPERATION_RETRY_DELAY_IN_MS);
            isOperationSuccess = doRemoveAndRestoreDataBundle(testData);
        }
        assertTrue(isOperationSuccess);
    }

I believe the assertion error is thrown from this function (i.e. "isOperationSuccess" is false).

Additional Information
Setup seems to be partially working. When running tests on firefox or chrome, the browser pops up and the browser window is manipulated.

What I have tried so far
Debugging: I've had issues using debugger in IntelliJ to stop at breakpoints when running e2e tests. Doesn't seem to work for me for some reason. Printing out values using System.out.println also doesn't seem to work. Manually tracing the route didn't give me any clues.

New Config: I've downloaded (as zip file) the latest master and setup the repo again according to the instructions in the docs.

Browser: I've tried using both firefox and chrome.

Not very sure how to approach this issue, any help or advice greatly appreciated!

[wkurniawan07]

test.app.url=http\://localhost\:4200

E2E tests don't operate in Angular dev server.

[samuelfangjw]

Hi, thanks for the reply. Does this mean I have to test against backend server (which gives me another set of errors) or does it mean I have to test against production server.

[wkurniawan07]

E2E tests can be run in any server that runs the Java back-end and built Angular front-end files at the same time.

[samuelfangjw]

I see, thanks for the help!

[samuelfangjw]

Finally solved my issues! Putting this here in case anyone can benefit from it.

Was confused because I was getting a 403 error while accessing localhost:8080. This should in fact redirect you to the home page of teammates (TIL!). Running e2e tests on localhost:8080 also threw errors for each test case (probably because there was no UI to interact with). Not very sure why the error was occurring, but restarting my computer solved the issues (i'm assuming because restarting killed all background processes).

Steps I took to fix and successfully run e2e tests:

1. Restart computer
2. From root folder, I ran the following commands
   • ./gradlew runDatastoreEmulator
   • ./gradlew serverRun
   • ./gradlew e2eTests

[nirikawa]

Hey I think I am having the same problem. I get a 403 error when I go to localhost:8080. What are you supposed to get?

I have tried many things in addition to restarting with no luck. Do you have any idea besides restarting computer to get it working right?

[samuelfangjw]

Hi @nirikawa, ensure that you have built the frontend using npm run build.

[samuelfangjw]

You are supposed to see the teammates home page when you go to localhost:8080

[nirikawa]

Hi @nirikawa, ensure that you have built the frontend using npm run build.

@samuelfangjw That was exactly it. Thanks!