Fix flaky of test_cancel_launch_and_exec_async

[zpoint]

Resolve #5436

The issue only occurs in the UP state. If we call sky down then, the job won't show as cancelled, and the grep "cancelled" command will fail.

When it's in INIT state, the test succeeds. This failure is more likely to happen in kubernetes tests because the kubernetes cluster provisions too quickly to reach the UP state.

I'm not sure if fixing the test case is the right approach, or if we should modify the sky job system to also cancel jobs for clusters in the UP state when sky down called? @aylei

Tested (run the relevant ones):

• Code formatting: install pre-commit (auto-check on commit) or bash format.sh
• Relevant individual tests: /smoke-test --kubernetes -k test_cancel_launch_and_exec_async (CI) or pytest tests/test_smoke.py::test_name (local)

[zpoint]

/smoke-test --kubernetes -k test_cancel_launch_and_exec_async

[zpoint]

/smoke-test --kubernetes -k test_cancel_launch_and_exec_async

[cg505]

I'm not sure if fixing the test case is the right approach, or if we should modify the sky job system to also cancel jobs for clusters in the UP state when sky down called?

@zpoint It's possible that this was the intended behavior, but I don't know. Can we confirm this behavior in the past (e.g. in 0.8)? Just want to double check that this isn't actually a regression.

[cg505]

otherwise, code changes look good

[zpoint]

This test was added by @aylei in #4867 after the client-server architecture change. Confirmed with aylei that INIT state jobs being canceled is the intended behavior.

Since he's OOO this week, we should be good to go with this PR if its a blocker.

@cg505

[zpoint]

0.8.x does not have this test.

[cg505]

sounds good, thanks for looking into it!

[zpoint]

/smoke-test --aws -k test_cancel_launch_and_exec_async

[zpoint]

/smoke-test --gcp -k test_cancel_launch_and_exec_async

[zpoint]

/smoke-test --azure -k test_cancel_launch_and_exec_async