gitlab-ci for test jobs

[Ross Vandegrift]

Hi everyone,

First complete pass at a framework for running tests is here:
https://salsa.debian.org/cloud-team/debian-cloud-images/merge_requests/36
It's working, you can see the last run at:
https://salsa.debian.org/rvandegrift-guest/debian-cloud-images/pipelines/21750

This is the simplest config I could find that has a separate test job for each
build job. It's complex and verbose. Makes it hard to know that all of the
cases are covered.

One alternative: extend the .build job scipt to run tests after FAI. The
benefit: a new build job automatically gets tests run without manually creating
a new job. The downside: if a job fails, you have to dig through logs to
determine if the build failed or if the tests failed.

If we go with separate jobs, we could add a CI-linting stage before the build.
This could check the pipline to ensure e.g., every build has a test, only
official builds get run on casulana, etc.

Any thoughts on the tradeoff, or maybe a better idea?

Ross

[Bastian Blank]

Hi Ross

Post by Ross Vandegrift
https://salsa.debian.org/cloud-team/debian-cloud-images/merge_requests/36
https://salsa.debian.org/rvandegrift-guest/debian-cloud-images/pipelines/21750

Yeah. There are nice echo calls. Will take a look at the code changes later.

Post by Ross Vandegrift
This is the simplest config I could find that has a separate test job for each
build job. It's complex and verbose. Makes it hard to know that all of the
cases are covered.

Yeah, I know that Gitlab does not really help for such cases.

Post by Ross Vandegrift
One alternative: extend the .build job scipt to run tests after FAI. The
benefit: a new build job automatically gets tests run without manually creating
a new job. The downside: if a job fails, you have to dig through logs to
determine if the build failed or if the tests failed.

Please don't. It also makes it harder to get results from the
artifacts.

Post by Ross Vandegrift
If we go with separate jobs, we could add a CI-linting stage before the build.
This could check the pipline to ensure e.g., every build has a test, only
official builds get run on casulana, etc.

We could go for a generated file approach. Sure, you need to be careful
then, but I think the advantages are greater. It would also avoid some
gitlab-runner limitations, like non-recursive variable expansion and
allow adding the git-sha1 to the dev build version.

Bastian

[Ross Vandegrift]

Post by Bastian Blank

Post by Ross Vandegrift
https://salsa.debian.org/cloud-team/debian-cloud-images/merge_requests/36
https://salsa.debian.org/rvandegrift-guest/debian-cloud-images/pipelines/21750

Yeah. There are nice echo calls. Will take a look at the code changes later.

Thanks for your comments - I haven't had time to address them yet, but I will.

Post by Bastian Blank
We could go for a generated file approach. Sure, you need to be careful
then, but I think the advantages are greater. It would also avoid some
gitlab-runner limitations, like non-recursive variable expansion and
allow adding the git-sha1 to the dev build version.

That sounds promising, but I've never done it. Since gitlab wants the pipeline
config commited in the repo, I guess we'd need to generate, commit, push? Do
you have a pattern for this approach that you like?

Thanks,
Ross

[Bastian Blank]

Post by Ross Vandegrift

Post by Bastian Blank
We could go for a generated file approach. Sure, you need to be careful
then, but I think the advantages are greater. It would also avoid some
gitlab-runner limitations, like non-recursive variable expansion and
allow adding the git-sha1 to the dev build version.

That sounds promising, but I've never done it. Since gitlab wants the pipeline
config commited in the repo, I guess we'd need to generate, commit, push? Do
you have a pattern for this approach that you like?

Neither did I. Yes, we need to commit the generated files and push
them. Not yet.

Do you have any idea when you might be able to run first tests in the CI?

Bastian

[Ross Vandegrift]

Hi Bastian,

Post by Bastian Blank

Post by Ross Vandegrift
That sounds promising, but I've never done it. Since gitlab wants the pipeline
config commited in the repo, I guess we'd need to generate, commit, push? Do
you have a pattern for this approach that you like?

Neither did I. Yes, we need to commit the generated files and push
them. Not yet.

I looked around for ideas today. This hack looks like a nice idea:
https://gitlab.com/gitlab-org/gitlab-ce/issues/45828#note_98607487

Post by Bastian Blank
Do you have any idea when you might be able to run first tests in the CI?

Sorry, I've been swamped. Hoping to make progress this weekend.

Ross