[pw-ci] [PATCH 2/2] github_get_logs: handle multiple failed step in a single job

[Salvatore Daniele]

On Wed, Aug 4, 2021 at 10:55 AM Aaron Conole <aconole at redhat.com> wrote:

> Salvatore Daniele <sdaniele at redhat.com> writes:
>
> > A github job can fail multiple steps within a single job.
> > The current implementation fails to pull logs when handed a
> > job with multiple failed steps.
> >
> > This patch generates logs for each step of each job.
> >
> > Signed-off-by: Salvatore Daniele <sdaniele at redhat.com>
> > ---
> >  github_get_logs.sh | 13 +++++++------
> >  1 file changed, 7 insertions(+), 6 deletions(-)
> >
> > diff --git a/github_get_logs.sh b/github_get_logs.sh
> > index 2f6cc9c..99b60a1 100755
> > --- a/github_get_logs.sh
> > +++ b/github_get_logs.sh
> > @@ -97,28 +97,29 @@ print_errored_logs_for_commit () {
> >
> >      # Print out which jobs failed
> >      echo "-----------------------Summary of failed
> steps-----------------------"
> > -    echo "$jobs_results" | jq -r ".[].name" | while read -r job; do
> > -        echo "\"$job\" failed at step \"$(echo "$jobs_results" | jq -r
> ".[] | \
> > -            select(.name==\"$job\") | .failed_step.name")\""
> > +        echo "$jobs_results" | jq -r ".[] | .name, .failed_step.name "
> | while \
> > +            read -r job && read -r step; do
>
> Why this:
>
>   read .. && read ..
>
> Can't it just be:
>
>   while IFS=',' read -r job step; do
>      ...
>   done
>
> Did I miss something?
>

So perhaps I am using IFS wrong. I did try a number of things like that,

E.g.
jobs_results='[{
        "name": "e2e (control-plane, true, true, true, ipv4, IPv4, true,
false)",
        "failed_step": {
                "name": "Run Tests",
                "conclusion": "failure",
                "number": 11
        }
}, {
        "name": "e2e (control-plane, true, true, true, ipv4, IPv4, true,
false)",
        "failed_step": {
                "name": "Generate Test Report",
                "conclusion": "failure",
                "number": 13
        }
}]'

echo "$jobs_results" | jq -r ".[] | .name, .failed_step.name " | while
IFS=',' read -r job step; do
    echo "job = \"$job\" and step = \"$step\""
done

Yields the output:

job = e2e (control-plane and step =  true, true, true, ipv4, IPv4, true,
false)
job = Run Tests and step =
job = e2e (control-plane and step =  true, true, true, ipv4, IPv4, true,
false)
job = Generate Test Report and step =

while IFS=$'\n' read -r job step; do

Yields

job = "e2e (control-plane, true, true, true, ipv4, IPv4, true, false)" and
step = ""
job = "Run Tests" and step = ""
job = "e2e (control-plane, true, true, true, ipv4, IPv4, true, false)" and
step = ""
job = "Generate Test Report" and step = ""

There must be some way to make it work since there is a similar loop to
what you're describing in "ci_mon" but I was
certainly struggling to find a cleaner solution that just multiple read
statements.



> > +        echo "\"$job\" failed at step $step"
> >      done
> >      echo "----------------------End summary of failed
> steps--------------------"
> >
> >      echo ""
> >      echo "-------------------------------BEGIN
> LOGS----------------------------"
> >      spacing=0
> > +
> >      # Print out logs for failed jobs
> > -    echo "$jobs_results" | jq -r ".[].name" | while read -r job; do
> > +    echo "$jobs_results" | jq -r ".[] | .name, .failed_step.name,
> .failed_step.number" | while \
> > +        read -r job && read -r step && read -r log_number; do
> > +
>
> Same as above.
>
> Do we still have a 2-job issue with this?  Can you send a link to an
> example job?
>

Yes, this was the example I was working on when the bug came up [1].
Michael pointed out that the ovn-kubernetes workflow often has two steps
within the same job (specifically run tests and generate test report) fail,
which was causing the following cat statement to fail, since $step and
$log_number would be equal to:

"Run Tests
Generate Test Report"

"11
13"

[1] https://github.com/ovsrobot/ovn/runs/3137144963?check_suite_focus=true
https://patchwork.ozlabs.org/api/patches/1508829/


> >          if [ ! "$spacing" -eq "0" ]
> >          then
> >              echo -ne "\n\n\n\n"
> >          fi
> >
> > -        step="\"$(echo "$jobs_results" | jq -r ".[] |
> select(.name==\"$job\") | .failed_step.name")\""
> >          echo
> "####################################################################################"
> >          echo "#### [Begin job log] \"$job\" at step $step"
> >          echo
> "####################################################################################"
> >
> > -        log_number=$(echo "$jobs_results" | jq ".[] |
> select(.name==\"$job\") | .failed_step.number")
> >          cat "build_logs_series_$series_id/$job/$log_number"_* | tail -n
> 25 | cut -d' ' -f2- | sed 's/\r$//'
> >
> >          echo
> "####################################################################################"
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.esnacc.org/pipermail/pwci/attachments/20210804/b9ac3b3c/attachment.html>