[pw-ci] [RFC v3 2/2] github_get_logs: handle multiple failed step in a single job
Michael Santana
msantana at redhat.com
Tue Aug 3 18:39:00 UTC 2021
On Tue, Aug 3, 2021 at 12:59 PM Salvatore Daniele <sdaniele at redhat.com> wrote:
>
>
>
> On Mon, Aug 2, 2021 at 5:42 PM Salvatore Daniele <sdaniele at redhat.com> wrote:
>>
>>
>>
>> On Mon, Aug 2, 2021 at 5:16 PM Michael Santana <msantana at redhat.com> wrote:
>>>
>>> On Thu, Jul 29, 2021 at 5:36 PM Salvatore Daniele <sdaniele at redhat.com> wrote:
>>> >
>>> > A github job can fail multiple steps within a single job.
>>> > The current implementation fails to pull logs when handed a
>>> > job with multiple failed steps.
>>> >
>>> > This patch generates logs for each step of each job.
>>> >
>>> > Signed-off-by: Salvatore Daniele <sdaniele at redhat.com>
>>> > ---
>>> > github_get_logs.sh | 9 +++++----
>>> > 1 file changed, 5 insertions(+), 4 deletions(-)
>>> >
>>> > diff --git a/github_get_logs.sh b/github_get_logs.sh
>>> > index 2f6cc9c..1cd4086 100755
>>> > --- a/github_get_logs.sh
>>> > +++ b/github_get_logs.sh
>>> > @@ -99,26 +99,27 @@ print_errored_logs_for_commit () {
>>> > echo "-----------------------Summary of failed steps-----------------------"
>>> > echo "$jobs_results" | jq -r ".[].name" | while read -r job; do
>>> > echo "\"$job\" failed at step \"$(echo "$jobs_results" | jq -r ".[] | \
>>> > - select(.name==\"$job\") | .failed_step.name")\""
>>> > + select(.name==\"$job\") | .failed_step.name")\"" | sed 'N;s/\n/, /'
>>> jq does not have a built-in way to make a comma separated list? if not, it's ok.
>>>
>> I didn't look very hard when I saw that sed worked, but I will double check this. That would probably be cleaner.
>>
>>> I thought sed could not replace new lines. Is that what the N flag is for?
>>
>>
>> Yes, it reads the next line into pattern space.
>
>
> To follow up, WDYT of this as output for this loop?
>
> -----------------------Summary of failed steps-----------------------
> "e2e (control-plane, true, true, true, ipv4, IPv4, true, false)" failed at step Run Tests
> "e2e (control-plane, true, true, true, ipv4, IPv4, true, false)" failed at step Generate Test Report
> "e2e (control-plane, true, true, true, ipv4, IPv4, true, false)" failed at step Third step
> "other workflow" failed at step other step
> ----------------------End summary of failed steps--------------------
>
> This could be achieved by the following which looks cleaner IMO.
>
> echo "-----------------------Summary of failed steps-----------------------"
> echo "$jobs_results" | jq -r ".[] | .name, .failed_step.name " | while \
> read -r job && read -r step; do
> echo "\"$job\" failed at step $step"
> done
> echo "----------------------End summary of failed steps--------------------"
>
> My current fix does not work for more than 2 failed steps, and I could not find a clean way to replace newlines with commas in jq. Tr only allows replacing 1 character with 1.
So wait, what's the issue now? I thought you got it working
>
> Open to other thoughts. Two sed pipes could work.
>
>>
>>
>>> > done
>>> > echo "----------------------End summary of failed steps--------------------"
>>> >
>>> > echo ""
>>> > echo "-------------------------------BEGIN LOGS----------------------------"
>>> > spacing=0
>>> > +
>>> > # Print out logs for failed jobs
>>> > - echo "$jobs_results" | jq -r ".[].name" | while read -r job; do
>>> > + echo "$jobs_results" | jq -r '.[] | .name, .failed_step.name, .failed_step.number'| while \
>>> > + read -r job && read -r step && read -r log_number; do
>>> > +
>>> Just nit-picking:
>>> why switch from " to ' ?
>>> add a space between ' and |
>>> remove 4 empty trailing spaces
>>
>>
>> Yes will do!
>>
>>>
>>>
>>>
>>> One question, do the arguments come out one on each line from jq? is
>>> that why you had to call read separately multiple times?
>>
>>
>> Yes, if I recall correctly when I put them all in 1 read statement it would read 1 work into each (so the first 3 words of the job title).
>> If I tried "while IFS="\n" read -r job read step" it would read only into job each time, leaving the other two variables empty.
>> Perhaps there is a cleaner way to get this while loop to behave, but I was struggling to find it.
>>
>>>
>>> > if [ ! "$spacing" -eq "0" ]
>>> > then
>>> > echo -ne "\n\n\n\n"
>>> > fi
>>> >
>>> > - step="\"$(echo "$jobs_results" | jq -r ".[] | select(.name==\"$job\") | .failed_step.name")\""
>>> > echo "####################################################################################"
>>> > echo "#### [Begin job log] \"$job\" at step $step"
>>> > echo "####################################################################################"
>>> >
>>> > - log_number=$(echo "$jobs_results" | jq ".[] | select(.name==\"$job\") | .failed_step.number")
>>> > cat "build_logs_series_$series_id/$job/$log_number"_* | tail -n 25 | cut -d' ' -f2- | sed 's/\r$//'
>>> >
>>> > echo "####################################################################################"
>>> > --
>>> > 2.31.1
>>> >
>>> > _______________________________________________
>>> > Pwci mailing list
>>> > Pwci at lists.esnacc.org
>>> > http://mail.esnacc.org/mailman/listinfo/pwci
>>> >
>>>
More information about the Pwci
mailing list