[pw-ci] [PATCH] github: alternate url requests

[Aaron Conole]

Michael Santana <msantana at redhat.com> writes:

> Hi Aaron,
> Thanks for working on this. See comments below
>
> On Tue, Apr 12, 2022 at 5:13 PM Aaron Conole <aconole at redhat.com> wrote:
>>
>> Github is currently broken when requesting filtered results for the
>> github actions runs.  While this stays broken, apply the following which
>> works around the issue by filtering on the client side.
>>
>> Plan is to remove when
>> https://github.community/t/no-longer-able-to-list-in-progress-builds-via-the-api/244175/3
>> and
>> https://github.community/t/workflow-call-has-an-inconsistent-status-in-github-rest-api/244164/2
>> are resolved.
>>
>> Signed-off-by: Aaron Conole <aconole at redhat.com>
>> ---
>> diff --git a/github_get_logs.sh b/github_get_logs.sh
>> index a646b47..9f44840 100755
>> --- a/github_get_logs.sh
>> +++ b/github_get_logs.sh
>> @@ -68,7 +68,7 @@ print_errored_logs_for_commit () {
>>      # Get run metadata
>>      select="select(.head_branch==\"series_${series_id}\") |
>> select(.head_sha==\"${sha}\") | select(.name==\"${test_name}\")"
>>      headers="{id, logs_url}"
>> -    run_meta="$(echo "$runs" | jq ".workflow_runs[] | $select | $headers")"
>> +    run_meta="$(echo "$runs" | jq "$select | $headers")"
>>
>>      redirect_url="$(echo "$run_meta" | jq -r ".logs_url")"
>>      run_id="$(echo "$run_meta" | jq -r ".id")"
>> @@ -88,7 +88,7 @@ print_errored_logs_for_commit () {
>>
>>      length="$(echo "$jobs_results" | jq -r "length")"
>>
>> -    if [ "$length" -eq "0" ]; then
>> +    if [ "X$length" == "X" -o "X$length" == "X0" ]; then
> is the original not working?
>>          echo "No build failures detected for series_${series_id}/${sha}. Exiting" 1>&2
>>          return 0
>>      fi
>> @@ -134,8 +134,9 @@ print_errored_logs_for_commit () {
>>  repo_name=$(echo "$repo_name" | sed -e 's@%2F@/@g' -e 's,%40,@,g')
>>
>>  # Get GHA runs
>> -tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?branch=series_$series_id&per_page=100"
>> -runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
>> +tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?per_page=9000"
> So at the time this was written I thought I had read that the max per
> page you could request was 100. Is this not the case anymore? You can
> instruct jq to print the size of the array/struct you get back to find
> out

That isn't the case now - additionally, if we don't specify the per_page
value, we get a significantly reduced set in worker_runs[] array and are
unable to find all the series we need.

> I guess in reality it doesnt matter much. We probably dont have more
> than 100 runs going on at the same time. Going over 100 would really
> only be useful we want to go back to get the status on old builds past
> 100. Maybe it has been changed? let me know

This gives the total runs associated with the system, so I don't know
about 'simultaneously' or 'parallel' anything.

>> +all_runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
>> +runs=$(echo $all_runs | jq -rc ".workflow_runs[] | select(.head_branch == \"series_$series_id\")")
> I would add quotes. I need to practice my quoting on nested shell
> scripts. According to shellcheck I have been doing them wrong this
> entire time

Why?  There are probably quotes where they don't need to exist, tbh.

> just quotes around all_runs is fine

ACK

>>  not_found="$(echo "$runs" | jq -rc ".message")"
>>  if [ "$not_found" == "Not Found" ]; then
>>      echo "\"$tmp_url\" could not be reached." 1>&2
>> diff --git a/github_mon b/github_mon
>> index 42478be..8670ce8 100755
>> --- a/github_mon
>> +++ b/github_mon
>> @@ -60,12 +60,12 @@ make_result_for_series () {
>>      repo_name="$6"
>>
>>      # Get the name of each respective workflow
>> -    workflows="$(echo "$runs" | jq -r ".workflow_runs[].name" | sort -u)"
>> +    workflows="$(echo "$runs" | jq -r ".name" | sort -u)"
>>
>>      # Check that workflows have all completed
>>      select="select(.head_branch==\"series_${series_id}\") | select(.head_sha==\"${sha}\")"
>>      headers="{status, conclusion, html_url}"
>> -    run_meta="$(echo "$runs" | jq ".workflow_runs[] | $select | $headers")"
>> +    run_meta="$(echo "$runs" | jq "$select | $headers")"
>>
>>      if [ "$(echo "$run_meta" | jq -r ".status" | sort -u)" != "completed" ]; then
>>          echo "patch_id=$patch_id belonging to series_id=$series_id not completed. Skipping" 1>&2
>> @@ -78,7 +78,7 @@ make_result_for_series () {
>>          # Get run metadata
>>          select="select(.head_branch==\"series_${series_id}\") | select(.head_sha==\"${sha}\") | select(.name==\"${WORKFLOWNAME}\")"
>>          headers="{status, conclusion, html_url}"
>> -        run_meta="$(echo "$runs" | jq ".workflow_runs[] | $select | $headers")"
>> +        run_meta="$(echo "$runs" | jq "$select | $headers")"
>>
>>          # success/failure?
>>          result="$(echo "$run_meta" | jq -r ".conclusion")"
>> @@ -100,6 +100,9 @@ make_result_for_series () {
>>
>>  prev_series=""
>>
>> +prev_url=""
>> +all_runs=""
>> +
>>  get_unsynced_series "$pw_instance" "$ci_instance" | \
>>  while IFS="|" read -r series_id patch_id patch_url patch_name sha patchwork_instance patchwork_project repo_name gap_sync; do
>>
>> @@ -111,8 +114,13 @@ while IFS="|" read -r series_id patch_id patch_url patch_name sha patchwork_inst
>>          prev_series="$series_id"
>>
>>          # Get GHA runs
>> -        tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?branch=series_$series_id&per_page=100"
>> -        runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
>> +        tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?per_page=9000"
>> +        if [ "$tmp_url" != "$prev_url" ]; then
>> +            prev_url="$tmp_url"
>> +            all_runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
>> +        fi
>> +
>> +        runs=$(echo "$all_runs" | jq -rc ".workflow_runs[] | select(.head_branch == \"series_$series_id\")")
>>          not_found="$(echo "$runs" | jq -rc ".message")"
>>      fi
>>
>> ---
>> 2.31.1
>>