[pw-ci] [PATCH] github: alternate url requests

[Michael Santana]

Hi Aaron,
Thanks for working on this. See comments below

On Tue, Apr 12, 2022 at 5:13 PM Aaron Conole <aconole at redhat.com> wrote:
>
> Github is currently broken when requesting filtered results for the
> github actions runs.  While this stays broken, apply the following which
> works around the issue by filtering on the client side.
>
> Plan is to remove when
> https://github.community/t/no-longer-able-to-list-in-progress-builds-via-the-api/244175/3
> and
> https://github.community/t/workflow-call-has-an-inconsistent-status-in-github-rest-api/244164/2
> are resolved.
>
> Signed-off-by: Aaron Conole <aconole at redhat.com>
> ---
> diff --git a/github_get_logs.sh b/github_get_logs.sh
> index a646b47..9f44840 100755
> --- a/github_get_logs.sh
> +++ b/github_get_logs.sh
> @@ -68,7 +68,7 @@ print_errored_logs_for_commit () {
>      # Get run metadata
>      select="select(.head_branch==\"series_${series_id}\") | select(.head_sha==\"${sha}\") | select(.name==\"${test_name}\")"
>      headers="{id, logs_url}"
> -    run_meta="$(echo "$runs" | jq ".workflow_runs[] | $select | $headers")"
> +    run_meta="$(echo "$runs" | jq "$select | $headers")"
>
>      redirect_url="$(echo "$run_meta" | jq -r ".logs_url")"
>      run_id="$(echo "$run_meta" | jq -r ".id")"
> @@ -88,7 +88,7 @@ print_errored_logs_for_commit () {
>
>      length="$(echo "$jobs_results" | jq -r "length")"
>
> -    if [ "$length" -eq "0" ]; then
> +    if [ "X$length" == "X" -o "X$length" == "X0" ]; then
is the original not working?
>          echo "No build failures detected for series_${series_id}/${sha}. Exiting" 1>&2
>          return 0
>      fi
> @@ -134,8 +134,9 @@ print_errored_logs_for_commit () {
>  repo_name=$(echo "$repo_name" | sed -e 's@%2F@/@g' -e 's,%40,@,g')
>
>  # Get GHA runs
> -tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?branch=series_$series_id&per_page=100"
> -runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
> +tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?per_page=9000"
So at the time this was written I thought I had read that the max per
page you could request was 100. Is this not the case anymore? You can
instruct jq to print the size of the array/struct you get back to find
out

I guess in reality it doesnt matter much. We probably dont have more
than 100 runs going on at the same time. Going over 100 would really
only be useful we want to go back to get the status on old builds past
100. Maybe it has been changed? let me know

> +all_runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
> +runs=$(echo $all_runs | jq -rc ".workflow_runs[] | select(.head_branch == \"series_$series_id\")")
I would add quotes. I need to practice my quoting on nested shell
scripts. According to shellcheck I have been doing them wrong this
entire time

just quotes around all_runs is fine
>  not_found="$(echo "$runs" | jq -rc ".message")"
>  if [ "$not_found" == "Not Found" ]; then
>      echo "\"$tmp_url\" could not be reached." 1>&2
> diff --git a/github_mon b/github_mon
> index 42478be..8670ce8 100755
> --- a/github_mon
> +++ b/github_mon
> @@ -60,12 +60,12 @@ make_result_for_series () {
>      repo_name="$6"
>
>      # Get the name of each respective workflow
> -    workflows="$(echo "$runs" | jq -r ".workflow_runs[].name" | sort -u)"
> +    workflows="$(echo "$runs" | jq -r ".name" | sort -u)"
>
>      # Check that workflows have all completed
>      select="select(.head_branch==\"series_${series_id}\") | select(.head_sha==\"${sha}\")"
>      headers="{status, conclusion, html_url}"
> -    run_meta="$(echo "$runs" | jq ".workflow_runs[] | $select | $headers")"
> +    run_meta="$(echo "$runs" | jq "$select | $headers")"
>
>      if [ "$(echo "$run_meta" | jq -r ".status" | sort -u)" != "completed" ]; then
>          echo "patch_id=$patch_id belonging to series_id=$series_id not completed. Skipping" 1>&2
> @@ -78,7 +78,7 @@ make_result_for_series () {
>          # Get run metadata
>          select="select(.head_branch==\"series_${series_id}\") | select(.head_sha==\"${sha}\") | select(.name==\"${WORKFLOWNAME}\")"
>          headers="{status, conclusion, html_url}"
> -        run_meta="$(echo "$runs" | jq ".workflow_runs[] | $select | $headers")"
> +        run_meta="$(echo "$runs" | jq "$select | $headers")"
>
>          # success/failure?
>          result="$(echo "$run_meta" | jq -r ".conclusion")"
> @@ -100,6 +100,9 @@ make_result_for_series () {
>
>  prev_series=""
>
> +prev_url=""
> +all_runs=""
> +
>  get_unsynced_series "$pw_instance" "$ci_instance" | \
>  while IFS="|" read -r series_id patch_id patch_url patch_name sha patchwork_instance patchwork_project repo_name gap_sync; do
>
> @@ -111,8 +114,13 @@ while IFS="|" read -r series_id patch_id patch_url patch_name sha patchwork_inst
>          prev_series="$series_id"
>
>          # Get GHA runs
> -        tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?branch=series_$series_id&per_page=100"
> -        runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
> +        tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?per_page=9000"
> +        if [ "$tmp_url" != "$prev_url" ]; then
> +            prev_url="$tmp_url"
> +            all_runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
> +        fi
> +
> +        runs=$(echo "$all_runs" | jq -rc ".workflow_runs[] | select(.head_branch == \"series_$series_id\")")
>          not_found="$(echo "$runs" | jq -rc ".message")"
>      fi
>
> ---
> 2.31.1
>