[pw-ci] [PATCH] github: alternate url requests

[Michael Santana]

On Thu, Apr 14, 2022 at 8:46 AM Aaron Conole <aconole at redhat.com> wrote:
>
> Michael Santana <msantana at redhat.com> writes:
>
> > Hi Aaron,
> > Thanks for working on this. See comments below
> >
> > On Tue, Apr 12, 2022 at 5:13 PM Aaron Conole <aconole at redhat.com> wrote:
> >>
> >> Github is currently broken when requesting filtered results for the
> >> github actions runs.  While this stays broken, apply the following which
> >> works around the issue by filtering on the client side.
> >>
> >> Plan is to remove when
> >> https://github.community/t/no-longer-able-to-list-in-progress-builds-via-the-api/244175/3
> >> and
> >> https://github.community/t/workflow-call-has-an-inconsistent-status-in-github-rest-api/244164/2
> >> are resolved.
> >>
> >> Signed-off-by: Aaron Conole <aconole at redhat.com>
> >> ---
> >> diff --git a/github_get_logs.sh b/github_get_logs.sh
> >> index a646b47..9f44840 100755
> >> --- a/github_get_logs.sh
> >> +++ b/github_get_logs.sh
> >> @@ -68,7 +68,7 @@ print_errored_logs_for_commit () {
> >>      # Get run metadata
> >>      select="select(.head_branch==\"series_${series_id}\") |
> >> select(.head_sha==\"${sha}\") | select(.name==\"${test_name}\")"
> >>      headers="{id, logs_url}"
> >> -    run_meta="$(echo "$runs" | jq ".workflow_runs[] | $select | $headers")"
> >> +    run_meta="$(echo "$runs" | jq "$select | $headers")"
> >>
> >>      redirect_url="$(echo "$run_meta" | jq -r ".logs_url")"
> >>      run_id="$(echo "$run_meta" | jq -r ".id")"
> >> @@ -88,7 +88,7 @@ print_errored_logs_for_commit () {
> >>
> >>      length="$(echo "$jobs_results" | jq -r "length")"
> >>
> >> -    if [ "$length" -eq "0" ]; then
> >> +    if [ "X$length" == "X" -o "X$length" == "X0" ]; then
> > is the original not working?
> >>          echo "No build failures detected for series_${series_id}/${sha}. Exiting" 1>&2
> >>          return 0
> >>      fi
> >> @@ -134,8 +134,9 @@ print_errored_logs_for_commit () {
> >>  repo_name=$(echo "$repo_name" | sed -e 's@%2F@/@g' -e 's,%40,@,g')
> >>
> >>  # Get GHA runs
> >> -tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?branch=series_$series_id&per_page=100"
> >> -runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
> >> +tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?per_page=9000"
> > So at the time this was written I thought I had read that the max per
> > page you could request was 100. Is this not the case anymore? You can
> > instruct jq to print the size of the array/struct you get back to find
> > out
>
> That isn't the case now - additionally, if we don't specify the per_page
> value, we get a significantly reduced set in worker_runs[] array and are
> unable to find all the series we need.
Ok, if it isnt the case now then this all looks good to me
Acked-by: Michael Santana <msantana at redhat.com>
>
> > I guess in reality it doesnt matter much. We probably dont have more
> > than 100 runs going on at the same time. Going over 100 would really
> > only be useful we want to go back to get the status on old builds past
> > 100. Maybe it has been changed? let me know
>
> This gives the total runs associated with the system, so I don't know
> about 'simultaneously' or 'parallel' anything.
>
> >> +all_runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
> >> +runs=$(echo $all_runs | jq -rc ".workflow_runs[] | select(.head_branch == \"series_$series_id\")")
> > I would add quotes. I need to practice my quoting on nested shell
> > scripts. According to shellcheck I have been doing them wrong this
> > entire time
>
> Why?  There are probably quotes where they don't need to exist, tbh.
>
> > just quotes around all_runs is fine
>
> ACK
>
> >>  not_found="$(echo "$runs" | jq -rc ".message")"
> >>  if [ "$not_found" == "Not Found" ]; then
> >>      echo "\"$tmp_url\" could not be reached." 1>&2
> >> diff --git a/github_mon b/github_mon
> >> index 42478be..8670ce8 100755
> >> --- a/github_mon
> >> +++ b/github_mon
> >> @@ -60,12 +60,12 @@ make_result_for_series () {
> >>      repo_name="$6"
> >>
> >>      # Get the name of each respective workflow
> >> -    workflows="$(echo "$runs" | jq -r ".workflow_runs[].name" | sort -u)"
> >> +    workflows="$(echo "$runs" | jq -r ".name" | sort -u)"
> >>
> >>      # Check that workflows have all completed
> >>      select="select(.head_branch==\"series_${series_id}\") | select(.head_sha==\"${sha}\")"
> >>      headers="{status, conclusion, html_url}"
> >> -    run_meta="$(echo "$runs" | jq ".workflow_runs[] | $select | $headers")"
> >> +    run_meta="$(echo "$runs" | jq "$select | $headers")"
> >>
> >>      if [ "$(echo "$run_meta" | jq -r ".status" | sort -u)" != "completed" ]; then
> >>          echo "patch_id=$patch_id belonging to series_id=$series_id not completed. Skipping" 1>&2
> >> @@ -78,7 +78,7 @@ make_result_for_series () {
> >>          # Get run metadata
> >>          select="select(.head_branch==\"series_${series_id}\") | select(.head_sha==\"${sha}\") | select(.name==\"${WORKFLOWNAME}\")"
> >>          headers="{status, conclusion, html_url}"
> >> -        run_meta="$(echo "$runs" | jq ".workflow_runs[] | $select | $headers")"
> >> +        run_meta="$(echo "$runs" | jq "$select | $headers")"
> >>
> >>          # success/failure?
> >>          result="$(echo "$run_meta" | jq -r ".conclusion")"
> >> @@ -100,6 +100,9 @@ make_result_for_series () {
> >>
> >>  prev_series=""
> >>
> >> +prev_url=""
> >> +all_runs=""
> >> +
> >>  get_unsynced_series "$pw_instance" "$ci_instance" | \
> >>  while IFS="|" read -r series_id patch_id patch_url patch_name sha patchwork_instance patchwork_project repo_name gap_sync; do
> >>
> >> @@ -111,8 +114,13 @@ while IFS="|" read -r series_id patch_id patch_url patch_name sha patchwork_inst
> >>          prev_series="$series_id"
> >>
> >>          # Get GHA runs
> >> -        tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?branch=series_$series_id&per_page=100"
> >> -        runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
> >> +        tmp_url="$GITHUB_API/repos/$repo_name/actions/runs?per_page=9000"
> >> +        if [ "$tmp_url" != "$prev_url" ]; then
> >> +            prev_url="$tmp_url"
> >> +            all_runs="$(curl -s -S -H "${AUTH}" -H "${APP}" "${tmp_url}")"
> >> +        fi
> >> +
> >> +        runs=$(echo "$all_runs" | jq -rc ".workflow_runs[] | select(.head_branch == \"series_$series_id\")")
> >>          not_found="$(echo "$runs" | jq -rc ".message")"
> >>      fi
> >>
> >> ---
> >> 2.31.1
> >>
>