Skip to content

scripts/maintenance: Fix dates (tomorrow vs. today, etc.) #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

wking
Copy link
Member

@wking wking commented Jul 26, 2018

Builds on #76; review that first.

tag-aws.sh is using grafiti, whose tagPatterns takes jq expressions. We've been using strftime since the script landed in 82bdd9f (coreos/tectonic-installer#1239). jq's strftime doesn't respect your configured $TZ, but the coming jq 1.6 will add strflocaltime which does. jq uses seconds since the epoch for date-time values. You can test the new construct with:

$ jq --null-input --raw-output 'now + 24*60*60 | strftime("%Y-%m-%d")'
2018-07-27

-d is not part of the POSIX date specification, but it (and the tomorrow value) are supported by GNU Coreutils [6,7]. But we've been using -d in clean-aws.sh for a while now, so this is now a new dependency.

I've also dropped date_override, since we can just set date_string directly. And I've shuffled around some of the conditionals to avoid calling the date (or jq) command needlessly when --date-override is set.

I've also replaced the multiple date calls in clean-aws.sh with a single call to jq. jq was already a required dependency for this script, and only needing a single child process is much faster:

$ time for i in $(seq 100); do A="$(jq --null-input '[["%Y-%m-%d", "%Y-%-m-%-d", "%m-%d-%Y", "%m-%-d-%-Y", "%-m-%-d-%-Y", "%d-%m-%Y", "%d-%-m-%-Y"][] | . as $format | [now, now - 24*60*60][] | strftime($format)]')"; done

real   0m0.256s
user   0m0.186s
sys    0m0.077s
$ time for i in $(seq 100); do A="$(date "+%Y-%m-%d" -d "-1 day")\",\"$(date "+%Y-%-m-%-d" -d "-1 day")\",\"$(date "+%m-%-d-%-Y" -d "-1 day")\",\"$(date "+%-m-%-d-%-Y" -d "-1 day")\",\"$(date "+%d-%m-%-Y" -d "-1 day")\",\"$(date "+%d-%-m-%-Y" -d "-1 day")\",\"$(date +%m-%d-%Y)\",\"$(date +%d-%m-%Y)\",\"$(date +%d-%-m-%Y)\",\"$(date +%Y-%m-%d)\",\"$(date +%Y-%-m-%-d)"; done

real   0m1.358s
user   0m0.604s
sys    0m0.832s

and that's despite the fact that the old approach skipped some formats for today (e.g. %m-%-d-%-Y had been only used to format yesterday).

@openshift-ci-robot openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 26, 2018
@wking wking force-pushed the maintenance-script-dates branch 2 times, most recently from 615e093 to c1d19a7 Compare July 26, 2018 18:22
@wking
Copy link
Member Author

wking commented Jul 26, 2018

The e2e-aws errors were more EIP exhaustion:

2 error(s) occurred:

* module.vpc.aws_eip.nat_eip[0]: 1 error(s) occurred:

* aws_eip.nat_eip.0: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached.
	status code: 400, request id: 6243c771-d3fe-42c7-8607-a578bdfff21e
* module.vpc.aws_eip.nat_eip[2]: 1 error(s) occurred:

* aws_eip.nat_eip.2: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached.
	status code: 400, request id: d90dddb6-66cd-4966-baf2-8d0af2fc946f

I've got a script cleaning out leaked EIPs now, so we should be clear on that front.

/retest

@yifan-gu
Copy link
Contributor

/lgtm

Good found!

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 27, 2018
@openshift-bot openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 27, 2018
tag-aws.sh is using grafiti, whose tagPatterns takes jq expressions
[1].  We've been using strftime since the script landed in 82bdd9f
(installer/scripts: AWS tag and delete scripts, 2017-06-28,
coreos/tectonic-installer#1239).  jq's strftime doesn't respect your
configured $TZ, but the coming jq 1.6 will add strflocaltime which
does [2,3].  jq uses seconds since the epoch for date-time values [4].
You can test the new construct with:

  $ jq --null-input --raw-output 'now + 24*60*60 | strftime("%Y-%m-%d")'
  2018-07-27

-d is not part of the POSIX date specification [5], but it (and the
'tomorrow' value) are supported by GNU Coreutils [6,7].  We've been
using -d in clean-aws.sh for a while now, so this is now a new
dependency.

I've also dropped date_override, since we can just set date_string
directly.  And I've shuffled around some of the conditionals to avoid
calling the 'date' and 'jq' commands needlessly when --date-override
is set.

I've also replaced the multiple date calls in clean-aws.sh with a
single call to jq.  jq was already a required dependency for this
script, and only needing a single child process is much faster:

  $ time for i in $(seq 100); do A="$(jq --null-input '[["%Y-%m-%d", "%Y-%-m-%-d", "%m-%d-%Y", "%m-%-d-%-Y", "%-m-%-d-%-Y", "%d-%m-%Y", "%d-%-m-%-Y"][] | . as $format | [now, now - 24*60*60][] | strftime($format)]')"; done

  real   0m0.256s
  user   0m0.186s
  sys    0m0.077s
  $ time for i in $(seq 100); do A="$(date "+%Y-%m-%d" -d "-1 day")\",\"$(date "+%Y-%-m-%-d" -d "-1 day")\",\"$(date "+%m-%-d-%-Y" -d "-1 day")\",\"$(date "+%-m-%-d-%-Y" -d "-1 day")\",\"$(date "+%d-%m-%-Y" -d "-1 day")\",\"$(date "+%d-%-m-%-Y" -d "-1 day")\",\"$(date +%m-%d-%Y)\",\"$(date +%d-%m-%Y)\",\"$(date +%d-%-m-%Y)\",\"$(date +%Y-%m-%d)\",\"$(date +%Y-%-m-%-d)"; done

  real   0m1.358s
  user   0m0.604s
  sys    0m0.832s

And that's despite the fact that the old approach skipped some formats
for today (e.g. %m-%-d-%-Y had been only used to format yesterday).

The plethora of date formats are mostly from 3995263 (ci: add more
date format when grafiti apply the cleanning, 2017-09-12,
coreos/tectonic-installer#1890), although we've had some since
82bdd9f.  The motivation seems to be matching human-generated tags
[8], which are less reliably formatted.

[1]: https://github.com/coreos/grafiti/blob/89a8bc92ad7fde49cd3dd78197c6b3616a857f36/README.md#configure-grafiti
[2]: https://github.com/stedolan/jq/wiki/FAQ
[3]: jqlang/jq@06f2060
[4]: https://stedolan.github.io/jq/manual/v1.5/#Dates
[5]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/date.html
[6]: https://www.gnu.org/software/coreutils/manual/html_node/Options-for-date.html
[7]: https://www.gnu.org/software/coreutils/manual/html_node/Relative-items-in-date-strings.html
[8]: coreos/tectonic-installer#1890 (comment)
@wking wking force-pushed the maintenance-script-dates branch from c1d19a7 to 184a0bf Compare July 27, 2018 14:15
@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jul 27, 2018
@openshift-bot openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 27, 2018
@wking
Copy link
Member Author

wking commented Jul 27, 2018

PR needs rebase.

Rebased with c1d19a7 -> 184a0bf. I think this was just the context change due to #75, but @openshift-bot and GitHub seem overly touchy about that, because git itself made no complaints about conflicts during the rebase. Anyhow, this will need another /lgtm, @yifan-gu.

@yifan-gu
Copy link
Contributor

yifan-gu commented Aug 1, 2018

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 1, 2018
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wking, yifan-gu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 1, 2018
@openshift-merge-robot openshift-merge-robot merged commit 5ab0ae5 into openshift:master Aug 1, 2018
@wking wking deleted the maintenance-script-dates branch August 1, 2018 02:08
stbenjam pushed a commit to stbenjam/installer that referenced this pull request Feb 10, 2021
Turn on auth for ironic json-rpc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants