This repository was archived by the owner on Feb 5, 2020. It is now read-only.
ci/maintenance: Add more date formats to filter when using grafiti to clean aws resources #1890
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cpanato
force-pushed
the
add_more_date_formats
branch
from
2b373db to
3995263
Compare
squat
reviewed
Sep 12, 2017
|
my understanding of grafiti is it takes the tag you defined to use when deleting resources and match if the tag in the resource. I ran one grafiti delete using this config and at least it not break anything :) |
squat
suggested changes
Sep 12, 2017
There was a problem hiding this comment.
I don't think we should be throwing all of these dates at Grafiti. We should pick ONE that works and use it. This is confusing and highlights our uncertainty in our own tool. If we need to fix Grafiti to be consistent then we should do that in the Grafiti repo.
|
@squat grafiti is consistent in tagging the resources. What I saw in AWS was other users tagging the |
|
Sure, the questions is: can Grafiti interpret all of these expiration date formats? My suspicion is "no". |
|
@squat I ran this in one region which I I had free tier machines and each of one had this specific date format in the |
|
Ok, sorry I misunderstood how we were leveraging this date in Grafiti. This tag has no special meaning, e.g. time, in Grafiti other than being a string. We are just using this as the set of strings to select resources to delete. |
squat
approved these changes
Sep 12, 2017
|
thanks! also was good because I did a real testing :) |
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Jul 27, 2018
tag-aws.sh is using grafiti, whose tagPatterns takes jq expressions [1]. We've been using strftime since the script landed in 82bdd9f (installer/scripts: AWS tag and delete scripts, 2017-06-28, coreos/tectonic-installer#1239). jq's strftime doesn't respect your configured $TZ, but the coming jq 1.6 will add strflocaltime which does [2,3]. jq uses seconds since the epoch for date-time values [4]. You can test the new construct with: $ jq --null-input --raw-output 'now + 24*60*60 | strftime("%Y-%m-%d")' 2018-07-27 -d is not part of the POSIX date specification [5], but it (and the 'tomorrow' value) are supported by GNU Coreutils [6,7]. We've been using -d in clean-aws.sh for a while now, so this is now a new dependency. I've also dropped date_override, since we can just set date_string directly. And I've shuffled around some of the conditionals to avoid calling the 'date' and 'jq' commands needlessly when --date-override is set. I've also replaced the multiple date calls in clean-aws.sh with a single call to jq. jq was already a required dependency for this script, and only needing a single child process is much faster: $ time for i in $(seq 100); do A="$(jq --null-input '[["%Y-%m-%d", "%Y-%-m-%-d", "%m-%d-%Y", "%m-%-d-%-Y", "%-m-%-d-%-Y", "%d-%m-%Y", "%d-%-m-%-Y"][] | . as $format | [now, now - 24*60*60][] | strftime($format)]')"; done real 0m0.256s user 0m0.186s sys 0m0.077s $ time for i in $(seq 100); do A="$(date "+%Y-%m-%d" -d "-1 day")\",\"$(date "+%Y-%-m-%-d" -d "-1 day")\",\"$(date "+%m-%-d-%-Y" -d "-1 day")\",\"$(date "+%-m-%-d-%-Y" -d "-1 day")\",\"$(date "+%d-%m-%-Y" -d "-1 day")\",\"$(date "+%d-%-m-%-Y" -d "-1 day")\",\"$(date +%m-%d-%Y)\",\"$(date +%d-%m-%Y)\",\"$(date +%d-%-m-%Y)\",\"$(date +%Y-%m-%d)\",\"$(date +%Y-%-m-%-d)"; done real 0m1.358s user 0m0.604s sys 0m0.832s And that's despite the fact that the old approach skipped some formats for today (e.g. %m-%-d-%-Y had been only used to format yesterday). The plethora of date formats are mostly from 3995263 (ci: add more date format when grafiti apply the cleanning, 2017-09-12, coreos/tectonic-installer#1890), although we've had some since 82bdd9f. The motivation seems to be matching human-generated tags [8], which are less reliably formatted. [1]: https://github.com/coreos/grafiti/blob/89a8bc92ad7fde49cd3dd78197c6b3616a857f36/README.md#configure-grafiti [2]: https://github.com/stedolan/jq/wiki/FAQ [3]: jqlang/jq@06f2060 [4]: https://stedolan.github.io/jq/manual/v1.5/#Dates [5]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/date.html [6]: https://www.gnu.org/software/coreutils/manual/html_node/Options-for-date.html [7]: https://www.gnu.org/software/coreutils/manual/html_node/Relative-items-in-date-strings.html [8]: coreos/tectonic-installer#1890 (comment)
frobware
pushed a commit
to frobware/installer
that referenced
this pull request
Sep 17, 2018
tag-aws.sh is using grafiti, whose tagPatterns takes jq expressions [1]. We've been using strftime since the script landed in 82bdd9f (installer/scripts: AWS tag and delete scripts, 2017-06-28, coreos/tectonic-installer#1239). jq's strftime doesn't respect your configured $TZ, but the coming jq 1.6 will add strflocaltime which does [2,3]. jq uses seconds since the epoch for date-time values [4]. You can test the new construct with: $ jq --null-input --raw-output 'now + 24*60*60 | strftime("%Y-%m-%d")' 2018-07-27 -d is not part of the POSIX date specification [5], but it (and the 'tomorrow' value) are supported by GNU Coreutils [6,7]. We've been using -d in clean-aws.sh for a while now, so this is now a new dependency. I've also dropped date_override, since we can just set date_string directly. And I've shuffled around some of the conditionals to avoid calling the 'date' and 'jq' commands needlessly when --date-override is set. I've also replaced the multiple date calls in clean-aws.sh with a single call to jq. jq was already a required dependency for this script, and only needing a single child process is much faster: $ time for i in $(seq 100); do A="$(jq --null-input '[["%Y-%m-%d", "%Y-%-m-%-d", "%m-%d-%Y", "%m-%-d-%-Y", "%-m-%-d-%-Y", "%d-%m-%Y", "%d-%-m-%-Y"][] | . as $format | [now, now - 24*60*60][] | strftime($format)]')"; done real 0m0.256s user 0m0.186s sys 0m0.077s $ time for i in $(seq 100); do A="$(date "+%Y-%m-%d" -d "-1 day")\",\"$(date "+%Y-%-m-%-d" -d "-1 day")\",\"$(date "+%m-%-d-%-Y" -d "-1 day")\",\"$(date "+%-m-%-d-%-Y" -d "-1 day")\",\"$(date "+%d-%m-%-Y" -d "-1 day")\",\"$(date "+%d-%-m-%-Y" -d "-1 day")\",\"$(date +%m-%d-%Y)\",\"$(date +%d-%m-%Y)\",\"$(date +%d-%-m-%Y)\",\"$(date +%Y-%m-%d)\",\"$(date +%Y-%-m-%-d)"; done real 0m1.358s user 0m0.604s sys 0m0.832s And that's despite the fact that the old approach skipped some formats for today (e.g. %m-%-d-%-Y had been only used to format yesterday). The plethora of date formats are mostly from 3995263 (ci: add more date format when grafiti apply the cleanning, 2017-09-12, coreos/tectonic-installer#1890), although we've had some since 82bdd9f. The motivation seems to be matching human-generated tags [8], which are less reliably formatted. [1]: https://github.com/coreos/grafiti/blob/89a8bc92ad7fde49cd3dd78197c6b3616a857f36/README.md#configure-grafiti [2]: https://github.com/stedolan/jq/wiki/FAQ [3]: jqlang/jq@06f2060 [4]: https://stedolan.github.io/jq/manual/v1.5/#Dates [5]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/date.html [6]: https://www.gnu.org/software/coreutils/manual/html_node/Options-for-date.html [7]: https://www.gnu.org/software/coreutils/manual/html_node/Relative-items-in-date-strings.html [8]: coreos/tectonic-installer#1890 (comment)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.