block: precalculate hashes if enabled and use them during compaction (downloading) (#3031)

* block: precalculate hashes if enabled and use them during compaction

Added the possibility to ignore certain directories in
objstore.{Download,DownloadDir}. Do not download files which have the
same hash as in remote object storage. Wire up `--hash-func` so that
writers could specify what hash function to use when uploading. There is
no performance impact if no hash function has been explicitly specified.
Clean up the removal of files logic in Thanos Compact to ensure we do
not remove something that exists on disk already.

Tested manually + new tests cover all of this more or less.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* block: expose GatherFileStats and use it

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Revert "block: expose GatherFileStats and use it"

This reverts commit 259c70bfaaf85a66a82f9458f4e1c310ad43b1bf.

Signed-off-by: Giedrius Statkevičius <[email protected]>

* block: do not calc hash for dirs, add locks

Signed-off-by: Giedrius Statkevičius <[email protected]>

* docs/tools: update

Signed-off-by: Giedrius Statkevičius <[email protected]>

* shipper: pass s.hashFunc

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Fix according to Bartek's comments

Signed-off-by: Giedrius Statkevičius <[email protected]>

* compact: clean up comment

Signed-off-by: Giedrius Statkevičius <[email protected]>

* block: close with log on error

Signed-off-by: Giedrius Statkevičius <[email protected]>

* *: remove unused FNs

Signed-off-by: Giedrius Statkevičius <[email protected]>

* compact: add e2e test for new hash functionality

Signed-off-by: Giedrius Statkevičius <[email protected]>

* Fix according to Bartek's comments

Signed-off-by: Giedrius Statkevičius <[email protected]>

Author: GiedriusS

objstore.go

@@ -223,15 +223,21 @@ func DownloadFile(ctx context.Context, logger log.Logger, bkt BucketReader, src,
 }
 
 // DownloadDir downloads all object found in the directory into the local directory.
-func DownloadDir(ctx context.Context, logger log.Logger, bkt BucketReader, src, dst string) error {
+func DownloadDir(ctx context.Context, logger log.Logger, bkt BucketReader, originalSrc, src, dst string, ignoredPaths ...string) error {
 	if err := os.MkdirAll(dst, 0777); err != nil {
 		return errors.Wrap(err, "create dir")
 	}
 
 	var downloadedFiles []string
 	if err := bkt.Iter(ctx, src, func(name string) error {
 		if strings.HasSuffix(name, DirDelim) {
-			return DownloadDir(ctx, logger, bkt, name, filepath.Join(dst, filepath.Base(name)))
+			return DownloadDir(ctx, logger, bkt, originalSrc, name, filepath.Join(dst, filepath.Base(name)), ignoredPaths...)
+		}
+		for _, ignoredPath := range ignoredPaths {
+			if ignoredPath == strings.TrimPrefix(name, string(originalSrc)+DirDelim) {
+				level.Debug(logger).Log("msg", "not downloading again because a provided path matches this one", "file", name)
+				return nil
+			}
 		}
 		if err := DownloadFile(ctx, logger, bkt, name, dst); err != nil {
 			return err