Skip to content

Add support for ingesting large items from remote sources #179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 22, 2022

Conversation

marchuffnagle
Copy link
Contributor

This is a replacement for #168

@marchuffnagle
Copy link
Contributor Author

@matthewhanson This now supports s3://, http://, and https:// sources.

@jkeifer The SNS format is now { "href": "s3://blah" }

@philvarner philvarner closed this Feb 22, 2022
@philvarner philvarner reopened this Feb 22, 2022
@philvarner
Copy link
Collaborator

sorry, accidentally clicked the wrong button.

@philvarner
Copy link
Collaborator

when retrieving from arbitrary http sources, I worry about very slow retrieval and never-ending streams. This probably isn't as big of a concern here since the people doing ingest have control over what's getting ingested.

@marchuffnagle marchuffnagle merged commit 70e80c0 into main Feb 22, 2022
@marchuffnagle marchuffnagle deleted the feature/s3-payloads-2 branch February 22, 2022 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants