[REVIEW]: Singularity Registry: Open Source Registry for Singularity Images #426
Comments
|
Hello human, I'm @whedon. I'm here to help you with some common editorial tasks for JOSS. @smoe it looks like you're currently assigned as the reviewer for this paper 🎉. ⭐ Important ⭐ If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As as reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all JOSS reviews 😿 To fix this do the following two things:
For a list of things I can do to help you, just type: |
|
@smoe started review with: Singularity is important and everything that fosters its acceptance is appreciated. The setup is technically sound. I like the distributed effort. What seems missing is the concept of mirroring images for long time availability when a site goes down, which then truly preserves setups for scientific reproducibility. For that to happen, we would need identifiers that are not URL-dependent, though. Another feature I am missing is the automated creation of images, which would nicely work together with the description of CWL workflows that features references to software registries which in turn started to have references to Linux distributions (OMICtools now, bio.tools soon, I hope) and Debian at least having references to the registries already. Not having an exact clue what you ask me to review now, I went through your JOSS author guidelines on http://joss.theoj.org/about#author_guidelines:
The other "requirements for submission" I find kind of redundant and I consider them to be fulfilled. What I would typically do now is to install this locally and see if this works, which I kind of hope not to be imposed on me because the authors declare to us that they are working with exactly that setup themselves and that what we see is live on https://singularity-hub.org/collections or elsewhere - this is what should be explicitly stated in the paper.md, too. Otherwise, I hope this was not too obvious, I am a fan. |
|
Thank you @smoe and @pjotrp for starting the discussion! This is my first review with JOSS so please tell me if I am doing something incorrectly. I will follow up here with answers to your questions, and then make a suggested list of changes for the repository. @whedon is asking for the issues to be in the target repository or linked, and so I will address the decided changes there.
Thank you! I feel the same way. Mirroring and Backup
I have the same sentiment that there is risk in any kind of infrastructure that is not properly mirrored. The challenge into integrating a mirror strategy into Singularity Registry is that it would be very challenging to impose a "one storage fits all" strategy. For example, by way of Docker the entire software can be brought up with: and this includes a postgres database image. However, the suggested deployment is to use a (non container) postgres image with something like a hot standby for backup, and then a mirror to some external storage (e.g., local filesystem) or cloud (e.g., object storage). What I think is in the scope of Singularity Registry is to provide the core software for organization, but leave the details about how backup is done to the user. This fits well with the two use cases that I see:
In other words, the responsibility of ensuring the replication of the registry is going to differ based on the deployment, and is understood to be the choice of the deploying institution. What I've done to address back up, from the base that I can control (singularity-hub) is to have a deployment with a mirrored database, and all images in backed up cloud storage. If the web server goes down and the postgres and/or its backup endure, then it's not an issue to bring up the application again. If the web server and both databases go down, the entire set of containers and metadata in Storage can be used to repopulate a new database, and users are linked via the Github repos. Identifiers
Yes this is definitely the case! When a user references a Singularity Hub / Registry image, it might look something like this: implicitly said in the above is the default url for the Singularity Hub registry - this produces the same outcome: and so if another registry was deployed with the same image uri, the registry name would need to be specified for the image. @smoe do you have suggestions for how this could be better? For Singularity Hub, the namespace is well maintained to correspond with Github. For a Singularity Registry, the naming schema is up to the owner. For example, if I were running a cluster, I might have namespace correspond to user names and/or groups, and then projects owned by them. Each image does carry it's unique version (a hash of the file) however it wouldn't be intuitive to search and find. What (I think) you are hinting at is some sort of global wrapper to registries (akin to a package manager) that can find an image based on criteria, no matter where it is! This is a great idea, although would be a separate project (that uses the registries). Building
Building is really important! And actually, I bit off more than I could chew a few years ago when my original vision for Singularity Hub was centered around the workflows. I realized we needed just storage and management before that. To best answer this question, I want to make a distinction between Singularity Registry, and it's original base (the current Singularity Hub) that does in fact provide a building service. Singularity Hub was originally created to build images for users, as a free service, directly from a Github repository, and then have the images available immediately via the Singularity command line tool (for example, here is the repository that builds the image example above, when I push the build is done in the cloud automatically) and here is a quick image to show how it works:
It's thus very possible for some higher level workflow tool to use Singularity Hub (or Singularity Registry) API to get the command to download an image. What is out of scope is actually running the workflow, and how it plugs into a workflow tool. Here is an example with nextflow. The Registry is a little bit of a different use case. I created the Registry because institutions / users came to me and said "I don't want to use Github, and I don't want even to share. I just want a manager for my institution images that I can push after build, and then pull from and be able to use on my resource." So Singularity Registry does not have builders or any cloud integrations - it's storage is a local file system as opposed to a cloud resource. However, what I'm hoping to do for Singularity Hub and a new addition called SCI-F apps (and this is probably outside of the scope of this review) is to have easy templates / buttons so that a user can easily plug it into a workflow. There are many workflows so likely I would start with major cloud providers. For apps, the idea is to provide easy templates (for example, something to produce a SLURM job file) that helps with container integration. Singularity Hub vs. Registry vs. Github
Docker Hub
Any Docker image can be immediately converted to Singularity by adding the .tar.gz layers to an empty image base. As far as I know, Docker doesn't have a nice conversion to get a Docker image from a Singularity one. Some users (myself included) find the Singularity recipe format to be more intuitive than Docker, and make the decision based on that. Are there features of singularity that are particularly supported by singularityhub?Singularity aims to support scientific containers, so Singularity Hub and Registry take an extra step to serve metadata about the containers via the API. It's important to know about usage (downloads and stars) but also software, environment variables, labels, and runscripts. This supports being able to do more research analytics across containers to better understand how containers (and more broadly software) help answer scientific questions. Given the issues we have with reproducibility, I think this is essential CitationsThank you for linking to the citation page. I'd be happy to add more references. Given that the description talks on a high level about Docker, containers, would the most appropriate references be to related web content? It's challenging to find papers that are specifically about container software and infrastructure - there are mostly analyses that use them. Ongoing Research ProjectsI can definitely say that community members are using it, reflected in discussion on the list and Github issues, but I can't comment on specific projects. Thank you for your thorough review! I can address the above points after you've taken a look, and I'll try to organize the issues and then fixes in context of the check boxes. |
|
Dear @vsoch, please rest assured that I did not expect you to add any extra functionality to your hub. I need to thank you for your detailed replies to my comments - I hope you could copy some good parts from elsewhere. It is also my first review for JOSS - interesting :) Easiest first: Concerning references I suggest you reference/cite whatever that you want to reference. It is the least that JOSS can do for you. Concerning the writeup in paper.md, I tend to think that JOSS gives "us" more freedom than the many very short papers in JOSS are suggesting that the casual reader finds. For instance I found some early submission by our editor himself that has a huge picture - yes, you can have diagrams in your JOSS paper or whatever you like, if you like. I propose to have the summary in paper.md in two parts. The first paragraph would be something alike "Introduction to containers, why we need them and why we share them" and that you have estsablished a (the first?) user-installable non-centralized Open Source infrastructure to facilitate that exchange. You run a public instance at URL. The text that you already have can then basically follow as it is. Let 90% of the readers stop reading at that point - they either know that they want it and go to the repository to read the documentation, that they do not want it, or they follow the link to the instance. The ones that are interested but do not want to read the documentation shall then get a second paragraph with selected lines of what you explained above to explain how you did it, how different parts relate to each other ... and whatever you want people to know that turn around in their chairs to tell their office mates about what they have just read. @pjotrp , you are the boss, direct me if you see this differently, but I am truly excited about @vsoch's writing above. It has a bit of a conference coffee-break atmosphere in which you give a summary of what you are doing. I can well imagine that many readers would appreciate a bit of that in that second paragraph of your paper.md summary. This is also a bit of a criticism of a few articles in JOSS that treat JOSS like a "it is in the docs" DOI-machine. But frankly, a singuarity-hub is more than an R API of some C library and should get a bit of a special treatment. |
|
Mind: The paper should not be lengthy, according to the JOSS guidelines. It is better to have real information in a README or other doc in the repo (which can be referenced from the paper). Still the paper is a point of call for readers, so be concise but informative. A figure usually is a good idea. |
|
BTW have you thought of using IPFS as a backend? That could solve the redundancy question. We are experimenting with IPFS in several projects. |
|
Right, not long. Dear @vsoch, you will know what to do and I will be happy with it. That I have not checked everything above, yet, is just because I did not find the time, yet. It has nothing to do with your work. |
|
Dear @smoe, I have done ample changes to the paper.md and will move them (from our conversation above) here. Submission Changes
@pjotrp - I don't know much about the interplanetary file system, but it sounds interesting! I will take a look. I think backup for storage is important and something we can think about and test with various users, and eventually provide a "best practice" suggestion. To foreshadow that, I added a section to the FAQ |
|
@vsoch I like the new version of the paper better. I think it still can be improved a bit. First of all, don't keep the reader guessing (as a rule). I would put main section 3 first. Sections 1 and 2 are long for what they need to say, better to shorten them to a few lines each and embed them in section 3. I mean, we can assume that people understand reproducibility. In the context of containers just reference a paper that discusses that. As for what a container is, I would keep that simple too and avoid words like hazard, hitch, ideal, storm, etc. which are not often seen in scientific papers. How about something like: Singularity Registry is a non-centralized Free and Open Source infrastructure to facilitate sharing of containers that can be installed at the institutional or personal level and facilitates caching of container objects. A container is a bundle of binary deployed software that can be run consistently if the platform supports it. It is an aid in reproducibility [citation] because different researchers can run the exact same software stack on different underlying (Linux Intel) systems. Docker has become popular as a general container system because it allows software to be bundled with administrator privileges without actually having root and it allows the running of tools with restricted privileges which has impact on security. Docker, however, is known to be complex software and that has prevented admission in HPC environments. Singularity offers similar features to Docker for software deployment but requires less privileges than Docker and is therefore easier accepted in HPC environments since its introduction. So far, Singularity has been deployed in 45 HPC environments (right?). We have created a solution etc. .... |
|
@pjotrp I shorted the introduction quite a bit, and moved the bulk of the background into the docs introduction. Here is the updated paper. Your suggested introduction is fantastic, and I only tweaked it slightly (Docker in fact has huge security issues so it was a non starter for HPC). I left the remainder of the description with pictures assuming that interested readers will continue, and dis-interested will have been satisfied by the early description and move on. |
|
@vsoch I like it. Do people not have root-like privileges inside the singularity container? It may read that way now. I think you mean that singularity does not actually use root privileges to run the container - it stays in userland. Correct? After this niggle, @smoe can you sign off on your review and we'll take it to the next level? Thanks! |
|
@pjotrp that is correct - inside a Singularity container you are the same user as outside the container: And thus if you run as root, you are root inside as well! |
|
OK, I think that means someone needs to be root to create containers? That would be an HPC administrator? How would apt or rpm work otherwise? |
|
@pboesu yes, the typical workflow is to build on a resource (a personal work station,for example) with root, and then run the container anywhere. The user running the container cannot escalate to root on the host system, as would be possible with Docker. It matches nicely to a typical scientific workflow, where you write and test scripts on a personal computer, and then run at scale. The workflow typically look like this (with Singularity 2.3 and lower)
For a hosted registry, the three general use cases I have in mind are the following:
For using, meaning running, executing, shelling (all the functions important for working on a cluster) the user's have the same permissions as they would as their normal user. Binding the host to the container is done automatically for some folders (eg, tmp, the present working directory, the user's home) and on request with |
|
Good. Have a look at your text to see if that is reflected. On to @smoe ... |
|
Dear @vsoch, I am happy, except that I have not yet installed it - and I really need to do something else today, I am afraid - hey, I have installed docker on a remote server already. @pjotrp, would it be OK to skip that as I have no doubt at all that this will either work or be immediately fixed if not should I run into a local issue? Something that I should possibly comment on are the references. The DOIs are not in the bib file. Being a JOSS-newbie myself I can only speculate that this is where our editor would like to see them as described here https://tex.stackexchange.com/questions/3802/how-to-get-doi-links-in-bibliography . Also, I had the impression that there is a syntax glitch with the ARTICLE tags in there. Seen by chance, the @Kurtzer2017-x reference lacks at terminal "j". @pjotrp, please kindly help me out if I got this wrong. I found your comment to @pjotrp important: There is no chance for a priviledge escalation. It is the special feature why so many HPC environments are prefering Singularity over Docker: It reduces the risks for running foreign images to the general risk of running foreign software, which is the precondition for sharing. Conversely, if you want to share images, you will prefer using what is safe. And that again is why this registry is so important for Singularity to shine. @pjotrp, I happily sign-off this review to bring the submission to some "next level". Is there anything special I need to do about it? |
|
@smoe could you try a quick install to be able to tick the Functionality boxes? It is a requirement rather. No rush, you have been quick so far. |
|
Sigh. How comes I knew this would happen. I ran what I understood to be run and ran into an issue. @vsoch, I get you an account on that machine (regular Debian Stretch) if that is of any help. I followed the instructions on https://github.com/singularityhub/sregistry/blob/master/scripts/prepare_instance.sh with docker already installed. I successfully run the hello-world as root. As a regular user I get a "permission denied" for building in the fresh sregistry git checkout. I am not fluent enough with docker - should the So, running with sudo, I end up with: If I get this right, then this was all executed in the docker environment, so it should not have mattered that I did not have the autoconf etc. package installed. The missing gsettings.m4 is in the libglic2.0-dev package: For the better overview of yours, starting the process again, I get: On a sidenote, the python3.5.1 image you ask for is based on Debian Jessie. That is the oldstable one - maintained for another while, but ... it should be the current release "Stretch", not "Jessie". I just checked the images for docker and among the readily prepared images it seems like that python 3.6.3 is the closest choice that uses Stretch (https://store.docker.com/images/python). Frankly, though, so much gets installed for that image, that it does not really matter if there is a Python already shipping or not. I propose you just get an empty Debian image and add python 3.5 with "apt-get install python3" - as 3.5 is still the default python3 installation (https://packages.qa.debian.org/p/python3-defaults.html). So, after taking some deep breath I propose we just get this [the build error, the version of the distro is on you] fixed. My offer to get you an account on the machine still holds (it is not doing anything but helping Einstein@Home these days, so don't worry and send me a public ssh key if you like). |
|
Hey @smoe ! Sorry for your troubles! There are two things here - preparing your host to use Docker and compose, and then running the image. I Preparing HostIt's best to follow the instructions for your distribution here. Note that I provided instructions for an Ubuntu host. There are detailed instructions provided by Docker for non ubuntu, along with docker-compose. ** Important ** you need to, after you add your user to Docker's group, log out and back in before you continue. The call to Bringing up RegistryYou shouldn't need to build the image, but if you want to: Note that I didn't run just so you should be able to do: Let's walk through these steps, one at a time, and discuss when you hit a snag (and please don't move on). I've had two others deploy without much issue, so I think it comes down to your installation of docker and compose. I will update the docs to explicitly say that the prepare script example that I provided is for an ubuntu host. |
|
hey @smoe ! I updated the docs to very specifically say to install docker an docker-compose for your particular choice, along with needing to log in and out. If you want to see one of my colleagues deployed sregistry, check out randomroad Please let me know what is (and isn't clear) and I'll improve the instructions in the docs. I want to make sure they are user friendly for a wide range of audiences. |
|
I had used the wrong open shell for the build in which I had just added me to the docker user group. So, thank you for that. Indeed, I can now run as a regular user for which I had to use sudo before. Good news first: The docker-compose just worked fine. For @pjotrp: Executed with sudo or after properly being accepted as a member of the docker group on the system, the error remains the same: Does the build still work for you? Concerning the documentation, I just followed https://github.com/singularityhub/sregistry/blob/master/scripts/prepare_instance.sh to which your documentation links and that does the build. But then again, our dear editor wants me to build it, so, the pointer to prepare_instance.sh was fine to this point. I had not even seen install.md before your kind indirect pointer on singularityhub/sregistry#35 . You are right in that you are granting me the opportunity to learn some docker over the process. Hey, I had to install it when what I really wanted to install was Singularity :) Please allow that I continue with my learning tomorrow, night time over here. Many thanks for all your friendly and quick help. |
|
It looks like it's having trouble with singularity itself (and not the registry). I updated the image to clone from development, if you want to try building again. Here is also a complete build I did from scratch to confirm that it works for me: https://asciinema.org/a/141559?speed=3 Do you think it's too easy to find the prepare_instance.sh script (over the install instructions?) If so, let's definitely fix that. No worries, chat tomorrow! Do |
|
Sigh. I have removed all images (and running containers), and even after your change to the Dockerfile, I still get but (and this is a big but) singularityhub/sregistry#39 fixed that for me. In my commit note I speculated that the mirrors of Debian may differ slightly in their punctuality of updates. Or do you have a web cache for the pip installation? Anyway, I can proceed to the configuration now. |
|
I had disabled all the remote authentications - could that be the culprit? Is there a logfile that would indicate what the problem truly is? I would need to attach to your image for that, right? |
|
Hi @smoe - please see my instruction above: that will show the logs of the main server, which is the image called uwsgi. You do need to follow the instructions and choose some method to authenticate. The other option is that I could allow you (the user) to bring up a registry that you are unable to login to. Please run that command and show me the output (which will correspond to the server error). Minimally we can set up dummy variables so the server comes up without error, albeit you probably won't be able to do much without the configurations. |
|
It says that the SECRET_KEY would be empty - which it is not on my end. I also failed to see at which point you forward that file into your containers - but I have complete trust that we can settle that independently from this review. So I pressed the last button. I personally like the idea of a "display only" image server very much where users could anonymously download the images they want. This could possibly come very handy e.g. for the CWL community to automatically prepare images for whatever favorite pipelines of theirs. Feedback on those images could be with some github solution elsewhere. I had another look at your infrastructure and you indeed set up everything with python3 on Debian Jessie. The prior noted location dependency is presumably caused by the automated mirror finding with httpredir. I find this somewhat shocking, I must admit. To help your new users it would be nice if you could catch the one or other error and come up with a minimal system. |
|
hey @smoe! The secret key is a django application detail, and you must have missed where I wrote about it in the instructions to generate one. It's important for security features for any django application, and as you discovered, it will get angry at you if you don't have it. Would it be useful to step back and go back to the instructions and go through them to make sure you set everything up proper?
Yes! And actually a registry can serve as exactly this. This doesn't do away with the need of someone (some admin) setting it up and managing the images. And we can do slightly better because we don't only download anonymously, we can track the number of uses (anonymously) to better understand which software is most useful.
Yes I had this in mind when I created push - images could be pushed automatically after a pull request (PR) and discussion on a version controlled base like Github.
I am using the Docker provided python image, which likely uses those mirrors. I don't have much opinion, but if you want to avoid building all together I can suggest just pulling from Docker Hub. |
|
Hi @vsoch, admittedly, my very initial run was without the secret key since the prepare_instance.sh script did not generate it. But a next invocation, with all images removed if I did not mess it up, had it in place. And this is how it still is: I had used the django generator you referred to create it. Concerning the python image I am still a bit unhappy. It seems like this comes with the autotools already installed, and it even has git. Why does have python image feature all that? If you do not mind too much, please add those bits in - "back in" for autoconf and friends. This is just because it is needed and might not work once you upgrade to a different version of that image. Btw, I had a building version with Debian Stretch instead of Debian Jessie-based python:3.5.1. |
|
hey @smoe! Are you still have trouble with the secret key? You shouldn't need to remove or rebuild images, just restart with docker compose (and this is true whenever a change is done and the images are running): I added more verbose documentation about this to this section, in case you missed it. I can't comment on why the base python chooses to include core git (likely a choice of the developer) but per your request, I've added back in the libraries you listed. |
Big sigh. Hm. Dear @vsoch, If you do not mind too much, please send me public ssh key of yours and then have a look yourself. |
|
hey @smoe ! You named your file https://singularityhub.github.io/sregistry/install.html#secrets The reason it isn't finding it is because it looks specifically for that file, and defaults to importing "bogus_secrets.py" We will get this running, don't worry! You are doing great! :) |
|
Wow! A classic one. Many thanks for spotting that. No sure if I deserve that praise, though. Now it is the "502 Bad Gateway" and |
|
Actually I think you got it working :) When it first comes up and tries to do all the migrations, it's common to see that blurp, but then as you see below, it finishes the migrations. If the 502 persists ( try refreshing ) then just do a restart. I would bet if you refresh after you see the uwsgi running it will show up. Give it a try - we are close! |
|
Indeed. Amazing. I leave it up for @pjotrp to toy around with it. For all review purposes I tend to think that together with your public instance this is just fine now. If I continue, then we soon have all possible and impossible images gathering various tools for biomedical workflows in Debian Med. And I need to do some other work :-7 I am right in my understanding that you do not need to bootstrap from Docker but that regular Singularity images are just as fine as a start, right? Is there any significant difference in performance when you start from either? Or from a security perspective - it is the root permissions that you need to bootstrap/build/change anything that SingularityHub takes from us since those images are rightly executable, right? That point is trivial but maybe you find a place for it. |
|
@smoe great to hear you got it working! You are correct that Singularity does not have a dependency on Docker. A lot of work (e.g., biocontainers) has gone into Docker, so an early addition was to ensure that a Docker container could be imported seamlessly into Singularity. We have several other bases, including the mirror urls for each distribution (debootstrap, arch linux, and more here). If you want to get your hands into Singularity I would wait a few days - we are about to release 2.4 and the building commands are a bit more intuitive than the previous. For security, Docker has huge issues. Installation on a shared resource is a non starter. This is why Singularity has taken the HPC world by storm. It does not carry these same issues. As for performance, there have been a few papers that have compared Docker vs Singularity (if you do a Google search a few blog posts will come up) and (also in my experience) Singularity is a bit closer to "bare metal." It also is more seamlessly integrated into the environment, unlike Docker that is isolated. But I digress! The Singularity software proper is separate from a Registry to serve it's images (Singularity Registry). This is also different from Singularity Hub (which is a build service). I hope I made these distinctions clear in the documentation, please let me know if you would want to see more. This (I think) has been the most satisfying review I've ever done! To actually have the reviewers test and give feedback on setup, install, and documentation, is fantastic. |
|
Oh, too late, I had my first hands-on experience with Singularity somewhen in July and liked it a lot. It is basically what saves Debian Med in the HPC world where otherwise everyone is much after BioConda. Please let us digress a bit more - just a tiny, tiny bit. On the Debian Med mailing list, Mr. CWL (Michael Crusoe) just asked about an automated Docker image build service for Debian packages for the biocontainers to adopt when there is no BioConda package for it (https://lists.debian.org/debian-med/2017/10/msg00029.html). I personally think that we should have Debian Med autobuilt for Singularity in some reasonable manner - "reasonable" as in not having a separate image for every tiny package. And if I get this all right, then Singularity Hub could do that for us?!? Is there something that you feel like replying to Michael on the Debian Med list? Should we not somehow investigate how much packages share in terms of runtime dependencies and use that information together with workflow information to decide what images to build? Thank you for your kind words about your reviewing experience with me. I tend to think I have been luckier with my reviews "on your side of the review" in the past. But I certainly never had anyone so active and positive to direct me towards getting bits and pieces to run like you, @vsoch . Many thanks for that. Please contact me for anything I can help with on the Debian front of yours. |
Don't worry - most of it is the same! We are adding a lot more security to the image format that is built, and a single interface for interacting with it (basically a command called
Just taking a look, this would work easily with Singularity Registry (if someone wants to host) or the (soon to be released) Singularity hub. WIth Singularity 2.4 you have two options for new bases - either another local Singularity image file: or if it's hosted at Singularity Hub / Registry, the same image from there: and so the entire build recipe would look like this: and unlike Docker, if you use a singularity image as a base, it isn't assembled from layers. It's an image. With Singularity Hub 2.0, the managers can even freeze the images, which is a guarantee that it cannot change if you specify a specific version.
As an academic turned software engineer, I'm all for using free services with minimal extra. Here I would advocate for using Github for recipes, and then Singularity Hub to build the images (the next release).
For sure! But maybe let me finish up this version 2.0 first - I have at most a week or two to getting everything just right and tested. It's actually up and running now at a secret url :)
oh, would you be interested in doing this sort of formally? It's exactly the kind of "analysis" that I love - I wrote an entire paper just on similarity metrics for containers that is based on comparing contents. One of the reasons I am under the gun for Singularity Hub is that I got a very good review back, but don't want to move forward until I make the next version as flawless as I can.
Awesome!! I take it you are one of the debian gurus? I've used a debian flavor (ubuntu) since 2009 and never looked back :) |
|
I am one of the Debian Developers ([email protected]), but there are more guru-ish folks out there that are non-developers :) For the formal bits, one could tap into the Unified Debian Database (udd.debian.org) that has a table on packages and its dependencies. So this would create us a directed acyclic (should be) graph to annotate with sizes and do some clustering in. I can help on that, but would uttermost happily leave all the lead on that to you. We are currently adding tags to Debian packages to link them to OMICtools, bio.tools and the SciCrunch RRIDs. It looks like this https://anonscm.debian.org/cgit/debian-med/bowtie2.git/tree/debian/upstream/metadata . This way we could assign references to Singularity images, too. People you may be interested to contact are the ones from snapshot.debian.org. It provides legacy packages - you basically give it a date and it finds you all the packages you want in the version they were available at that date. You get a dynamic apt repository from which you could debootstrap your images. This should be helpful for reproductive sciene or for investigating the differences that advances of software bring. I do not know if this would be used often or if this only sounds nice to have. |
|
That thread on the Debian Med was rather fruitful - the NeuroDebian folks created a singularity image, see https://lists.debian.org/debian-med/2017/10/msg00037.html . |
|
@vsoch - please merge this PR singularityhub/sregistry#43 and then |
|
Thanks @arfon! Here is the completed DOI, which I've added additionally to the README.md: |
|
@whedon set 10.5281/zenodo.1012531 as archive |
|
OK. 10.5281/zenodo.1012531 is the archive. |
|
@smoe - many thanks for reviewing this submission and to @pjotrp for editing this one ✨ @vsoch - your paper is now accepted into JOSS and your DOI is http://dx.doi.org/10.21105/joss.00426 ⚡️ 🚀 💥 |
whedon
added
published
Submitting author: @vsoch (Vanessa Sochat)
Repository: https://www.github.com/singularityhub/sregistry
Version: v1.0.0
Editor: @pjotrp
Reviewer: @smoe
Archive: 10.5281/zenodo.1012531
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer questions
@smoe, please carry out your review in this issue by updating the checklist below (please make sure you're logged in to GitHub). The reviewer guidelines are available here: http://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @pjotrp know.
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
paper.mdfile include a list of authors with their affiliations?The text was updated successfully, but these errors were encountered: