Project maintenance/move project to marge-bot/marge-bot namespace?

[nejch]

Hi all, I see this project has quite an active community for its size, but it seems to be pretty hard to get things merged. From what I see in other issues/PRs, this means a lot of people are running their private forks with their unmerged changes.

I'm not sure if this is still meant to be actively maintained or if it was something that solved the author's problem at some point but is no longer needed. But it'd be really helpful, and if some of the open PR's (from early 2020 in some cases) get merged this could be used/easily expanded to basically have quite a few advanced features/workflows in GitLab via marge-bot.

How about giving some of the more active community contributors merge access, or if that's not feasible, moving the project to its own namespace where they can help get rid of the PR backlog and open up the project for new contributions?

I know I'd have some stuff coming in because we're looking at using it for our on-prem projects and would need some changes to make it scale better for large instances instead of running an army of bots.

@tclh123 @JaimeLennox

/cc @brettdh @flaviouk @FlorianLudwig @Hi-Angel @lkm @matthiasbalke @snim2 @nirizr @qqshfox are some of the people I've seen contributing/interacting lately but there are probably more.

Just wanted to start the discussion, sorry if this seems intrusive. Thanks! 🙇

[tclh123]

Hi @nejch

Thank you for bringing this up. I do think Marge bot needs more love and I'm glad there're new contributors like you.
I think bringing in some active contributors to the Marge maintainer team is a good idea. It's not about having the rights to merge, it's about having a common vision of this project and guide us and new contributors to continually improve this project.
Actually, I'm so glad that @qqshfox just became a new maintainer of this project. If anyone have passion on this just let us know.
Cheers!

[nejch]

Thanks for the update @tclh123 great to hear!

I guess what I meant by merge access is, whatever helps getting PR's merged in a timely manner so that people are motivated to contribute. There are some really small PR's open where having a few maintainers would already help just to move things along.

Would it help you if people suggest ideas on next steps to take? If so here are just some thoughts I'd have on what would make sense to me, not necessarily in this particular order:

1. Add a fully native python development environment (@qqshfox is on this already, Remove nix #291)
2. Cleanup/merge at least part of the PR backlog, especially the smaller/easier ones (like AccessLevel enum matches GitLab docs #294)
3. Add GitHub Actions to always run existing suites and clean up tests if needed, review existing testenv/linting etc
4. Add HTTPS cloning support so people don't need to provide duplicate credentials (implement HTTPS support for cloning (#225) #283)
5. Add support for codeowners & approvals: Emoji approve based on changes and code owners #254
6. Add support for per-project config files (i.e. for project-specific options, read a yml from the default branch in the target project when looping projects)
7. Maybe use libraries to avoid reinventing the wheel for all of the above (I can take this as I already help with python-gitlab, but same would go for parsing codeowners)
8. Make MR processing non-blocking so marge can scale better to more projects (I haven't given this one much thought yet).
9. Add some integration tests against a running Gitlab container to really test the tricky stuff like failed rebases etc, that some people describe in the other issues. I have some experience there too if needed.

I'm sure others would have more ideas as well, this is just from what I've seen so far.

[snim2]

This is a good summary, thanks @nejch .

The "small issues" (your item 2.) do include some quite important changes, for example #180 and #169 may be useful for some users, and many would be simple to implement (or as a minimum it might be better to document some things rather than have them in the codebase). Some of the documentation / message issues such as #234 would also be very helpful.

Scale

Like you, I've also had issues with scalability. I had to extend the "wait for pipeline to resolve" setting to 10 minutes for one of our projects, but the result is that devs working on projects with faster pipelines sometimes think Marge is "broken" because they have no way of knowing that they are waiting on another merge.

The suggestion in your item 8. is a good one (issues #224 and #97 are also related), and I'd love to see that implemented.

Plugins

There are also a number of other issues around workflow, integration and configuration, that might be resolved by adding "hooks" into the core merge algorithm, so that users can write plugins to either expose the bot's current state or integrate marge into another system or their usual workflow. These issues might include:

1. Support for extra pre-merge checks #296 Support for extra pre-merge checks
2. use emoji reactions instead of approver system #145 Use emoji reactions instead of approver systems
3. Add label on issue when MR is merged #138 Add label on issue when MR is merged
4. Feature request: Show marge's status #147 Show Marge status
5. Allow Marge to re-assign to last assignee after merge #290 Allow Marge to re-assign to last assignee after merge

I left a comment on #138 about how this might be implemented, but it needs more thought, especially around concurrency.

The reason I mention this, is that although Marge is not huge, the core algorithm is relatively complicated, and if we want it to support a wide range of workflows, we need to avoid adding complexity in the codebase as far as possible. Allowing users to create their own workflow would give Marge that flexibility. Those workflows might well include quite arbitrary integrations, such as sending Marge's status to a Slack webhook, or setting the bot user's GitLab status...

At the very least, more complexity could be added to a plugin system (which might provide some support for posting comments on issues / MRs, adding/removing labels, etc.) which would be separate from the core Marge algorithm.

State

Marge currently doesn't save any state, I believe, but issue 1 which would be nice to have, suggests that it might need to. This isn't a high priority issues, but it's the only one I could find that might require substantial changes to the current core algorithm.

[nirizr]

Like you, I've also had issues with scalability. I had to extend the "wait for pipeline to resolve" setting to 10 minutes for one of our projects, but the result is that devs working on projects with faster pipelines sometimes think Marge is "broken" because they have no way of knowing that they are waiting on another merge.

I think my #293 improves this. Within and outside of a single project (We have an hour long pipeline :X ).

Allowing users to create their own workflow would give Marge that flexibility.

+1 for plugins and hooks. I've added a functionality to my Marge fork that forces the "only allow merge when pipeline succeeds" and protection for the master branch, to fight high privileged users that occasionally turn those off to bypass tests :-X

I can see this is something nobody will want in Marges' codebase but will benefit few users.

[snim2]

Thanks @nirizr I like the idea of #293 but I think one of the problems you've highlighted is that we need some good issue triage, because there are several possible approaches here, and with some of the other feature requests.

[nejch]

Just wanted to follow up on this two months after. It's nice that there are ideas and direction laid out here but I still think it's more about getting real activity here and just merging in the basic stuff first, before the rest can even be addressed.

Otherwise contributors have zero confidence their PRs will even be considered, see #254 (comment). @tclh123, can @qqshfox as the new maintainer help merge the existing backlog if others don't have time? See e.g. even trivial PRs like dependency updates not getting attention etc.

If not, I think forking to a more maintained project will start to make more sense for people as @lkm said in the other issue, I'd just try to avoid that as then you have PRs and issues all over the place.

[ssvvarunshreyas]

Just wanted to check in and see if there has been any progress or update on #147.

Thanks in advance.

[gforcada]

Hi all! I'm also interested in seeing this moving forward 🚀

marge-bot is already taken as github username, but margebot is still not, is anyone interested in joining the effort there? Is anyone from the actual maintainers team active and willing to help on the transition? Or setting a plan to help get PRs merged? 🍀 🤞🏾

I'm thinking about waiting after Christmas period is over (January 10th 2023) and if no reaction has been made, I'm willing to create a new organization here in GitHub and ask on PRs to move over there.

Hopefully before that we have reactions and comments here? Anyone would like to join, so they can be added from the first day? 🤝 🎉

[nirizr]

I'd love moving my PR (and perhaps a few smaller changes) to a better-maintained fork. It seems current repo maintainers have other priorities nowadays and I encourage a fork if no progress can be made in the current repo.

[nejch]

Sorry for the late reply @gforcada. I don't want to overcommit as I'm involved in a few things atm, but I'd be open to help as well if you migrate. Maybe at least migrating the backend to python-gitlab (which I help maintain), to avoid reinventing the API client every time here (as you did in #362 etc, but would also help with other issues that pop up here).

Edit for context: I've given up a bit on this since the original post almost 2 years ago as there was not much movement - many of the PRs above have not been merged - but if there is a fork with more activity I'll probably reconsider :)

[okias]

@nejch What about talking about it over the beer?