Using elixir with partisan

[jaybe78]

For the context:

I’m currently building an app with an expectation of 2/3 millions concurrent users.
Those users would spread around 300 000 + channels.
So each channel would contain something like 100 users or less joining and they would mostly exchange chat messages.

I read somewhere that after 40/50 nodes, you can start having issues like loss of messages and it generally does not scale well above that number, and thus where I stumbled upon Partisan.

I hear that Partisan is used for very large cluster.
I have not found any documentation to set up locally Partisan in an elixir cluster.

Yes there is [link](Partisan Cheatsheet — partisan v5.0.0-beta.14) to help join nodes manually, though is there way to set up the cluster so that it finds node automatically ?

I think it would be great for the elixir community to have a chance to use it in their project.

Is that something you could provide in this repo?

I've also currently set up a distributed cluster on production on AWS ECS using service discovery and lib cluster which works quite well.

I would like to know whether that would be possible to reproduce the same thing using Partisan, and would that be as seamless as libcluster, like barely any config needed except setting up service discovery.

Cheers.

[aramallo]

Hi @jaybe78, very happy for you to consider Partisan!

So, we currently have some cluster formation strategies supported and it should be easy to add more.

partisan_peer_discovery_agent

A statem that uses a defined strategy to pool for new peers

partisan_peer_discovery_list

The simplest strategy takes a static list of nodes

config :partisan,
  peer_discovery: [
    enabled: true,
    type: :partisan_peer_discovery_list,
    timeout: 5_000,
    polling_interval: 10_000,
    initial_delay: 10_000,
    config: %{
      name: :mynode,
      addresses: [
        "192.168.40.1:9000",
        "192.168.40.10:9000",
        "[email protected]:9000"
      ]
    }
  ]

partisan_peer_discovery_dns

Uses DNS for service discovery. We use this one for K8s.

import Config

config :partisan,
  peer_discovery: [
    enabled: true,
    type: :partisan_peer_discovery_dns,
    timeout: 5_000,
    polling_interval: 10_000,
    initial_delay: 10_000,
    config: %{
      record_type: :fqdns,
      query: "foo.local",
      node_basename: "foo"
    }
  ]

I was thinking about adding one for ECS as I will need it soon.

[jaybe78]

Hi @aramallo

thanks so much for the quick response

Using partisan_peer_discovery_dns for ECS should work as well right ?

I'm going to test that on ECS and I will share the result here.

Hopefully I should be fine :)

Just last question:

Is it possible to disable global.erl (connect_all false) when using partisan ?

If I do a GenServer.cast to a process in another node, it's not gonna anymore right ?
I have to switch to using the partisan gen server ?
Same for monitoring nodes, I can't use kernel anymore ?

thanks

[jaybe78]

Hey @aramallo

the documentation says:

In addition to using Partisan-specific functions to manage a cluster and send messages, adopting Partisan implies the need to replace some Erlang BIFs with Partisan's counterparts.

Does that mean Phoenix Tracker, Presence, who relies on PubSubServer would not work anymore ?
Example:

config

config :messages, MessagesWeb.Endpoint,
  adapter: Bandit.PhoenixAdapter,
  pubsub_server: MyPubSub

Presence

defmodule MyPubSub.Presence do

  use Phoenix.Presence,
    otp_app: :messages,
    pubsub_server: MyPubSub

  def fetch(topic, presences) do
  ...
  end

I'm asking because Razvan Draghici explains here that he created a specific adapter to use PubSub. Is that necessary ?

What about GenServer ?

the documentation says

This is mainly to cope with the impossibility for Partisan to represent remote pids and references in the way Distributed Erlang does. For that reason, pids and references are encoded manually (or automatically is the configuration options pid_encoding and ref_encoding are enabled) using the partisan_remote_ref:from_term/1 function

If I understand correctly, this would not work

 GenServer.cast(pid, {:xxx, self(), room, join_ref, user_id})

Cheers,

[aramallo]

Hi @jaybe78

I am afraid it will be necessary. The key issue is that OTP behaviours and or any Elixir module accepting remote pids and references will crash if they receive a partisan_remote_ref:t().

Lets say I have a partisan remote pid

iex> ppid = :partisan.self()
[:"[email protected]" | "#Pid<0.4768.0>"]

The following will crash because GenServer expectes a pid()

iex> GenServer.cast(ppid)
** (UndefinedFunctionError) function GenServer.cast/1 is undefined or private. Did you mean:

      * cast/2

    (elixir 1.18.4) GenServer.cast([:"[email protected]" | "#Pid<0.4768.0>"])
    iex:3: (file)

Obvisouly if the server is local you could convert partisan encoded terms to local terms

iex> GenServer.cast(partisan_remote_ref:to_term(ppid))

But partisan_remote_ref:to_term(ppid) will fail with an exception when ppid is not local e.g.

iex> remote_pid = [:"[email protected]" | "#Pid<0.10000.0>"] 
...> :partisan_remote_ref.to_term(remote_pid)
** exception error: bad argument
     in function  partisan_remote_ref:to_term/1
        called as partisan_remote_ref:to_term(['[email protected]'|<<"#Pid<0.10000.0>">>])

If you want to GenServer.cast to a remote server using Partisan, then the server should have been implemented using the partisan_gen_server behaviour instead. This is why, I guess, Razvan created his own adapter.

partisan_gen* modules are forks of the OTP behaviours that allow them to use partisan-encoded pids and references.

[jaybe78]

ok I understand.
So When implementing a Partisan GenServer

Instead of

use GenServer

That would be

use partisan_gen_server

What about Phoenix Presence/Tracker ?
Do we have to use specific Partisan pub/sub to make them work ?
If so, would you have an example ?

Sorry for all the questions... :D

Cheers

[aramallo]

Correct!

What about Phoenix Presence/Tracker ?

I would need to check how they work as I do not use Phoenix at all :-) For PubSub we use our own solution i.e. Bondy :-)

But I suspect you might need to fork it or do an adapter/wrapper.

Also since you are in Elixir, you might want to roll-your-own Macro to aid in that.

Also something very important, if your messages between Partisan nodes will contain distribution-sensitive types e.g. pids, references, ports and/or aliases, you will need to encode them using partisan_remote_ref first !!! Same goes if you are storing those in an external storage e.g. file, db.

[jaybe78]

But I suspect you might need to fork it or do an adapter/wrapper.

So more or less what what razvan started to do there https://github.com/tzumby/phoenix_pubsub_partisan/blob/main/mix.exs
right ?