Differentiable histogram

[JuanFMontesinos]

Hi,
I was wondering if there is any implementation of differentiable histograms. Maybe a in 3rd party library?

Since jax.numpy.histogram is supposed to match numpy's I don't think it is differentiable.

Thanks!

[jakevdp]

Can you say more about what you mean by a "differentiable histogram"? My understanding is that histograms are integer counts of occurrances within bins, and integers are not differentiable, because differentiability requires continuous values. Maybe there's something else you have in mind?

[pmocz]

Chiming in here as an unrelated party. What may make sense here for a differentiable "histogram" could be actually to implement cloud-in-cell binning (i.e. deposit particles into 2 nearest nearest neighbor bins (per dimension) with weights that scale as distance to bin edge.) The output of such a method changes continuously when you perturb the particle locations.

[jakevdp]

Even with weights, "nearest bin" is also problematic from an autodiff perspective because it is a fundamentally discontinuous quantity.

What about a kernel density estimation? With an appropriately smooth kernel, that would be continuously differentiable and share many of the properties of the scheme you describe.

JAX has an implementation of jax.scipy.stats.gaussian_kde if you'd like to try it.

[pmocz]

Thanks for the suggestion! That makes sense. What I describe (cic-binning) is similar to kernel density estimation with compact support, but I can see the "nearest bin" being an issue. Formally the answer is differentiable so there may be another implementation. I'd have to think more

[jakevdp]

I think you could probably differentiate with respect to weights within a fixed bin, but the gradient would not reflect the fact that changing the primal value may change the bin assignment.

[pmocz]

I think I see what you mean. For certain classes of optimization problems using autodiff, you do not want a binning method with compact support. kde would be the right way to do it