categorical accuracy doesn't account for mask (w/ proposed solution)

[braingineer]

Hi all,

currently, accuracy is

def categorical_accuracy(y_true, y_pred):
    return K.mean(K.equal(K.argmax(y_true, axis=-1),
                          K.argmax(y_pred, axis=-1)))

I am running an RNN with variable length sentences and I want to get the accuracy of predictions at every time step. So basically, for a 3dim tensor, I have a binary matrix. I want each timestep of each batch to only be counted in the accuracy mean if it has a 1 entry in the binary matrix.

Proposed solution mwe (though, this is in theano, not using the abstracted keras backend.. but the change is trivial):

import numpy as np
import theano.tensor as T

### (batch size=2, seq len=3, output classes=5)
pred = np.arange(30, dtype=np.float32).reshape(2,3,5)
pred /= pred.sum(axis=-1, keepdims=True)

### target matrix to match pred
target1 = np.ones_like(pred)
target1[:,:,:-1] = 0

### target matrix to match mask
target2 = np.ones_like(pred)
target2[:,:,:-1] = 0
target2[0,-1,-1] = 0

### mask; 
# [   [1, 1, 0]
#     [1, 1, 1]  ] 
mask = np.ones(pred.shape[:-1])
mask[0,-1] = 0

# preserve last dimension 
flat_shape = lambda mat: (reduce(lambda x,y: x*y, mat.shape[:-1]), mat.shape[-1])
F_flat = lambda mat: T.reshape(mat, flat_shape(mat))

# compare two matrices
F_comp = lambda mat1, mat2: T.eq(T.argmax(mat1,axis=-1), T.argmax(mat2,axis=-1))

# mask the matrix
F_mask = lambda mat, mask: mat[mask.flatten().nonzero()]

# get the mean
F_mean = lambda mat: T.mean(mat)

## current method for categorical accuracy
F_score1 = lambda mat1, mat2: F_mean(F_comp(mat1, mat2))

## masked method
F_score2 = lambda mat1, mat2, mask: F_mean(F_mask(F_comp(F_flat(mat1), F_flat(mat2)), mask))

F_score1(pred, target1).eval() ## returns 1.0
F_score1(pred, target2).eval() ## returns 0.83
F_score2(pred, target1, mask).eval() ## returns 1.0
F_score2(pred, target2, mask).eval() ## returns 1.0

edit: currently using this

def categorical_accuracy(y_true, y_pred, mask=None):
    if mask is not None:
        eval_shape = (reduce(mul, y_true.shape[:-1]), y_true.shape[-1])
        y_true_ = K.reshape(y_true, eval_shape)
        y_pred_ = K.reshape(y_pred, eval_shape)
        flat_mask = K.flatten(mask)
        comped = K.equal(K.argmax(y_true_, axis=-1),
                          K.argmax(y_pred_, axis=-1))
        ## not sure how to do this in tensor flow
        good_entries = flat_mask.nonzero()[0]
        return K.mean(K.gather(comped, good_entries))

    else:
        return K.mean(K.equal(K.argmax(y_true, axis=-1),
                              K.argmax(y_pred, axis=-1)))

and i've shot from loss: 3.9160 - acc: 0.1424 to loss: 3.9165 - acc: 0.2614

[braingineer]

if anyone knows the equivalent of vector.nonzero() in tensorflow, I could write a pull request to update the backend and metrics. I also have a perplexity implementation related to categorical entropy I could put in as well:

def perplexity(y_true, y_pred, mask=None):
    if mask is not None:
        y_pred /= K.sum(y_pred, axis=-1, keepdims=True)
        mask = K.permute_dimensions(K.reshape(mask, y_true.shape[:-1]), (0, 1, 'x'))
        truth_mask = K.flatten(y_true*mask).nonzero()[0]  ### How do you do this on tensorflow?
        predictions = K.gather(y_pred.flatten(), truth_mask)
        return K.pow(2, K.mean(-K.log2(predictions)))
    else:
        return K.pow(2, K.mean(-K.log2(y_pred)))

[braingineer]

ya that's what I did:

https://github.com/braingineer/keras/blob/master/keras/metrics.py#L10

Though, it's better if you use the mask that's passed with the network. It requires a commitment inside the main part of Keras too to push the mask through. But, it requires some some operations that I don't know how to implement inside Tensorflow, so I haven't pushed them upstream.

[braingineer]

No, I solved it in theano ages ago (as per my first answer). I just didn't solve it for tensorflow, hence why this is still open.

The passing of the mask to the accuracy calculations should be happening anyway (imo).

[tbsflk]

I used the implementation provided by @braingineer

 def categorical_accuracy(y_true, y_pred, mask=None)
    if mask is not None:
       ...
    else:
       ...

but the problem is that mask is never passed into the function. Do I have to change something else somewhere to make that happen?

[OmniaZayed]

Hi @braingineer,
Thank you for sharing your code.
I have the same problem as @tbsflk where the mask is not passed to the function. Any thoughts on that?