Implementing abstraction to score final sequences in BeamSearch

[danieldeutsch]

Implements feature requests 2 and 3 from #5205 and closes #5113.

Changes proposed in this pull request:

• Other libraries, such as transformers or fairseq pick the top outputs from beam search by taking the sequence with the highest average log probability per token. See here for the transformers implementation. The AllenNLP implementation takes the sequence with the best total log probability of the sequence. This change adds an abstraction called FinalSequenceScorer to decide how the final sequences will be scored. The default is SequenceLogProbabilityScorer, which assigns a score equal to sum of the log probabilities per token (i.e., the current implementation). This PR also includes LengthNormalizedSequenceLogProbabilityScorer which assigns a score equal to the average log probability per token.
• The LengthNormalizedSequenceLogProbabilityScorer also has a length_penalty parameter, which will increase or decrease sequence scores based on their length. This is also included in the transformers beam search.

Notes

• This change does not break the code, but it does change the semantics of the second tensor returned by the search method in BeamSearch. Previously it was the log probabilities of the sequences, but now it is the score that is returned by the FinalSequenceScorer class. The new default is to return what is currently returned, so this should not break any models.

Before submitting

• I've read and followed all steps in the Making a pull request
  section of the CONTRIBUTING docs.
• I've updated or added any relevant docstrings following the syntax described in the
  Writing docstrings section of the CONTRIBUTING docs.
• If this PR fixes a bug, I've added a test that will fail without my fix.
• If this PR adds a new feature, I've added tests that sufficiently cover my new functionality.

After submitting

• All GitHub Actions jobs for my pull request have passed.
• codecov/patch reports high test coverage (at least 90%).
  You can find this under the "Actions" tab of the pull request once the other checks have finished.

[epwalsh]

This also looks great! I appreciate the thorough documentation 🙂 Just a few minor comments

[epwalsh]

Looks great! Thank you @danieldeutsch!