any way to let int64 marshal to int not string?

[jinleileiking]

#296

But I really need this feature.

[jinleileiking]

use float

[achew22]

You can also use int32 if you want your data to be represented as an int still.

[jinleileiking]

@achew22 my data is bigger than int32. Can int32 work?

[achew22]

If your data is bigger than an int32 then probably not but it depends on your usecase. Talk around with your team and see if there may be another strategy for attacking the problem (or use a float)

[jinleileiking]

I just want to set a unixtime(epoch) which is a int64

[achew22]

If you want to store a time, you should use the well known type Timestamp. This will correctly get marshalled to a Date object if you're using a swagger generated client.

https://developers.google.com/protocol-buffers/docs/reference/csharp/class/google/protobuf/well-known-types/timestamp

[Raviaryacsc]

What is the reasoning behind marshaling the int64, uint64 to string instead of number. unit32, float, doubles are marshaled as numbers. Is not it a bug which should be fixed?
Suggestion of using timestamp for epoch is not going to work.

[achew22]

To partially quote an earlier issue comment:

gRPC gateway follows the serialization rules of proto3 and the proto3 spec says that int64, fixed64, and uint64 are all strings.

This is the case because of IEEE 754 Double precision which JS implements using a 52 bit mantissa (1 for sign, and 11 for exponent). This means the largest number that JS can represent without losing precision is 2^52. Try it out:

(Math.pow(2,53) + 0) == (Math.pow(2,53) + 1)
true

So, rather than pretending that Javascript can handle it (and lose data) proto3 prefers to have the client interact with strings. Fewer footguns == better libraries.

If you want to interact with int64 as numbers, there are a bunch of JS bigint libraries out there that you can use to parse the string, perform whatever operations you want and get back a string. That would probably be a better way to handle this.

That is why I am suggesting using timestamp for epoch, since it was designed to work around the issue of the 52 bit mantissa.

[Raviaryacsc]

Proto3 to JSON Mapping:
int64, uint64 ---> String
float, double ----> number.
Does it seems right? I see point in your reply however should not this logic should apply to floating point, doubles instead of 64 bit integers?

[achew22]

A float (or single as it is sometimes called) as defined in proto as a 32 bit sequence:

• Sign bit: 1 bit
• Exponent width: 8 bits
• Significand precision: 24 bits (23 explicitly stored)

So it can only faithfully hold values up to 2^23, which is smaller than 2^52 no problem there. "Single precision is termed REAL in Fortran, float in C, C++, C#, Java, Float in Haskell, and Single in Object Pascal (Delphi), Visual Basic, and MATLAB. However, float in Python, Ruby, PHP, and OCaml and single in versions of Octave before 3.2 refer to double-precision numbers. In most implementations of PostScript, the only real precision is single." Javascript does not have an implementation of single.

While a Double is defined as:

• Sign bit: 1 bit
• Exponent: 11 bits
• Significand precision: 53 bits (52 explicitly stored)

So it can faithfully hold values up to 2^52 which isn't a coincidence because it was specified by the ECMAScript standard. "All arithmetic in JavaScript shall be done using double-precision floating-point arithmetic."

All quotes from the linked wikipedia page.

[Raviaryacsc]

Exactly. Double too fits in that requirement and should be JSONed as string instead of number why it is not ?
Also, JSON does not defined that Unit64 should be represented as string in JSON. it is a shortcoming of JS and not JSON. We can create the JSON with big numbers by many ways and JS is going to have problem with that. Then why this trick is done in Proto3 to Json mapping?

Today JSON is used with all sort of different languages and not just ties to JS.
I am fine with writing Un-Marshaller to convert String to Unit64 based on the field Name and Type and continue with my stuff but I see this is not a logical mapping.

[achew22]

Exactly. Double too fits in that requirement and should be JSONed as string instead of number why it is not ?

No, the double definition used in proto is the same float64/double definition used by javascript, IEE754 double. There is no need to convert since they are representing the same data the same way with the same edge cases.

Also, JSON does not defined that Unit64 should be represented as string in JSON. it is a shortcoming of JS and not JSON. We can create the JSON with big numbers by many ways and JS is going to have problem with that. Then why this trick is done in Proto3 to Json mapping?

Unfortunately the JSON (JavaScript Object Notation) spec defines everything relative to number so it does suffer from the same problem as Javascript since it is, definitionally, Javascript.

Today JSON is used with all sort of different languages and not just ties to JS.
I am fine with writing Un-Marshaller to convert String to Unit64 based on the field Name and Type and continue with my stuff but I see this is not a logical mapping.

You are always welcome to write a custom marshaller and if you think this is the best behavior for you then you should go for it. I would just advise you that you need to inspect all of your client libraries implementations to make sure they are not adhering to the JSON spec, which again has every number defined as a IEEE754 double precision number. If you fail to verify their incorrect behavior then you will be in a world of hurt and data loss. Every browser will do the correct thing and destroy your data.

However, if you're taking the time to write this in a non-browser environment, why not just use the gRPC generated client code in the language of your choosing? There are generators for most every language.

[kurin]

I'd like to revisit/reopen this if possible. While the proto3 spec defines the int64/uint64 json representations as strings, and the above discussion points out the good reasons for that, the fact is that very often json is used as an interchange format between parties who aren't using javascript.

Anyone moving to a protobuf service definition from a different system is probably going to hit a particularly difficult snag when they need to send timestamps, file sizes, or any other data that can reasonably expected to grow beyond a 32 bit representation.

For my part, I had to copy and paste not only the jsonpb package (and then remove seven lines), but I had to copy a lot of the runtime package as well.

I don't know how to provide this behavior without also modifying jsonpb (and having to make my case there as well), but would you be open to providing a toggle that would enable this behavior?

[johanbrandhorst]

If you're using JSON as an interchange format, why not just use float64 in your proto definitions?