file.replace turns CRLF line endings to CRCRLF

[slai]

I've been having a look at how the file.* functions work under Windows, and I've found that file.replace, with args to replace 'charlie' to 'charlie2', turns -

File 1

alpha<CR><LF>
bravo<CR><LF>
charlie<CR><LF>
delta<CR><LF>
echo

... into

File 2

alpha<CR>
<CR><LF>
bravo<CR>
<CR><LF>
charlie2<CR>
<CR><LF>
delta<CR>
<CR><LF>
echo

I've narrowed this down to what seems like a Python bug with the fileinput module, but I'm nowhere near certain. Below is the script I'm using to reproduce this behaviour, derived from the code used by Salt's file.replace with unimportant bits stripped out.

from __future__ import print_function

import fileinput
import re
import sys


def test_fileinput(path, pattern, repl):
    fi_file = fileinput.input(path,
                    inplace=True,
                    backup='.bak',
                    bufsize=1,
                    mode='rb')
    count = 0
    cpattern = re.compile(pattern)  # no flags

    for line in fi_file:
        result, nrepl = re.subn(cpattern, repl, line, count)

        print(result, end='', file=sys.stdout)
        print(repr(result), file=sys.stderr)

    fi_file.close()


if __name__ == '__main__':
    test_fileinput(sys.argv[1], sys.argv[2], sys.argv[3])

Running file 1 through this script, with the args 'charlie' to 'charlie2', results in file 2. If the mode argument to fileinput.input in the script above is r instead of rb then it works as expected.

I think it is due to the file being opened in text mode for writing, while being open in binary mode for reading. Looking at the code for the fileinput module (Python built-in) it seems like when inplace=True, the output file is always opened in text mode. Is this a Python bug?

[basepi]

I'm unsure if this is a Python bug or if we're misusing r and rb. In any case, thank you for the detailed report! Could you just add the output of salt --versions-report please? We'll investigate this.

[slai]

I'm running off the develop branch, commit f34dc6dadb0be0101db02abe3b898664e007ba09. The versions you see below are from the Windows installer I used to create my test environment.

           Salt: 2014.1.0
         Python: 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)]
         Jinja2: 2.7.1
       M2Crypto: 0.21.1
 msgpack-python: 0.4.0
   msgpack-pure: Not Installed
       pycrypto: 2.6
         PyYAML: 3.10
          PyZMQ: 14.0.1
            ZMQ: 4.0.3

[basepi]

Thanks for the update!

[whiteinge]

I think we need to replace the fileinput usage. It has caused more than a
few headaches.

[whiteinge]

Cross-ref #8051.

[lorengordon]

@basepi, fileinput is out, and this behavior around CRLF line endings is no longer occurring. I think this issue can be closed.

[basepi]

Awesome!

[Ch3LL]

Closing due to issue being resolved. Please ping me or open another issue if someone thinks this is incorrect.