Error with reddit-parser

[tfg1434]

I am getting an error while training my own reddit data from this website, https://files.pushshift.io/reddit/comments/
2017-8.

Trying it the first time:
Traceback (most recent call last):
File "reddit_parse.py", line 258, in
main()
File "reddit_parse.py", line 37, in main
parse_main(args)
File "reddit_parse.py", line 91, in parse_main
args.print_subreddit, args.min_conversation_length)
File "reddit_parse.py", line 242, in write_comment_cache
output_file.write(output_string + '\n')
File "reddit_parse.py", line 151, in write
self.file_reference.write(data)
File "C:\Users\16175\AppData\Local\Programs\Python\Python36\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f602' in position 404: character maps to
Second time, I added 'encoding=utf8' to line 151:
Traceback (most recent call last):
File "reddit_parse.py", line 258, in
main()
File "reddit_parse.py", line 37, in main
parse_main(args)
File "reddit_parse.py", line 91, in parse_main
args.print_subreddit, args.min_conversation_length)
File "reddit_parse.py", line 242, in write_comment_cache
output_file.write(output_string + '\n')
File "reddit_parse.py", line 151, in write
self.file_reference.write(data, encoding='utf8')
TypeError: write() takes no keyword arguments
Python 3.6.8
Tensorflow 1.9.0

Could someone please help me?

[DSMJR]

I personally could never get it to work, so I ended up just programming my own.

[tfg1434]

Could you send me the code? I have no idea how to do parsing.

[DSMJR]

Let me take a look at your data set and I might be able to help you out. My previous parser was for the Cornell corpus but I can look in to making one for your case. I emailed the person on the link you provided so I can find out how that data is structured.

[tfg1434]

Ok. Thanks a lot. You're awesome.

[DSMJR]

Thanks, we will keep in touch.

[tfg1434]

May be of some use?
https://www.youtube.com/watch?v=dvOnYLDg8_Y&list=PLQVvvaa0QuDdc2k5dwtDTyT9aCja0on8j

[DSMJR]

Thanks, that might help. the data seems to be a similar layout.

[tfg1434]

Might be json format
https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/

[DSMJR]

Yep, got that.I just downloaded some testing data to look at and start coding. This parser shouldn't be too complicated.

[tfg1434]

Thanks SO much.

…

On Sun, 1 Sep 2019 at 19:12, DSMJR ***@***.***> wrote: Yep, got that.I just downloaded some testing data to look at and start coding. This parser shouldn't be too complicated. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#59?email_source=notifications&email_token=AML4CLMV7MEJOUR7BU2LPE3QHRK47A5CNFSM4ISVGFXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UNTKQ#issuecomment-526965162>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AML4CLKLZRFM2TLJQRQCARDQHRK47ANCNFSM4ISVGFXA> .

[DSMJR]

No problem.

[DSMJR]

Some of the datasets on that website are corrupted after extraction. I didn't find any corrupted ones in the "daily" directory. Done with the parser, if you have any errors with it, do tell. It skips over deleted comments and some with no comments. I also included my cornell corpus parser, if you are interested. And also a link to a pre-parsed reddit dataset.
you're welcome! The parser link will stay open until you mark the issue closed
Don't forget to mark the issue as closed if your problem is solved.👍

Parser ---- https://drive.google.com/file/d/1YgDZrQGJXZybXAo_5_4SZXycBFUJ3jCo/view?usp=sharing

pre-parsed data ---- https://drive.google.com/uc?id=1s77S7COjrb3lOnfqvXYfn7sW_x5U1_l9&export=download

cornell corpus ---- https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html

[tfg1434]

You're the best. What data did you pre-parse?

[DSMJR]

It was the pre-parsed data in the README of this project. It seemed to be decent data so I sent it in case you weren't aware of it👍.

[tfg1434]

I'm sorry, I'm getting an error while parsing. It seems to just be a simple unicode decode error;

Traceback (most recent call last):
File "C:/Users/16175/Documents/chatbot-rnn/reddit-parse/parser.py", line 19, in
for i in file: #
File "C:\Users\16175\AppData\Local\Programs\Python\Python36\lib\codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 10: invalid continuation byte