Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for WARC creation/writing #2

Closed
ikreymer opened this issue Apr 14, 2020 · 0 comments · Fixed by #10
Closed

Add support for WARC creation/writing #2

ikreymer opened this issue Apr 14, 2020 · 0 comments · Fixed by #10

Comments

@ikreymer
Copy link
Member

No description provided.

ikreymer added a commit that referenced this issue May 23, 2020
@ikreymer
serialization support as async iterator, #2 and digest computation #6
c79f6ad 
- WARCSerializer for serializing warc records as is
- WARCEnsureDigestSerializer for serializing warc records and computing block, payload digests and optional signature (experimental)
- Experimental SWARCSerializer for serializing a stream warc with unknown payload size in 'SWARC' format, appending payload, block
and signature as trailers. SWARCReader with SWARCParser for converting SWARC -> WARC
ikreymer added a commit that referenced this issue Jun 6, 2020
@ikreymer
error handling: improve errors if trying to read already consumed record
862ec64 
serialization via s async iterator, #2 and digest computation #6
- WARCSerializer for serializing warc records as is
- WARCEnsureDigestSerializer for serializing warc records and computing block, payload digests from full payload (non-streaming)
@ikreymer ikreymer mentioned this issue Jun 9, 2020
Merged
ikreymer added a commit that referenced this issue Jun 9, 2020
@ikreymer
WARC Serialize (#10)
25bbc00 
* error handling: improve errors if trying to read already consumed record
serialization via s async iterator, #2 and digest computation #6
- WARCSerializer for serializing WARC records and computing block, payload digests from full payload (non-streaming). Also computes the total content-length

* warcserializer:
- support gzipped output
- add createWARCInfo, support revisit params, add tests

* fix and update dist build

* update README: add writing WARCs, add collapsible sections for examples
switch to different uuid package, update dist/
update serializer tests

* README tweaks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

WARC Serialize webrecorder/warcio.js
1 participant
@ikreymer