Skip to content

Commit 7e1c60f

Browse files
committed
Rewrite test-tree generation in Python
This is essentially a fresh start. I originally thought we might re-use the list of special characters from testTree.php, but then I realized it was better to just loop over the potentially troublemaking regions of ASCII and get all the special characters that way. I also changed the license to MIT since Permanent's upstream code is under that. There will likely be non-trivial bits of code in this repository, so we might as well use a real FOSS license. If we later also have a non-trivial amount of raw data (e.g., sample source data) then we could put the data under CC0 while keeping the code under MIT.
1 parent 0aa1801 commit 7e1c60f

11 files changed

+28656
-142
lines changed

.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
test-tree

LICENSE

-121
This file was deleted.

LICENSE.md

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
The MIT License (MIT)
2+
=====================
3+
4+
Copyright © `2023` `Open Tech Strategies, LLC`
5+
6+
Permission is hereby granted, free of charge, to any person
7+
obtaining a copy of this software and associated documentation
8+
files (the “Software”), to deal in the Software without
9+
restriction, including without limitation the rights to use,
10+
copy, modify, merge, publish, distribute, sublicense, and/or sell
11+
copies of the Software, and to permit persons to whom the
12+
Software is furnished to do so, subject to the following
13+
conditions:
14+
15+
The above copyright notice and this permission notice shall be
16+
included in all copies or substantial portions of the Software.
17+
18+
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
19+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
20+
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
21+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
22+
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
23+
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
24+
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
25+
OTHER DEALINGS IN THE SOFTWARE.

README.md

+10-5
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,15 @@
11
# Permanent rclone QA Resources
22

3-
This project is intended for testing the [SFTP service](https://github.com/PermanentOrg/sftp-service). This repository
4-
will have a bunch of helper tools in a separate repository, whose purpose is just to help us do manual QA, and then
5-
later that gets turned into automated tests in the SFTP service repository. Refer to our discussions
6-
[here](https://chat.opentechstrategies.com/#narrow/stream/73-Permanent/topic/QA/near/155527)
3+
This code generates sample data for testing the [SFTP
4+
service](https://github.com/PermanentOrg/sftp-service) for
5+
[Permanent.org](Permanent.org). It creates test trees containing
6+
files with various pathological names, which are meant to be uploaded
7+
to Permanent via [rclone](https://rclone.org/) using the SFTP
8+
protocol.
9+
10+
For more context, see [this
11+
discussion](https://chat.opentechstrategies.com/#narrow/stream/73-Permanent/topic/QA/near/155527).
712

813
## License
914

10-
This code and data is licensed under [CC0v1.0](LICENSE).
15+
This code and data is open source under the [MIT license](LICENSE).

generate-tree.py

+100
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
#!/usr/bin/env python3
2+
# -*- coding: utf-8 -*-
3+
4+
# Copyright (c) 2023 Open Tech Strategies, LLC
5+
#
6+
# Permission is hereby granted, free of charge, to any person obtaining a copy
7+
# of this software and associated documentation files (the "Software"), to deal
8+
# in the Software without restriction, including without limitation the rights
9+
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
10+
# copies of the Software, and to permit persons to whom the Software is
11+
# furnished to do so, subject to the following conditions:
12+
#
13+
# The above copyright notice and this permission notice shall be included in all
14+
# copies or substantial portions of the Software.
15+
#
16+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
17+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
18+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
19+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
20+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
21+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
22+
# SOFTWARE.
23+
24+
"""Create directory 'test-tree' containing files with pathological names.
25+
(If 'test-tree' already exists, delete it and recreate it.)
26+
You must run this from this directory because sample source data is here.
27+
"""
28+
29+
import os
30+
import sys
31+
import shutil
32+
33+
def main():
34+
test_tree_top = "test-tree"
35+
36+
# Filename components.
37+
front = "front"
38+
back = "back"
39+
text_ext = "txt"
40+
image_ext = "png"
41+
42+
# Ensure that an empty test-tree/ dir exists.
43+
if os.path.exists(test_tree_top):
44+
shutil.rmtree(test_tree_top)
45+
os.makedirs(test_tree_top)
46+
47+
# Create all the files in test-tree/.
48+
#
49+
# A really rigorous test would try every UTF-8 character. We're
50+
# not going to do that, though: it would be too many files and
51+
# take too long to upload/download over the network. But we will
52+
# cover all the funny stops on the 7-bit ASCII railroad at least.
53+
# We try all the non-alphanumerics, including the lower-ASCII
54+
# control chars, at the front, middle, pre-extension end,
55+
# post-extension end, post-dot end, and raw end of the basename:
56+
#
57+
# %frontback.txt
58+
# front%back.txt
59+
# frontback%.txt
60+
# frontback.txt%
61+
# frontback.%
62+
# frontback%
63+
#
64+
# We also do the same with ".img" instead of ".txt".
65+
#
66+
# Note that we skip '/' because there's no way on a normal Unix
67+
# filesystem to create a file with '/' in its name. There might
68+
# be a way to persuade rclone to *send* such a filename, however,
69+
# and that's something we should try at some point.
70+
#
71+
# We skip 0 (NULL) because null bytes cause all sorts of trouble
72+
# -- which you would think would mean we *want* to include it, but
73+
# in practice Python's path-accepting functions don't like NULL
74+
# bytes either and it's quite unlikely to be found in a user's
75+
# filename... I think? I dunno. <shrug> If we need to test it,
76+
# we'll figure out how to get it through to disk below. For now,
77+
# I'm skipping it in the interests of not letting a perfect yak be
78+
# the enemy of a perfectly okay yak.
79+
for ch in (list(range(1, ord('/'))) # SOH through '.'
80+
+ list(range(ord(':'), ord('A'))) # ':" through '@'
81+
+ list(range(ord('['), ord('a'))) # '[" through '`'
82+
+ list(range(ord('{'), 128))): # '{" through DEL
83+
ch = chr(ch)
84+
for ext in (text_ext, image_ext,):
85+
for fname in (ch + front + back + "." + ext,
86+
front + ch + back + "." + ext,
87+
front + back + ch + "." + ext,
88+
front + back + "." + ch + ext,
89+
front + back + "." + ch,):
90+
if ext == "txt": # text: use the tiny sample text
91+
src = os.path.join("sample-sources", "text-small.txt")
92+
else: # image: use the PNG
93+
src = os.path.join("sample-sources", "single-pixel.png")
94+
dst = os.path.join(test_tree_top, fname)
95+
# sys.stderr.write(f"DEBUG: creating {dst}\n")
96+
shutil.copy(src, dst)
97+
98+
99+
if __name__ == '__main__':
100+
main()

sample-sources/single-pixel.jpg

631 Bytes
Loading

sample-sources/single-pixel.png

95 Bytes
Loading

0 commit comments

Comments
 (0)