Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create NeptuneHook mechanism for automatic metadata logging #1

Merged
merged 33 commits into from
Dec 30, 2022
Merged
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
1c07e5b
add code for NeptuneHook
AleksanderWWW Dec 12, 2022
fb2581e
apply pre-commit suggestions
AleksanderWWW Dec 12, 2022
a028895
make logging final model conditional
AleksanderWWW Dec 14, 2022
35336ab
add e2e test
AleksanderWWW Dec 15, 2022
81bdc4d
add output dir to gitignore
AleksanderWWW Dec 15, 2022
821b560
add torch to pyproject and pip installation of detectron to workflow
AleksanderWWW Dec 15, 2022
d604e9e
make custom_run_id a local variable
AleksanderWWW Dec 15, 2022
2c75c71
remove windows from workflow
AleksanderWWW Dec 16, 2022
e7484b2
add run syncing before assertions
AleksanderWWW Dec 16, 2022
d9c2145
give time to upload files
AleksanderWWW Dec 16, 2022
b8ba20a
temporarily remove problematic assert to see how the rest goes
AleksanderWWW Dec 16, 2022
65326e1
explicitly pass run to NeptuneHook and call sync after training
AleksanderWWW Dec 16, 2022
567c8dc
sync active run, not the closed one
AleksanderWWW Dec 16, 2022
78f86e0
sync before stoping run
AleksanderWWW Dec 16, 2022
c502ad5
change connecting with custom id to run id
AleksanderWWW Dec 19, 2022
e6f21c4
force installing lower version of numpy
AleksanderWWW Dec 19, 2022
922ca97
add sync after uploading checkpoint
AleksanderWWW Dec 19, 2022
0ade49b
increase number of epochs
AleksanderWWW Dec 19, 2022
dfaec88
fix checkpointing error
AleksanderWWW Dec 19, 2022
38fc29c
add removing checkpoint files after train (+sync before)
AleksanderWWW Dec 20, 2022
a13e8ea
force lower version of fvcore
AleksanderWWW Dec 20, 2022
1efc286
force precise version of fvcore
AleksanderWWW Dec 20, 2022
de82c8c
fix typo
AleksanderWWW Dec 20, 2022
44304e4
bring back previous version specification of fvcore
AleksanderWWW Dec 20, 2022
5465adf
modularize the code - create private methods for individual activities
AleksanderWWW Dec 20, 2022
c2fdbe8
verify type of config in _log_config method
AleksanderWWW Dec 20, 2022
c786969
verityf type of run before creating base handler
AleksanderWWW Dec 22, 2022
0ff6907
fix checkpointing issue by uploading from stream
AleksanderWWW Dec 27, 2022
56e1a7a
apply review suggestions
AleksanderWWW Dec 28, 2022
27793e2
test accuracy, not loss
AleksanderWWW Dec 28, 2022
24091e5
remove TODOs
AleksanderWWW Dec 30, 2022
a876116
delete train images
AleksanderWWW Dec 30, 2022
b101f49
add train images to gitignore
AleksanderWWW Dec 30, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add run syncing before assertions
AleksanderWWW committed Dec 16, 2022
commit e7484b2dd44de9dced71bf6c5628e3a6ea7451d5
2 changes: 2 additions & 0 deletions tests/test_e2e.py
Original file line number Diff line number Diff line change
@@ -21,6 +21,8 @@ def test_e2e(cfg, trainer):

npt_run = neptune.init_run(custom_run_id=custom_run_id)

npt_run.sync()

assert npt_run.exists("training/config")

assert npt_run.exists("model/checkpoints/checkpoint_iter_0")