Development pipeline and Module APIs unified (#385)

1. Interface Unify
1.1 Abstract uniform training pipeline into modules:
    *Augmentor* : used to  augment the image training data. Basic class is `Model.BasicAugmentor`.
    *PreProcessor* : used to generate target heatmap from the augmented image data and the key point annotations for model to regression. Basic class is `Model.BasicPreProcessor`.
    *PostProcessor*: used to generated detected human body joints from the model predict heatmap. Basic class is `Model.BasicPostProcessor`.
    *Visualizer*: used to visualize the model predict heatmap and the humans detected together with the origin images. Basic class is `Model.BasicVisualizer`.
    The difference of the training procedure between different pose estimation methods are then divided into pre-processing handled by the PreProcessor, post-processing handled by the PostProcessor and visualizing handled by the Visualizer. Inherit the corresponding Basic class, implementing member functions according to the provided function protocals, and then use `Config` module to set these 3 custom modules to implement any pose estimation pipeline you want.
1.2 Abtract uniform model API protocal
    uniform model APIs including `forwarding`, `cal_loss` and `infer` and provide a basic class `Model.BasicModel` for model customization. Inherite the model basic class to implement any pose estimation model you want. 

2. Additional handy component
2.1 Metric Manager
    Introducing `Model.MetricManager` that provides `update` and `report` function to statistic loss values during training and generate report massages for logging.
2.2 Image Processor
    Introducing `Model.ImageProcessor` that provides useful interfaces to read images, pad and scale them for easily converting images into model input format.
2.3 Weight Examination
    Introducing weight examination APIs in Model module for model, npz file and npz_dict file to easily exam the model weights.

3. Issue Fix
3.1 Python Demo
    Providing `python_demo.py` as a python demo to replace the old problematic demo program `infer.py`.  python_demo.py is used to easily try the npz model weights (both pretrained model weights or weights trained by users themselves) and also demonstraste the usage of PostProcessor, Visualizer and ImageProcessor modules. 
3.2 Shape mismatch issues
    Fix the shape mismatch issue that occurs when loading the pretrained model weights. (The issue was introduced by version compatibility) 
3.3 Domain adapation
    Fix the domain adaptation loss calculation and optimization issue occurs in tensorflow tape scope. warp all the domain adaptation data pipeline into the Domainadapt_dataset class. Domain adaptation can be put into pratical usage now.
3.4 Parallel training
    Fit Kungfu new APIs for parallel and distributed training.
3.5 Other
    other known issues in Processor modules such as tensorflow eager tensor and numpy ndarray compatibility issue, pyplot value clip issue.

4.  Standarize
4.1 Logging info standardize
    Use standard file stream and std-out stream in logging module to output log, split the logging information of hyperpose into [DATA],[MODEL],[TRAIN] 3 parts to regulate the logging information.  Formate human body joints output string format
4.2 Channels format standardize.
    Adapt all the pre-processing, post-processing, and visualizing functions to accept `channel_first` data in default to make the system clearer.
4.3 Model weights format standardize.
    Adapat all the model weights loading and saving procedure default format from `npz` to `npz_dict` format to make model weight conversion and examination convenient. (`npz` format weights are ordered arrays while `npz_dict` format weights are dictionary, which is more convenient to locate and examine specific weight.)
4.4 Help information standardize.
  Add help informations about variable definition, object definition, development platform basic usage, development platform custom usage, additional features when constructing models.
4.5 Tidy up all 10 model backbones provided.
4.6 Tidy up custom APIs in Config module.

Author: Gyx-One

.gitignore

@@ -60,6 +60,12 @@ debug.*
 !docs/Makefile
 !docs/markdown/images/*
 
+test_cmd.txt
+
 
 /.build
+test_dir
+*_save_dir
+cvt_dir
+test_cmd.txt
 

eval.py

@@ -58,7 +58,7 @@
                         help='number of visible evaluation')
     parser.add_argument('--multiscale',
                         type=bool,
-                        default=False,
+                        default=True,
                         help='enable multiscale_search')
 
 

export_pb.py

@@ -72,7 +72,7 @@ def analyze_inputs_outputs(graph):
 
     export_batch_size=args.export_batch_size
     export_h,export_w=args.export_h,args.export_w
-    print(f"{export_batch_size = }\t{export_h = }\t{export_w = }")
+    print(f"export_batch_size={export_batch_size}\texport_h={export_h}\texport_w={export_w}")
     input_path=f"{config.model.model_dir}/newest_model.npz"
     output_dir=f"{args.output_dir}/{config.model.model_name}"
     output_path=f"{output_dir}/frozen_{config.model.model_name}.pb"

hyperpose/Config/__init__.py

@@ -10,12 +10,17 @@
 
 #default train
 update_train.optim_type=OPTIM.Adam
+update_train.kungfu_option = KUNGFU.Sync_avg
 
 #defualt model config
 update_model.model_type=MODEL.Openpose
 #userdef model
-update_model.userdef_parts=None
-update_model.userdef_limbs=None
+update_model.custom_parts = None
+update_model.custom_limbs = None
+update_model.custom_augmentor = None
+update_model.custom_preprocessor = None
+update_model.custom_postprocessor = None
+update_model.custom_visualizer = None
 
 #default dataset config
 #official dataset
@@ -56,7 +61,7 @@ def get_config():
         an edict object contains all the configuration information.
 
     '''
-    #import basic configurations
+    # import basic configurations
     if(update_model.model_type==MODEL.Openpose):
         from .config_opps import model,train,eval,test,data,log
     elif(update_model.model_type==MODEL.LightweightOpenpose):
@@ -67,15 +72,15 @@ def get_config():
         from .config_ppn import model,train,eval,test,data,log
     elif(update_model.model_type==MODEL.Pifpaf):
         from .config_pifpaf import model,train,eval,test,data,log
-    #merge settings with basic configurations
+    # merge settings with basic configurations
     model.update(update_model)
     train.update(update_train)
     eval.update(update_eval)
     test.update(update_test)
     data.update(update_data)
     log.update(update_log)
     pretrain.update(update_pretrain)
-    #assemble configure
+    # assemble configure
     config=edict()
     config.model=model
     config.train=train
@@ -84,7 +89,7 @@ def get_config():
     config.data=data
     config.log=log
     config.pretrain=pretrain
-    #path configure
+    # path configure
     import tensorflow as tf
     import tensorlayer as tl
     tl.files.exists_or_mkdir(config.model.model_dir, verbose=True)  # to save model files 
@@ -93,18 +98,78 @@ def get_config():
     tl.files.exists_or_mkdir(config.test.vis_dir, verbose=True)  # to save visualization results
     tl.files.exists_or_mkdir(config.data.vis_dir, verbose=True)  # to save visualization results
     tl.files.exists_or_mkdir(config.pretrain.pretrain_model_dir,verbose=True)
-    #device configure
-    #FIXME: replace experimental tf functions when in tf 2.1 version
+    # device configure
+    # FIXME: replace experimental tf functions when in tf 2.1 version
     tf.debugging.set_log_device_placement(False)
     tf.config.set_soft_device_placement(True)
     for gpu in tf.config.experimental.get_visible_devices("GPU"):
         tf.config.experimental.set_memory_growth(gpu,True)
-    #limit the cpu usage when pretrain
-    #logging configure
+
+    # logging configure
+
+    # logging file path init
     tl.files.exists_or_mkdir(os.path.dirname(config.log.log_path),verbose=True)
     tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.WARN)
     tl.logging.set_verbosity(tl.logging.WARN)
-    return deepcopy(config)
+
+    # Info logging configure
+    info_logger = logging.getLogger(name="INFO")
+    info_logger.setLevel(logging.INFO)
+    # stream handler
+    info_cHandler = logging.StreamHandler()
+    info_cFormat = logging.Formatter("[%(name)s]: %(message)s")
+    info_cHandler.setFormatter(info_cFormat)
+    info_logger.addHandler(info_cHandler)
+    # file handler
+    info_fHandler = logging.FileHandler(config.log.log_path,mode="a")
+    info_fFormat = logging.Formatter("%(asctime)s [%(name)s] %(levelname)s: %(message)s")
+    info_fHandler.setFormatter(info_fFormat)
+    info_logger.addHandler(info_fHandler)
+
+    # Dataset logging configure
+    data_logger = logging.getLogger(name="DATA")
+    data_logger.setLevel(logging.INFO)
+    # stream handler
+    data_cHandler = logging.StreamHandler()
+    data_cFormat = logging.Formatter("[%(name)s] %(levelname)s: %(message)s")
+    data_cHandler.setFormatter(data_cFormat)
+    data_logger.addHandler(data_cHandler)
+    # file handler
+    data_fHandler = logging.FileHandler(config.log.log_path,mode="a")
+    data_fFormat = logging.Formatter("%(asctime)s [%(name)s] %(levelname)s: %(message)s")
+    data_fHandler.setFormatter(data_fFormat)
+    data_logger.addHandler(data_fHandler)
+
+    # Model logging configure
+    model_logger = logging.getLogger(name="MODEL")
+    model_logger.setLevel(logging.INFO)
+    # stream handler
+    model_cHandler = logging.StreamHandler()
+    model_cFormat = logging.Formatter("[%(name)s] %(levelname)s: %(message)s")
+    model_cHandler.setFormatter(model_cFormat)
+    model_logger.addHandler(model_cHandler)
+    # file handler
+    model_fHandler = logging.FileHandler(config.log.log_path,mode="a")
+    model_fFormat = logging.Formatter("%(asctime)s [%(name)s] %(levelname)s: %(message)s")
+    model_fHandler.setFormatter(model_fFormat)
+    model_logger.addHandler(model_fHandler)
+
+    # Train logging configure
+    train_logger = logging.getLogger(name="TRAIN")
+    train_logger.setLevel(logging.INFO)
+    # stream handler
+    train_cHandler = logging.StreamHandler()
+    train_cFormat = logging.Formatter("[%(name)s] %(levelname)s: %(message)s")
+    train_cHandler.setFormatter(train_cFormat)
+    train_logger.addHandler(train_cHandler)
+    # file handler
+    train_fHandler = logging.FileHandler(config.log.log_path,mode="a")
+    train_fFormat = logging.Formatter("%(asctime)s [%(name)s] %(levelname)s: %(message)s")
+    train_fHandler.setFormatter(train_fFormat)
+    train_logger.addHandler(train_fHandler)
+    
+    info("Configuration initialized!")
+    return config
 
 #set configure api
 #model configure api
@@ -235,12 +300,6 @@ def set_model_name(model_name):
     update_data.vis_dir=f"./save_dir/{update_model.model_name}/data_vis_dir"
     update_log.log_path= f"./save_dir/{update_model.model_name}/log.txt"
 
-def set_model_parts(userdef_parts):
-    update_model.userdef_parts=userdef_parts
-
-def set_model_limbs(userdef_limbs):
-    update_model.userdef_limbs=userdef_limbs
-
 #train configure api
 def set_train_type(train_type):
     '''set single_train or parallel train
@@ -280,9 +339,6 @@ def set_learning_rate(learning_rate):
     '''
     update_train.lr_init=learning_rate
 
-def set_save_interval(save_interval):
-    update_train.save_interval=save_interval
-
 def set_batch_size(batch_size):
     '''set the batch size in training
 
@@ -423,7 +479,8 @@ def set_dataset_filter(dataset_filter):
     '''
     update_data.dataset_filter=dataset_filter
 
-#log configure api
+# interval APIs
+# configure log interval
 def set_log_interval(log_interval):
     '''set the frequency of logging
 
@@ -439,10 +496,51 @@ def set_log_interval(log_interval):
     -------
     None
     '''
-    update_log.log_interval=log_interval
+    if(log_interval is not None):
+        update_log.log_interval=log_interval
+
+# configure save_interval
+def set_save_interval(save_interval):
+    if(save_interval is not None):
+        update_train.save_interval = save_interval
+
+# configure vis_interval
+def set_vis_interval(vis_interval):
+    if(vis_interval is not None):
+        update_train.vis_interval = vis_interval
+
+# custome module interfaces
+# custom parts
+def set_custom_parts(custom_parts):
+    update_model.custom_parts = custom_parts
+
+# custom limbs
+def set_custom_limbs(custom_limbs):
+    update_model.custom_limbs = custom_limbs
+
+# custom augmentor
+def set_custom_augmentor(augmentor):
+    update_model.augmentor = augmentor
+
+# custom preprocessor
+def set_custom_preprocessor(preprocessor):
+    update_model.preprocessor = preprocessor
+
+# custom postprocessor
+def set_custom_postprocessor(postprocessor):
+    update_model.postprocessor = postprocessor
+
+# custom visualizer
+def set_custom_visualizer(visualizer):
+    update_model.visualizer = visualizer
+
 
 def set_pretrain(enable):
     update_pretrain.enable=enable
 
 def set_pretrain_dataset_path(pretrain_dataset_path):
     update_pretrain.pretrain_dataset_path=pretrain_dataset_path
+
+def info(msg):
+    info_logger = logging.getLogger("INFO")
+    info_logger.info(msg)

hyperpose/Config/config_lopps.py

@@ -35,10 +35,11 @@
 train.weight_decay_factor = 2e-4
 train.train_type=TRAIN.Single_train
 train.vis_dir=f"./save_dir/{model.model_name}/train_vis_dir"
+train.vis_interval=1000
 
 #eval configuration
 eval =edict()
-eval.batch_size=22
+eval.batch_size=8
 eval.vis_dir=f"./save_dir/{model.model_name}/eval_vis_dir"
 
 #test configuration

hyperpose/Config/config_opps.py

@@ -22,7 +22,7 @@
 
 #train configuration
 train=edict()
-train.batch_size = 8
+train.batch_size = 4
 train.save_interval = 2000
 # total number of step
 train.n_step = 1000000

hyperpose/Config/define.py

@@ -3,12 +3,15 @@
 class BACKBONE(Enum):
     Default=0
     Mobilenetv1=1
-    Vgg19=2
-    Resnet18=3
-    Resnet50=4
-    Vggtiny=5
-    Mobilenetv2=6
-    Vgg16=7
+    Mobilenetv2=2
+    MobilenetDilated=3
+    MobilenetThin=4
+    MobilenetSmall=5
+    Vggtiny=6
+    Vgg19=7
+    Vgg16=8
+    Resnet18=9
+    Resnet50=10
 
 class MODEL(Enum):
     Openpose=0

hyperpose/Dataset/__init__.py

@@ -5,7 +5,8 @@
 from .mpii_dataset import MPII_dataset
 from .mscoco_dataset import MSCOCO_dataset
 from .imagenet_dataset import Imagenet_dataset
-from .common import imread_rgb_float,imwrite_rgb_float
+from .dmadapt_dataset import Domainadapt_dataset
+from .common import log_data as log
 
 def get_dataset(config):
     '''get dataset object based on the config object
@@ -49,7 +50,7 @@ def get_dataset(config):
     model_type=config.model.model_type
     dataset_type=config.data.dataset_type
     if(dataset_type==DATA.MSCOCO):
-        print("using MSCOCO dataset!")
+        log("Using MSCOCO dataset!")
         if(model_type==MODEL.LightweightOpenpose or model_type==MODEL.MobilenetThinOpenpose or model_type==MODEL.Openpose):
             from .mscoco_dataset.define import opps_input_converter as input_kpt_cvter
             from .mscoco_dataset.define import opps_output_converter as output_kpt_cvter
@@ -62,7 +63,7 @@ def get_dataset(config):
         dataset=MSCOCO_dataset(config,input_kpt_cvter,output_kpt_cvter)
         dataset.prepare_dataset()
     elif(dataset_type==DATA.MPII):
-        print("using MPII dataset!")
+        log("Using MPII dataset!")
         if(model_type==MODEL.LightweightOpenpose or model_type==MODEL.MobilenetThinOpenpose or model_type==MODEL.Openpose):
             from .mpii_dataset.define import opps_input_converter as input_kpt_cvter
             from .mpii_dataset.define import opps_output_converter as output_kpt_cvter
@@ -72,18 +73,18 @@ def get_dataset(config):
         dataset=MPII_dataset(config,input_kpt_cvter,output_kpt_cvter)
         dataset.prepare_dataset()
     elif(dataset_type==DATA.USERDEF):
-        print("using user-defined dataset!")
+        log("Using user-defined dataset!")
         userdef_dataset=config.data.userdef_dataset
         dataset=userdef_dataset(config)
     elif(dataset_type==DATA.MULTIPLE):
-        print("using multiple-combined dataset!")
+        log("Using multiple-combined dataset!")
         combined_dataset_list=[]
         multiple_dataset_configs=config.data.multiple_dataset_configs
-        print(f"total {len(multiple_dataset_configs)} datasets settled, initializing combined datasets individualy....")
+        log(f"Total {len(multiple_dataset_configs)} datasets settled, initializing combined datasets individualy....")
         for dataset_idx,dataset_config in enumerate(multiple_dataset_configs):
-            print(f"initializing combined dataset {dataset_idx},config:{dataset_config.data}...")
+            log(f"Initializing combined dataset {dataset_idx},config:{dataset_config.data}...")
             combined_dataset_list.append(get_dataset(dataset_config))
-        print("initialization finished")
+        log("Initialization finished")
         dataset=Multi_dataset(config,combined_dataset_list)
     else:
         raise NotImplementedError(f"invalid dataset_type:{dataset_type}")
@@ -93,12 +94,15 @@ def get_dataset(config):
 def get_pretrain_dataset(config):
     return Imagenet_dataset(config)
 
+def get_domainadapt_dataset(config):
+    return Domainadapt_dataset(config.domainadapt_train_img_paths)
+
 def enum2dataset(dataset_type):
     if(dataset_type==DATA.MSCOCO):
         return MSCOCO_dataset
     elif(dataset_type==DATA.MPII):
         return MPII_dataset
     elif(dataset_type==DATA.MULTIPLE):
-        raise NotImplementedError("multiple dataset shouldn't be nested!")
+        raise NotImplementedError("Multiple dataset shouldn't be nested!")
     else:
         raise NotImplementedError("Unknow dataset!")