fix: cache now can correctly discard old version

Author: timzaak

docs/develop/change-log.md

@@ -1,22 +1,28 @@
 # Change Log
-### Version 2.1.0(client:2.0.1)
+
+### Version 2.1.1
+
+- [ ] fix:
+
+### Version 2.1.0
+
 - [x] fix: upload file api is not correct.
 - [x] fix: command line compile successfully.
-- [x] feat: add `_version` api to get domain current version.
-- [ ] feat: support serving multiple spa in one domain.
+- [x] feat: support serving multiple spa in one domain.
 - [x] chore: js client sdk add node.js version restriction.
-- [x] chore: it's now used in production environment! 
+- [x] chore: it's now used in production environment!
 
 ### Version 2.0.0(client:2.0.0)
+
 - [x] move project from timzaak to ForNet
 - [x] upgrade warp and other dependencies
 - [x] upgrade vitePress to 1.0.1
 - [x] change js sdk implementation from Rust to typescript
 - [x] doc: rewrite to support new doc
 - [x] cicd: remove client docker, server s3 docker.
 
-
 ### Version v1.2.6(client:v0.1.4)
+
 - [x] chore(break change): change server docker config and binary location
 - [x] feat: add cron job to delete deprecated version
 - [x] feat(with client): delete deprecated version to save storage
@@ -26,16 +32,19 @@
 - [x] doc: add algolia search, thanks for algolia company!
 
 ### Version 1.2.5(client v0.1.3)
+
 - [x] build: add docker image cache for (spa-client|spa-server)-docker-cd.yml to speed cd process
 - [x] doc: use VitePress to rebuild docs, ready to get the world known it
 - [x] build: add CD for doc release
 - [x] feat: support multiple config for different domain (break change for config file)
 - [x] feat: support multiple ssl
-- [ ] ~~fix: disable put online domain which does not have correct ssl in server when https opened.~~(need to confirm if it's a bug?)
+- [ ] ~~fix: disable put online domain which does not have correct ssl in server when https opened.~~(need to confirm if
+  it's a bug?)
 - [x] fix: fix wrong check when release new domain
 - [x] fix(js-client): npm package error
 
 ### Version 1.2.4(client:v0.1.1)
+
 - [x] release commandline of spa-client for mac/ios/linux (by GitHub Actions), put them with GitHub release page
 - [x] fix possible bugs about uploading and spa-client(-js)
 - [x] build: release docker image by GitHubActions
@@ -44,44 +53,56 @@
 - [x] improve: add debug log for spa-server request
 
 ### version 1.2.3(client:v0.1.0)
+
 - [x] admin server export http api to accept files to local file system
 - [x] add client to sync local files to admin server（retry support）
-- [ ] ~~release server/client to crate~~ [crate needs dep version, need replace warp firstly](https://github.com/rust-lang/cargo/issues/1565)
+- [ ] ~~release server/client to
+  crate~~ [crate needs dep version, need replace warp firstly](https://github.com/rust-lang/cargo/issues/1565)
 - [x] doc about how to use with shell client
+
 #### add js plugin
+
 - [x] add js wrapper for spa-client
 - [x] and example/test frontend repo
 - [x] doc about how to use with js client
 - [x] release js wrapper to npm.org
 
 ### version 1.2.2
+
 - [x] cache File `Range` Header support
-- [ ] ~~drop self maintained `Warp`(copy out needed code from Warp)~~ (so much code from warp/fs, I give up after try, will wait Warp release proper version)
+- [ ] ~~drop self maintained `Warp`(copy out needed code from Warp)~~ (so much code from warp/fs, I give up after try,
+  will wait Warp release proper version)
 - [x] `HEAD` request support or drop(support, don't need to do anything)
 
 ### version 1.2.1
+
 - [x] more log for debug and trace
 - [x] basic CORS
-- [x] compress regression support(~~if client don't send accept-encoding header(including gzip), will send back data from file instead of cache~~ improved by v1.2.3)
+- [x] compress regression support(~~if client don't send accept-encoding header(including gzip), will send back data
+  from file instead of cache~~ improved by v1.2.3)
 - [x] hot reload web static server(use SO_REUSEPORT *nix api, so it may be wrong with Windows).
 - [ ] ~~different config(cors/cache strategy/https and so on) for different domain.~~ (if this is needed?)
 
 ### version 1.1.x
+
 - [x] more doc(how to update static files)
 - [x] rewrite Dockerfile to reduce docker image size
 - [x] cache improve(big file ignore config option and if-range header support)
 - [x] header`cache-control` for client cache
 - [ ] ~~header `etag` for client cache~~ [warp #462](https://github.com/seanmonstar/warp/issues/462)
 - [x] 80 redirect to 443 config option
-- [x] compression for js/icon/json/css/html (only support gzip algo, only compress cached files, and ~~will occur error when client don't support gzip~~(fix @ v1.2))
+- [x] compression for js/icon/json/css/html (only support gzip algo, only compress cached files, and ~~will occur error
+  when client don't support gzip~~(fix @ v1.2))
 
 ### version 1.0.x
+
 - [x] 80 and 443 both support
 - [ ] ~~compression~~ done @ v1.2.0.
 - [ ] ~~multiple tls support~~ the feature may do not need.
 - [x] cache file(cache all files in memory without LRU)
 
 ### before release
+
 - [x] very simple http1 spa server
 - [x] very simple admin server(http api)
 - [x] ssl(including wildcard domain ssl)

docs/index.md

@@ -3,24 +3,29 @@ sidebarDepth:2
 ---
 
 # What is spa-server?
+
 spa-server is to provide a static web http server with cache and hot reload.
-It supports multiple config for different domain, and has a client tool(npm package, command line) to help upload static web files to server. 
+It supports multiple config for different domain, and has a client tool(npm package, command line) to help upload static
+web files to server.
 
 ::: info Need Feedback
-sap-server features have been done, we are willing to get your feedback, fell free to open [issues](https://github.com/fornetcode/spa-server/issues).
+sap-server features have been done, we are willing to get your feedback, fell free to
+open [issues](https://github.com/fornetcode/spa-server/issues).
 :::
 
-
 ## Motivation
-In my company, every single page application needs a nginx docker image, as time long, there containers takes lots of resources
-of memory and storage, and these nginx don't have proper config. 
 
-So I tried to develop a static web server to solve above problem, and create a client tool `spa-client` to help users to release
+In my company, every single page application needs a nginx docker image, as time long, there containers takes lots of
+resources
+of memory and storage, and these nginx don't have proper config.
+
+So I tried to develop a static web server to solve above problem, and create a client tool `spa-client` to help users to
+release
 SPA.
 
 ## Feature
 
-- Built with Hyper and Warp, fast and small! 
+- Built with Hyper and Warp, fast and small!
 - SSL with Rustls.
 - Memory cache, client cache and compression(gzip).
 - Static web version control, you can regress or release new version easily.
@@ -29,4 +34,4 @@ SPA.
 - Http auto redirect to https.
 - Docker support(compressed size: 32M).
 - Provide command line/npm package to deploy spa.
-- Multiple configs for different domain.
+- Multiple configs for different domain and Multiple SPA in on domain.

server/src/domain_storage.rs

@@ -1,5 +1,5 @@
 use crate::file_cache::{CacheItem, FileCache};
-use anyhow::{anyhow, Context};
+use anyhow::{anyhow, bail, Context};
 use dashmap::DashMap;
 use lazy_static::lazy_static;
 use md5::{Digest, Md5};
@@ -80,11 +80,13 @@ impl DomainStorage {
                                 let data = cache.cache_dir(
                                     domain_dir_name,
                                     Some(sub_path.as_str()),
+                                    max_version,
                                     &path_buf,
                                 )?;
                                 cache.update(
                                     domain_dir_name.to_string(),
                                     Some(sub_path.as_str()),
+                                    max_version,
                                     data,
                                 );
                                 match domain_version.get_mut(domain_dir_name) {
@@ -119,8 +121,9 @@ impl DomainStorage {
                             let path_buf = path_prefix_buf
                                 .join(domain_dir_name)
                                 .join(max_version.to_string());
-                            let data = cache.cache_dir(domain_dir_name, None, &path_buf)?;
-                            cache.update(domain_dir_name.to_string(), None, data);
+                            let data =
+                                cache.cache_dir(domain_dir_name, None, max_version, &path_buf)?;
+                            cache.update(domain_dir_name.to_string(), None, max_version, data);
                             domain_version.insert(
                                 domain_dir_name.to_string(),
                                 DomainMeta::OneWeb(path_buf, max_version),
@@ -288,8 +291,8 @@ impl DomainStorage {
                 }
             };
             let path = if path == "" { None } else { Some(path) };
-            let data = self.cache.cache_dir(host, path, &new_path)?;
-            self.cache.update(host.to_string(), path, data);
+            let data = self.cache.cache_dir(host, path, version, &new_path)?;
+            self.cache.update(host.to_string(), path, version, data);
             debug!(
                 "domain: {host} all keys:{:?}",
                 self.cache.get_all_keys(host)
@@ -568,6 +571,27 @@ impl DomainStorage {
                 ));
             }
             let mut p = self.get_version_path(&domain, version);
+            let (host, path) = match domain.split_once('/') {
+                Some(v) => v,
+                None => (domain.as_str(), ""),
+            };
+            let multiple = self.prefix.join(host).join(MULTIPLE_WEB_FILE_NAME);
+            if path != "" {
+                if !multiple.exists() {
+                    let parent = self.prefix.join(host);
+                    if !parent.exists() && fs::create_dir_all(&parent).is_err() {
+                        bail!(
+                            "create host directory {} failure",
+                            parent.display().to_string()
+                        )
+                    }
+                    if File::create_new(multiple).is_err() {
+                        bail!("files in same domain should not create at same time")
+                    }
+                }
+            } else if multiple.exists() {
+                bail!("This already has multiple SPA, should not upload single SPA at top path")
+            }
             fs::create_dir_all(&p)?;
             p.push(UPLOADING_FILE_NAME);
             File::create(p)?;
@@ -676,6 +700,7 @@ mod test {
     use crate::domain_storage::URI_REGEX_STR;
     use hyper::Uri;
     use regex::Regex;
+    use std::fs::File;
     use std::path::PathBuf;
     use std::str::FromStr;
 
@@ -731,4 +756,10 @@ mod test {
     //     assert!(path.join("usr/lib/pam/").is_dir());
     //     println!("{:?}", path.join("usr/lib/pam/").to_str());
     // }
+
+    // would crea
+    // #[test]
+    // fn test_create_file() {
+    //     File::create_new(PathBuf::from("/tmp/bccskwef")).unwrap();
+    // }
 }

server/src/file_cache.rs

@@ -104,39 +104,50 @@ impl FileCache {
         &self,
         domain: String,
         sub_path: Option<&str>,
+        version: u32,
         data: HashMap<String, Arc<CacheItem>>,
     ) {
-        let data = match sub_path {
-            Some(sub_path) => match self.data.get(&domain) {
-                Some(ref info) => {
-                    let mut new_hash_map: HashMap<String, Arc<CacheItem>> = info
-                        .value()
-                        .iter()
-                        .filter_map(|(v, k)| {
-                            if v.starts_with(sub_path) {
+        let data = match self.data.get(&domain) {
+            Some(ref info) => {
+                let mut new_hash_map: HashMap<String, Arc<CacheItem>> = info
+                    .value()
+                    .iter()
+                    .filter_map(|(v, k)| match sub_path {
+                        Some(sub_path) => {
+                            if v.starts_with(sub_path)
+                                && version > k.version
+                                && version - k.version > 2
+                            {
                                 None
                             } else {
                                 Some((v.clone(), k.clone()))
                             }
-                        })
-                        .collect();
-                    // inesrt before get would trigger deadlock. so move out insert function
-                    //drop(info);
-                    new_hash_map.extend(data);
-                    new_hash_map
-                    //self.data.insert(domain, new_hash_map);
-                }
-                None => data,
-            },
+                        }
+                        None => {
+                            if version > k.version && version - k.version > 2 {
+                                None
+                            } else {
+                                Some((v.clone(), k.clone()))
+                            }
+                        }
+                    })
+                    .collect();
+                // inesrt before get would trigger deadlock. so move out insert function
+                //drop(info);
+                new_hash_map.extend(data);
+                new_hash_map
+            }
             None => data,
         };
+
         self.data.insert(domain, data);
     }
 
     pub fn cache_dir(
         &self,
         domain: &str, //www.example.com
         sub_path: Option<&str>,
+        version: u32,
         path: &PathBuf,
     ) -> anyhow::Result<HashMap<String, Arc<CacheItem>>> {
         let prefix = path
@@ -147,6 +158,7 @@ impl FileCache {
         let conf = self.conf.get_domain_cache_config(domain);
         let expire_config = self.conf.get_domain_expire_config(domain);
         let result: HashMap<String, Arc<CacheItem>> = WalkDir::new(path)
+            .min_depth(1)
             .into_iter()
             .filter_map(|x| x.ok())
             .filter_map(|entry| {
@@ -208,6 +220,7 @@ impl FileCache {
                                     mime,
                                     meta: metadata,
                                     data: data_block,
+                                    version,
                                     expire: expire_config.get(&extension_name).cloned(),
                                 }),
                             )
@@ -239,6 +252,7 @@ impl FileCache {
     }
 }
 
+#[derive(Debug)]
 pub enum DataBlock {
     CacheBlock {
         bytes: Bytes,
@@ -263,6 +277,7 @@ pub struct CacheItem {
     pub meta: Metadata,
     pub data: DataBlock,
     pub mime: Mime,
+    pub version: u32,
     pub expire: Option<Duration>,
     //    pub etag: ETag,
 }
@@ -278,3 +293,18 @@ pub fn etag_calculate(meta: &Metadata, real_len: u64) -> anyhow::Result<ETag> {
     };
     Ok(etag)
 }*/
+
+#[cfg(test)]
+mod test {
+    use std::collections::HashMap;
+
+    #[test]
+    fn test_extend() {
+        let mut hash = HashMap::new();
+        hash.insert(1, 1);
+        let mut hash2 = HashMap::new();
+        hash2.insert(1, 2);
+        hash.extend(hash2);
+        println!("hash2 {:?}", hash.get(&1));
+    }
+}

tests/data/template/self.noti.link/27/1/1.html

@@ -1 +1 @@
-1 index
+1

tests/data/template/self.noti.link/27/1/index.html

@@ -1 +1 @@
-1
+1 index

tests/data/template/self.noti.link/27/2/index.html

@@ -1 +1 @@
-2
+2 index

tests/data/template/self.noti.link/27/3/index.html

@@ -1 +1 @@
-3
+3 index

tests/data/template/self.noti.link/27/4/3.html

@@ -1 +1 @@
-3
+4-3

tests/data/template/self.noti.link/27/4/index.html

@@ -1 +1 @@
-4
+4 index