PG-1393 Release notes RC

contrib/pg_tde/documentation/_resource/overrides/main.html

@@ -4,7 +4,7 @@
 {% extends "base.html" %}
 
 {% block announce %}
-  This is a Beta2 version of Percona Transparent Encryption extension and it is 
+  This is a Release Candidate of Percona Transparent Data Encryption extension and it is 
   <strong>not recommended for production environments</strong> yet. We encourage you to test it and <a href= "https://forums.percona.com/c/postgresql/pg-tde-transparent-data-encryption-tde/82">give your feedback</a>.
   This will help us improve the product and make it production-ready faster.
 {% endblock %}

contrib/pg_tde/documentation/docs/faq.md

@@ -75,14 +75,18 @@ You can use the following KMSs:
 
 HashiCorp Vault can also act as the KMIP server, managing cryptographic keys for clients that use the KMIP protocol. 
 
-Here’s how encryption of data files works:
+Let's break the encryption into two parts:
+
+### Encryption of data files 
 
 First, data files are encrypted with internal keys. Each file that has a different OID, has an internal key. For example, a table with 4 indexes will have 5 internal keys - one for the table and one for each index.	
 
 The initial decision on what file to encrypt is based on the table access method in PostgreSQL. When you run a `CREATE` or `ALTER TABLE` statement with the `USING tde_heap` clause, the newly created data files are marked as encrypted, and then file operations encrypt/decrypt the data. Later, if an initial file is re-created as a result of a `TRUNCATE` or `VACUUM FULL` command, the newly created file inherits the encryption information and is either encrypted or not. 
 
 The principal key is used to encrypt the internal keys. The principal key is stored in the key management store. When you query the table, the principal key is retrieved from the key store to decrypt the table. Then the internal key for that table is used to decrypt the data.
 
+### WAL encryption
+
 WAL encryption is done globally for the entire database cluster. All modifications to any database within a PostgreSQL cluster are written to the same WAL to maintain data consistency and integrity and ensure that PostgreSQL cluster can be restored to a consistent state. Therefore, WAL is encrypted globally. 
 
 When you turn on WAL encryption, `pg_tde` encrypts entire WAL files starting from the first WAL write after the server was started with the encryption turned on.

contrib/pg_tde/documentation/docs/index.md

@@ -27,9 +27,9 @@ Learn more [what is Transparent Data Encryption](tde.md#how-does-it-work) and [w
 * System tables are currently not encrypted. This means that statistics data and database metadata are currently not encrypted.
 
 * `pg_rewind` doesn't work with encrypted WAL for now. We plan to fix it in future releases.
+* `pb_tde` Release candidate is incompatible with `pg_tde`Beta2 due to significant changes in code. There is no direct upgrade flow from one version to another. You must [uninstall](uninstall.md) `pg_tde` Beta2 first and then [install](install.md) and configure the new Release Candidate version.
 
 
-<i warning>:material-alert: Warning:</i> Note that introducing encryption/decryption affects performance. Our benchmark tests show less than 10% performance overhead for most situations. However, in some specific applications such as those using JSONB operations, performance degradation might be higher.
 
 ## Versions and supported PostgreSQL deployments
 

contrib/pg_tde/documentation/docs/release-notes/alpha1.md

@@ -0,0 +1,44 @@
+# pg_tde Alpha 1 (2024-03-28)
+
+`pg_tde` extension brings in [Transparent Data Encryption (TDE)](tde.md) to PostgreSQL and enables you to keep sensitive data safe and secure.
+
+[Get started](../install.md){.md-button}
+
+
+!!! important
+
+    This version of Percona Transparent Data Encryption extension **is 
+    not recommended for production environments yet**. We encourage you to test it and [give your feedback](https://forums.percona.com/c/postgresql/pg-tde-transparent-data-encryption-tde/82).
+
+    This will help us improve the product and make it production-ready faster.
+
+
+## Release Highlights
+
+The Alpha1 version of the extension introduces the following key features:
+
+* You can now rotate principal keys used for data encryption. This reduces the risk of long-term exposure to potential attacks and helps you comply with security standards such as GDPR, HIPAA, and PCI DSS.
+
+* You can now configure encryption differently for each database. For example, encrypt specific tables in some databases with different encryption keys while keeping others non-encrypted.
+
+* Keyring configuration has undergone several improvements, namely:
+
+    * You can define separate keyring configuration for each database
+    * You can change keyring configuration dynamically, without having to restart the server
+    * The keyring configuration is now stored in a catalog separately for each database, instead of a configuration file
+    * Avoid storing secrets in the unencrypted catalog by configuring keyring parameters to be read from external sources (file, http(s) request)
+
+## Improvements 
+
+* Renamed the repository and Docker image from `postgres-tde-ext` to `pg_tde`. The extension name remains unchanged
+* Changed the Initialization Vector (IV) calculation of both the data and internal keys
+
+## Bugs fixed
+
+* Fixed toast related crashes
+* Fixed a crash with the DELETE statement 
+* Fixed performance-related issues
+* Fixed a bug where `pg_tde` sent many 404 requests to the Vault server
+* Fixed сompatibility issues with old OpenSSL versions
+* Fixed сompatibility with old Curl versions 
+

contrib/pg_tde/documentation/docs/release-notes/beta.md

@@ -0,0 +1,25 @@
+# pg_tde Beta (2024-06-30)
+
+`pg_tde` extension brings in [Transparent Data Encryption (TDE)](tde.md) to PostgreSQL and enables you to keep sensitive data safe and secure.
+
+[Get started](../install.md){.md-button}
+
+!!! important
+
+    This version of Percona Transparent Data Encryption extension **is 
+    not recommended for production environments yet**. We encourage you to test it and [give your feedback](https://forums.percona.com/c/postgresql/pg-tde-transparent-data-encryption-tde/82).
+
+    This will help us improve the product and make it production-ready faster.
+
+## Release Highlights
+
+Starting with `pg_tde` Beta, the access method for `pg_tde` extension is renamed `tde_heap_basic`. Use this access method name to create tables. Find guidelines in [Test TDE](../test.md) tutorial.
+
+## Changelog
+
+* Fixed the issue with `pg_tde` running out of memory used for decrypted tuples. The fix introduces the new component `TDEBufferHeapTupleTableSlot` that keeps track of the allocated memory for decrypted tuples and frees this memory when the tuple slot is no longer needed.
+
+* Fixed the issue with adjusting a current position in a file by using raw file descriptor for the `lseek` function. (Thanks to user _rainhard_ for providing the fix)
+
+* Enhanced the init script to consider a custom superuser for the POSTGRES_USER parameter when `pg_tde` is running via Docker (Thanks to _Alejandro Paredero_ for reporting the issue)
+

contrib/pg_tde/documentation/docs/release-notes/beta2.md

@@ -0,0 +1,58 @@
+# pg_tde Beta 2 (2024-12-16)
+
+`pg_tde` extension brings in [Transparent Data Encryption (TDE)](tde.md) to PostgreSQL and enables you to keep sensitive data safe and secure.
+
+[Get started](../install.md){.md-button}
+
+
+!!! important
+
+    This version of Percona Transparent Data Encryption extension **is 
+    not recommended for production environments yet**. We encourage you to test it and [give your feedback](https://forums.percona.com/c/postgresql/pg-tde-transparent-data-encryption-tde/82).
+
+    This will help us improve the product and make it production-ready faster.
+
+## Release Highlights
+
+With this release, `pg_tde` extension offers two database specific versions:
+
+*  PostgreSQL Community version provides only the `tde_heap_basic` access method using which you can introduce table encryption and WAL encryption for data in the encrypted tables. Index data remains unencrypted.
+* Version for Percona Server for PostgreSQL provides the `tde_heap`access method. using this method you can encrypt index data in encrypted tables thus increasing the safety of your sensitive data. For backward compatibility, the `tde_heap_basic` method is available in this version too. 
+
+## Changelog
+
+The Beta 2 version introduces the following features and improvements:
+
+### New Features
+
+* Added the `tde_heap` access method with which you can now enable index encryption for encrypted tables and global WAL data encryption. To use this access method, you must install Percona Server for PostgreSQL. Check the [installation guide](../install.md)
+* Added event triggers to identify index creation operations on encrypted tables and store those in a custom storage. 
+* Added support for secure transfer of keys using the [OASIS Key Management Interoperability Protocol (KMIP)](https://docs.oasis-open.org/kmip/kmip-spec/v2.0/os/kmip-spec-v2.0-os.html). The KMIP implementation was tested with the PyKMIP server and the HashiCorp Vault Enterprise KMIP Secrets Engine. 
+
+### Improvements
+
+* WAL encryption improvements:
+
+   * Added a global key to encrypt WAL data in global space
+   * Added WAL key management
+
+* Keyring improvements:
+
+    * Renamed functions to point their usage for principal key management
+    * Improved keyring provider management across databases and the global space.
+    * Keyring configuration now uses common JSON API. This simplifies code handling and enables frontend tools like `pg_waldump` to read the code thus improving debugging.
+
+* The `pg_tde_is_encrypted` function now supports custom schemas in the format of `pg_tde_is_encrypted('schema.table');`
+* Changed the location of internal TDE files: instead of the database directory, now all files are stored in ` $PGDATA/pg_tde`
+* Improved error reporting when `pg_tde` is not added to the `shared_preload_libraries`
+* Improved memory usage of `tde_heap_basic `during sequential reads
+* Improved `tde_heap_basic` for select statements
+* Added encryption support for (some) command line utilities
+
+### Bugs fixed
+
+* Fixed multiple bugs with `tde_heap_basic` and TOAST records
+* Fixed various memory leaks
+
+
+

contrib/pg_tde/documentation/docs/release-notes/mvp.md

@@ -0,0 +1,8 @@
+# pg_tde MVP (2023-12-12)
+
+The Minimum Viable Product (MVP) version of `pg_tde` introduces the following functionality:
+
+* Encryption of heap tables, including TOAST
+* Encryption keys are stored either in Hashicorp Vault server or in local keyring file (for development) 
+* The key storage is configurable via separate JSON configuration files
+* Replication support

contrib/pg_tde/documentation/docs/release-notes/rc.md

@@ -0,0 +1,108 @@
+# pg_tde Release Candidate {{date.RC}}
+
+`pg_tde` extension brings in [Transparent Data Encryption (TDE)](tde.md) to PostgreSQL and enables you to keep sensitive data safe and secure.
+
+[Get started](../install.md){.md-button}
+
+
+## Release Highlights
+
+This release provides the following features and improvements:
+
+* **Improved performance with redesigned WAL encryption**. 
+
+    The approach to WAL encryption has been redesigned. Now, `pg_tde` encrypts entire WAL files starting from the first WAL write after the server was started with the encryption turned on. The information about what is encrypted is stored in the internal key metadata. This change improves WAL encryption flow with native replication and increases performance for large scale databases. 
+
+* **Default encryption key for single-tenancy**. 
+
+    The new functionality allows you to set a default principal key for the entire database cluster. This key is used to encrypt all databases and tables that do not have a custom principal key set. This feature simplifies encryption configuration and management in single-tenant environments where each user has their own database instance.
+
+* **Ability to change key provider configuration**
+
+    You no longer need to configure a new key provider and set a new principal key if the provider's configuration changed. Now can change the key provider configuration both for the current database and the entire PostgreSQL cluster using [functions](../functions.md#key-provider-management). This enhancement lifts existing limitations and is a native and common way to operate in PostgreSQL.
+
+* **Key management permissions**
+
+    The new functions allow you to manage permissions for global and database key management separately. This feature provides more granular control over key management operations and allows you to delegate key management tasks to different roles.
+
+* **Additional information about principal keys and providers**
+
+    The new functions allow you to display additional information about principal keys and providers. This feature helps you to understand the current key configuration and troubleshoot issues related to key management.
+
+* **`tde_heap_basic` access method deprecation**
+
+    The `tde_heap_basic` access method has limitations in encryption capabilities and affects performance. Also, it poses a potential security risk when used in production environments due to indexes remaining unencrypted. Considering all the above, we decided to deprecate this access method and remove it in future releases. Use the `tde_heap` access method instead that is available with Percona Server for PostgreSQL 17 - a drop-in replacement for PostgreSQL Community.
+
+## Upgrade considerations
+
+`pg_tde` Release Candidate is not backward compatible with `pg_tde` Beta2 due to significant changes in code. This means you cannot directly upgrade from one version to another. You must [uninstall](../uninstall.md) `pg_tde` Beta2 first and then [install](../install.md) and configure the new Release Candidate version.
+
+## Changelog
+
+### New Features
+
+* [PG-1234](https://perconadev.atlassian.net/browse/PG-1234) - Added functions for separate global and database key management permissions 
+
+* [PG-1255](https://perconadev.atlassian.net/browse/PG-1255) - Added functionality to delete key providers
+
+* [PG-1256](https://perconadev.atlassian.net/browse/PG-1256) - Added single-tenant support via the default principal key functionality
+
+* [PG-1258](https://perconadev.atlassian.net/browse/PG-1258) - Added functions to display additional information  about principal keys / providers 
+
+* [PG-1294](https://perconadev.atlassian.net/browse/PG-1294) - Redesigned WAL encryption
+
+* [PG-1303](https://perconadev.atlassian.net/browse/PG-1303) - Deprecated tde_heap_basic access method
+
+## Improvements
+
+* [PG-858](https://perconadev.atlassian.net/browse/PG-858) - Refactored internal/principal key LWLocks to make local databases inherit a global key provider
+
+* [PG-1243](https://perconadev.atlassian.net/browse/PG-1243) - Investigated performance issues at a specific threshold and large databases and updated documentation about handling hint bits
+
+* [PG-1310](https://perconadev.atlassian.net/browse/PG-1310) - Added access method enforcement via the GUC variable
+
+* [PG-1361](https://perconadev.atlassian.net/browse/PG-1361) - Fixed pg_tde relocatability
+
+* [PG-1380](https://perconadev.atlassian.net/browse/PG-1380) - Added support for `pg_tde_is_encrypted()` function on indexes and sequences
+
+### Bugs Fixed
+
+
+* [PG-821](https://perconadev.atlassian.net/browse/PG-821) - Fixed the issue with `pg_basebackup` failing when configuring replication
+
+* [PG-847](https://perconadev.atlassian.net/browse/PG-847) - Fixed the issue with `pg_basebackup` and `pg_checksum` throwing an error on files created by `pg_tde` when the checksum is enabled on the database cluster
+
+* [PG-1004](https://perconadev.atlassian.net/browse/PG-1004) - Fixed the issue with `pg_checksums` utility failing during checksum verification on `pg_tde` tables. Now `pg_checksum` skips encrypted relations by looking if the relation has a custom storage manager (SMGR) key
+
+* [PG-1373](https://perconadev.atlassian.net/browse/PG-1373) - Fixed the issue with potential unterminated strings by using the `memcpy()` or `strlcpy()` instead of the `strncpy()` function  
+
+* [PG-1378](https://perconadev.atlassian.net/browse/PG-1378) - Fixed the issue with toast tables created by the `ALTER TABLE` command not being encrypted 
+
+* [PG-1379](https://perconadev.atlassian.net/browse/PG-1379) - Fixed sequence and alter table handling in the event trigger
+
+* [PG-1222](https://perconadev.atlassian.net/browse/PG-1222) - Fixed the bug with  confused relations with the same `RelFileNumber` in different databases
+
+* [PG-1400](https://perconadev.atlassian.net/browse/PG-1400) - Corrected the pg_tde_change_key_provider naming in help 
+
+* [PG-1401](https://perconadev.atlassian.net/browse/PG-1401) - Fixed the issue with inheriting an encryption status during the ALTER TABLE SET access method command execution by basing a new encryption status only on the new encryption setting
+
+* [PG-1414](https://perconadev.atlassian.net/browse/PG-1414) - Fixed the error message wording when configuring WAL encryption by referencing to a correct function
+
+* [PG-1450](https://perconadev.atlassian.net/browse/PG-1450) - Fixed the `pg_tde_delete_key_provider()` function behavior when called multiple times by ignoring already deleted records.
+
+* [PG-1451](https://perconadev.atlassian.net/browse/PG-1451) -Fixed the issue with the repeating error message about inability to retrieve a principal key even when a user creates non-encrypted tables by checking the current transaction ID in both the event trigger start function and during a file creation. If the transaction changed during the setup of the current event trigger data, the event trigger is reset.
+
+* [PG-1473](https://perconadev.atlassian.net/browse/PG-1473) - Allowed only users with key viewer privileges to execute `pg_tde_verify_principal_key()` and `pg_tde_verify_global_principal_key()` functions
+
+* [PG-1474](https://perconadev.atlassian.net/browse/PG-1474) - Fixed the issue with the principal key reference corruption when reassigning it to a key provider with the same name by setting the key name in vault/kmip getters
+
+* [PG-1476](https://perconadev.atlassian.net/browse/PG-1476) - Fixed the issue with the server failing to start when WAL encryption is enabled by creating a new principal key for WAL in case only one default key exists in the database 
+
+* [PG-1479](https://perconadev.atlassian.net/browse/PG-1479), [PG-1480](https://perconadev.atlassian.net/browse/PG-1480) - Fixed the issue with the lost access to data after the global key provider change and the server restart by fixing the incorrect parameter order in default key rotation
+
+* [PG-1489](https://perconadev.atlassian.net/browse/PG-1489) - Fixed the issue with replicating the keys and key provider configuration by creating the `pg_tde` directory on the replica server.
+/browse/PG-1476) - Fixed the issue with the server failing to start when WAL encryption is enabled by creating a new principal key for WAL in case only one default key exists in the database 
+
+* [PG-1479](https://perconadev.atlassian.net/browse/PG-1479), [PG-1480](https://perconadev.atlassian.net/browse/PG-1480) - Fixed the issue with the lost access to data after the global key provider change and the server restart by fixing the incorrect parameter order in default key rotation
+
+* [PG-1489](https://perconadev.atlassian.net/browse/PG-1489) - Fixed the issue with replicating the keys and key provider configuration by creating the `pg_tde` directory on the replica server.