Tonic Structural release information

Bug fixes and other internal updates.

Fixed an issue where Microsoft Entra ID SSO groups were not retrieved if the membership in them was transitive.

Bug fixes and other internal updates.

Fixed an issue where PostgreSQL container artifacts failed to generate.

You can now specify the port that dotnet-monitor listens on within a worker. To set the port, use the environment setting TONIC_WORKER_DOTNET_MONITOR_PORT. Workers that are colocated on the same instance should be assigned different port bindings.

File connector - For custom Amazon S3 URLs, you can now indicate to trust the server certificate. To configure this, use the TONIC_AWS_S3_TRUST_SERVER_CERT environment setting.

MongoDB - Fixed an issue that caused layout problems for Privacy Hub.

Snowflake - Fixed an issue where a worker would sometimes become unresponsive after cancelling a data generation job.

Google BigQuery - Improved progress tracking for tearing down and recreating entities in the destination dataset.

Databricks - For new workspaces, changed the default Databricks runtime to 14.3.x-scala2.12.

When you configure a workspace to write output to a container repository, you can now provide a custom password for the destination database superuser.

File connector - The file upload now restricts accepted files to those that have supported file extensions.

MySQL - Improved resilience to transient issues when applying table constraints.

Fixed an issue where the first recommendation in the Recommended Generators by Sensitivity Type panel was cut off.

Fixed an issue where the generator recommendation preview sometimes failed with the message No Workspaces have been created yet.

Added API endpoints to enable and disable subsetting for a workspace.

Database View redesign and sensitivity confidence - On Database View, the Applied Generator column is now split into two columns. The Status column shows the current sensitivity status (Not Sensitive, At Risk, or Protected). From the Status column, you can display information about the detected sensitivity type and configure whether the column is sensitive. For at risk columns, the Status column also indicates the confidence level in the sensitivity detection. The Applied Generator column indicates the current generator assignment and allows you update the generator configuration.

Improved the detection accuracy for values in defined categories, such as gender.

Added the option to collect performance metrics when starting a data generation job that uses the Data Pipeline V2 processing.

When you write output to a container repository, you can now use assumed roles for the AWS credentials for Amazon ECR.

Snowflake - Improved performance when loading workspaces that connect to large databases.

Bug fixes and other internal updates.

Fixed an issue that caused the job details view to crash for some jobs that had warnings.

Added a new configuration setting TONIC_SENSITIVITY_SCAN_IGNORE_PII_TYPES. Accepts a comma-separated list of sensitivity type identifiers that the sensitivity scan should ignore.

Fixed an issue where missing data could cause the application to crash.

Bug fixes and other internal updates.

On Database View, when you create a sensitivity rule from the bulk edit function, Structural now automatically populates the data type.

Bug fixes and other internal updates.

Fixed an issue where generator application errors appeared to be encrypted.

Bug fixes and other internal updates.

Fixed an issue where generator application errors appeared to be encrypted.

Fixed an issue where generator suggestions caused the sensitivity scan to fail.

Fixed an issue that caused an error to sometimes occur when configuring a workspace to write output to Tonic Ephemeral.

Fixed an issue that caused display issues for Collection View rows that had applied generators.

Fixed an issue that prevented the application from redirecting after the workspace settings were saved.

Docker Compose files that are generated for output to a container repository now no longer contain an obsolete version specification.

Removed data science mode from Structural. The option to enable data science mode no longer appears in workspace settings.

International Address generator - The new International Address Generator can generate street addresses and postal codes for Canada, and postal codes for the United Kingdom.

Snowflake - Improved performance when loading workspaces that connect to large databases.

For new workspaces that support it, Data Pipeline V2 is now the default data generation option.

Structural now provides suggested generators for additional sensitivity types such as marital status.

The Protection Audit Trail now includes sensitivity rule events, including when a sensitivity rule is created, edited, or deleted.

Bug fixes and other internal updates.

Fixed an issue that caused a horizontal scrollbar to appear beneath the Privacy Hub sensitive column cards.

Bug fixes and other internal updates.

Structural no longer requires an external mount to monitor workers.

The new environment setting TONIC_DB_SCHEMA configures the Tonic Structural application database schema. Use the new environment setting TONIC_MIGRATION_ENABLE_LOGGING to enable or disable logging when applying migrations to the Tonic Structural application database. Note that if TONIC_DB_SCHEMA is a non-public value, then you must restart the API container. If there was existing data in the Tonic Structural application database, that data is not migrated. After the API is restarted and the migrations are applied, you can import a backup of the existing data.

When you create a custom sensitivity rule from the Bulk Edit panel of Database View, you can now choose to apply the generator preset to all of the workspace columns that match the new rule. The Save and Apply option replaces the previous Save and Scan option.

Bug fixes and other internal updates.

Oracle - When TONIC_ORACLE_SKIP_CREATE_DB is true, tables in the destination database are preserved. If a table filter is provided in TONIC_TABLE_WHITELIST, then destination tables that are not in the whitelist are preserved. Before this change, those tables were truncated.

Bug fixes and other internal updates.

MongoDB - Added the environment setting TONIC_MONGO_MASK_FIELD_IDS. When set to true, Structural assigns the Mongo ObjectId Key generator to ObjectId field names. Consistency is enabled. The default value is false.

Bug fixes and other internal updates.

Improved error handling for PII detectors.

Google BigQuery - Fixed an issue where some BigQuery workspaces did not release rows from memory in a timely manner.

MySQL - The new environment setting TONIC_MYSQL_USE_NATIVE_DUMP_TOOL, when set true, prefers the mysql native tools over the existing default of mariadb.

MySQL - You can now configure the following environment settings to override the Structural default behavior when a connection opens and a session is established:

• TONIC_MYSQL_NETWORK_READ_TIMEOUT

• TONIC_MYSQL_NETWORK_WRITE_TIMEOUT

• TONIC_MYSQL_WAIT_TIMEOUT

• TONIC_MYSQL_LOCK_WAIT_TIMEOUT

• TONIC_MYSQL_INNODB_LOCK_WAIT_TIMEOUT

Azure SSO - Added support for authenticating application service principals using the EntraID client-credentials flow. Service principals can access the Structural API. For configuration requirements, refer to the Azure/EntraID SSO configuration information in the Structural User Guide.

SQL Server - Fixed the copy of stored procedures that reference full-text indexes.

Structural now supports diagnostic logging for upsert data generation.

Snowflake - Fixed an issue where, when using the connection string and key-pair authentication options, the test connection button was disabled unless a password was entered.

Updated the legal text that is displayed for new account creation on self-hosted Structural instances.

For custom sensitivity rules, column matching rules are now always case insensitive. Previously, the column matching rules were always case sensitive.

SQL Server - Added support for:

• Full-Text Search catalogs and indexes
• Change tracking for database and table scopes

‍

Bug fixes and other internal updates.

Improved the free trial onboarding flow. Added to the available data connectors. Fixed an issue with creating a Google BigQuery workspace. Fixed a duplicate display issue.

File connector - Fixed an issue that a caused authorization failures when using Assume Role to authorize access to Amazon S3 from Structural Cloud.

Fixed an issue where after an import from a JSON file, Subsetting view did not immediately reflect the state of the workspace.

Spark - Removed support for Livy on Hive.

PostgreSQL - Fixed an issue where Structural failed to process tables that contained a generated column that referenced a user-defined type.

Scheduling data generation - From the Jobs view (renamed from Job History) for a workspace, you can now configure the data generation to run automatically on a schedule. The schedule consists of one or more cron expressions, along with the time zone to use for the schedule. The Structural API includes new endpoints to manage the job schedule.

MongoDB - Fixed an issue that caused connection errors for connections to a Mongo Atlas cluster from Tonic Cloud.

Bug fixes and other internal updates.

File connector - You can now specify a separate IAM role for the output location.

Create sensitivity rule from Database View - When you select the Database View bulk edit option for columns that have the same data type, do not have an assigned generator, and do not have a recommended generator, you now have the option to create a custom sensitivity rule. You can then immediately run a new sensitivity scan to catch matching columns.

Fixed an issue with the JSON Mask generator configuration panel where the example data did not update correctly.

Structural now displays a warning when the pre-job checks determine that the source database is on a newer major version than the destination database.

Salesforce - Rewrote the connector algorithm to avoid using sentinels, and to improve subset creation.

Snowflake - Fixed a regression introduced in v1213 that limited table parallelism for all data generations that use the V2 pipeline.

Oracle - Fixed an issue where for subsetting data generation, the Maximum Character Limit was not calculated properly.

Databricks - Structural now supports writing Identity columns to tables.File connector - You can now assign the Timestamp Shift and Date Truncation generators to Parquet date fields.

PostgresSQL - Removed the option to run PostgreSQL jobs using the older flow. All jobs now run with the Data Pipeline v2.

Snowflake

• Fixed an issue where after any row failed to parse, Structural did not process the rest of an unloaded table file.
• For AWS, added retry support to address some transient issues with Amazon S3.

Bug fixes and other internal updates.

The detector for city names now ignores misleading values that are not city names.

Bug fixes and other internal updates.

Bug fixes and other internal updates.

Updated the application to reflect the rename to Tonic Structural. Includes renaming the Tonic Settings view to Structural Settings.

From the Access Management tab of Structural Settings, users with permission to manage Structural access can now restore deleted users.

For a column that is part of a unique compound index, Structural now only suggests generators that can be used for unique columns.

Structural now detects SWIFT codes based on the format of the data in addition to the column name.

Fixed an issue where all subsetting WHERE clauses failed internally.

Databricks

• On self-hosted instances, you can now configure whether Structural creates the destination database schema for Databricks tables. The environment setting TONIC_DATABRICKS_SKIP_CREATE_DB indicates whether to skip the schema creation. The default is false. The environment setting TONIC_DATABRICKS_ENABLE_WORKSPACE_SKIP_CREATE_DB indicates whether to include the option in the workspace configuration, and use TONIC_DATABRICKS_SKIP_CREATE_DB to determine the default. The default is true. You can add these settings to the Environment Settings list on Structural Settings.
• When a cluster is resizing, and the workspace is configured with Use Databricks Job Cluster turned off, Structural data generation now waits for the resizing to complete.

File connector

• Fixed an issue where a copied local files workspace that contained Avro or Parquet files could not be loaded.
• Structural now supports authentication to Amazon S3 buckets using assumed roles for cross-account access.
• Fixed an issue where file connector workspaces did not load when they used an Amazon S3 source with environment-based authentication.

Snowflake

• Fixed a rare subsetting issue where processing a large table could cause the job to hang.
• Fixed an issue where Structural did not retry transient failures when it read unloaded files from a cloud storage provider.

The scheduled sensitivity scans are now daily instead of weekly. By default, the scans run every day at midnight. Structural scans the 10 workspaces that have the most recent activity. Activity is defined as either a user-initiated workspace event that is added to the Protection Audit Trail, or a data generation job.

On the details view for custom sensitivity rules, fixed an issue where the Edit Current Preset button was always disabled.

When a generation to Ephemeral fails, Structural job logs now include the Ephemeral logs and destination database pod logs.

For users who do not have permission to manage sensitivity rules, the Sensitivity Rules option now displays in a disabled state.

When you configure a workspace to write to a self-hosted Ephemeral instance, or to write to Ephemeral Cloud from a self-hosted Structural instance, the workspace configuration now includes an option to test the Ephemeral connection.

Yugabyte data connector - Structural now allows you to connect to databases on Yugabyte version 2024.1 and above. The Yugabyte data connector is available with a Professional or Enterprise license. It only supports Yugabyte SQL (YSQL).

When you configure a custom security rule, you can now create or edit the assigned generator preset. You can also use a workspace to preview the security rule results. The preview displays the matching columns for the selected workspace.

Structural can now detect the following additional sensitivity types:

• US driver’s license number
• Passport number
• Marital status
• GPS coordinates
• Non-birthday dates: admission date, discharge date, date of death
• US license plate

MySQL

• When the SQL mode ALLOW_INVALID_DATES is set, Structural now allows Passthrough for columns that contain invalid dates.

Snowflake

• For the Data Pipeline V2 process, reduced the frequency of polling cloud platforms for new unloaded files.

Fixed an issue on the webhook configuration panel where users could not click Save when the Message Body tab contained large property values.

Fixed an issue that caused the Notifications service to stop processing webhooks.

Improved the detection of name values to identify more specific types of names.

Amazon EMR

• Fixed an issue with previewing data after applying the Timestamp Shift Generator or Date Truncation Generator to timestamp columns with timezones.

Self-hosted instances can now schedule sensitivity scans to run automatically on a weekly basis. By default, the weekly scans are enabled and run each Sunday at midnight.

Structural can now detect the following additional sensitivity types:

• Money Amount
• Usernames

File connector

• Fixed a regression that made Table View unusable.

Oracle

• Fixed an issue that caused data generation to fail when subsetting with a date-based upstream table filter.

Salesforce

• Resolved issues where Salesforce workspaces became unrecoverable.
• Fixed issues with the use of session refresh tokens.

PostgreSQL

• Preserve Destination and Incremental tables no longer drop columns that reference user-defined types (UDTs) or extensions.
• Added support for the pgvector vector data type as a non-replaceable type that is always passed through to the destination database.

Snowflake on AWS

• Improved performance when copying passthrough tables when a workspace uses separate source and destination S3 buckets.

Structural can now detect the following additional sensitivity types that are defined by the HIPAA Safe Harbor method:

• Medical record numbers
• Health plan beneficiary numbers
• Account numbers
• Certificate and license numbers
• Web URLs
• Full face photographic images and similar images
• Biometric identifiers, including finger and voice prints

Removed the environment setting TONIC_SUBSETTING_CYCLE_BREAK_GREEDY_ALGORITHM. The greedy algorithm to compute the required cycle breaks for subsetting is no longer available.

Snowflake

• A new environment setting allows you to control whether Structural creates the destination database schema before it populates the destination data. By default, TONIC_SNOWFLAKE_SKIP_CREATE_DB is false, meaning that Structural creates the destination database and schema. If you set this to true, then Structural does not create the schema. You must create the destination database with the full schema. You can add TONIC_SNOWFLAKE_SKIP_CREATE_DB to the Environment Settings list on Tonic Settings.
• Improved performance when writing to Snowflake for de-identified tables and some Passthrough tables.

For post-job webhook URLs, you cannot use URLs that resolve to a private IPv4 range.

To provide the column name matching criteria for custom sensitivity rules, you can now use a regular expression.

The Structural sensitivity scan can now detect UK and Canada postal codes.

You can now use the Structural API to manage custom sensitivity rules.

When you configure a PostgreSQL or MySQL workspace to write the destination data to a container repository, you can now specify the name of the database.

Fixed an issue where the column sensitivity type was not updated when a later sensitivity scan detected a different type. Columns that are manually marked as sensitive are not affected.

Increased the number of column names that Structural uses to detect sensitivity types.

Amazon EMR

• Iceberg now correctly handles schemas that contain capitalized column names.

Salesforce

• The Continuous generator is now available.
• The Algebraic generator is now available.
• You can now use WHERE clauses in subsetting target table configuration.

When Structural detects a state abbreviation, it no longer identifies it as a full state name.

During a sensitivity scan, the value finders now look more holistically at both the data and the column name instead of assessing them individually.

Custom sensitivity rules - On self-hosted Enterprise instances, you can now configure custom sensitivity rules, which allow you to create your own sensitivity types. For each rule, you configure the general data type, text matching rules for the column name, and the recommended generator. Structural uses these rules during the sensitivity scan. Matching columns are included on the Recommended Generators by Sensitivity Type panel.

Toleration configuration for output to container repositories - Self-hosted customers who write output to a container repository can now set pod tolerations to enable pods to be scheduled on nodes that have taints. The tolerations are configured in environment settings. You can add these settings to the Environment Settings list on Tonic Settings.

MySQL

• Fixed an issue with subsetting for instances of MySQL that were deployed using Amazon Aurora where downstream tables were not populated properly.

When configuring a workspace to write output to an Ephemeral snapshot, you can now optionally configure the compute resources. By default, the resources are based on the size of the source database.

Sensitivity scans now detect name values more accurately.

Fixed an issue with certificate uploads for database settings.

Fixed an issue where the Structural application would hang after you created a workspace.

Shared logs are now transferred to an HTTPS endpoint instead of an Amazon S3 endpoint.

Amazon EMR

• Fixed an issue where Table View filters and table filters reported all WHERE clauses as invalid.

Amazon Redshift

• You can now configure a workspace to include or exclude specific schemas.

PostgreSQL

• When you choose to write output to an Ephemeral snapshot, you can now provide a custom configuration file.

Fixed an issue where sensitivity scans suggested generators based on substrings within a column name.

HTML is now removed from text in comment fields.

Fixed an issue where the XML Path generator did not work correctly.

A new environment setting, TONIC_SUBSETTING_CYCLE_BREAK_GREEDY_ALGORITHM, indicates whether to use a new, faster greedy algorithm to compute the required cycle breaks for subsetting. By default, the setting is false.

File connector

• Fixed an issue with uploading .txt files for local file workspaces.
• For CSV file groups, added an option to specify the encoding format of all files. If not specified, Structural attempts to detect the encoding automatically. When encoding cannot be determined, the automatic encoding detection now defaults to UTF-8 instead of windows-1252.

MongoDB

• Added an API endpoint to retrieve all of the field paths in a database.

MySQL

• Writing output to a container repository now works with multiple database schemas.
• Improved resilience to transient issues when copying tables.
• Fixed an issue with delayed retries of failed file uploads during data generation.

Oracle

• You can now write output data from an Oracle workspace to a Tonic Ephemeral snapshot.

Salesforce

• You can now provide connected app credentials in the workspace configuration. These fields are only displayed if the credentials are not configured in the TONIC_SALESFORCE_CONSUMER_KEY and TONIC_SALESFORCE_CONSUMER_SECRET environment settings.

Snowflake

• Fixed a regression where ALTER statements were inappropriately run through the GetDdl flow.

Improved the accuracy of name detection.

MongoDB

• On Collection View, for hybrid view, added a Filters panel. For single view, you can now filter fields by value.

Oracle

• The Data Generation Pipeline v2 for Oracle now supports subsetting.

Salesforce data connector - The Salesforce data connector is now available for self-hosted instances that have a Professional or Enterprise license. It is currently only available by request. To request access to the Salesforce data connector, contact Tonic.ai support.

Linking address columns for recommended generators - The recommended generators panel in Privacy Hub now indicates when address columns should be linked. The columns are displayed in groups. You then apply the recommended generators to all of the columns in the group, and the columns are automatically linked.

Other updates

Fixed an issue with subsetting. When processing upstream tables with nullable foreign keys that had no referenced key values to process, upstream filters were not applied.

Improved performance of the Conditional generator when using the IS IN operator.

The upsert option for workspaces is now out of beta.

Fixed an issue where the number of generators that are slow to compute was calculated incorrectly, which affected how we parallelized the generator processing.

The default value for the environment setting TONIC_ORACLE_DBLINK_ENABLED is changed to false. The plan is to eventually remove the feature.

Fixed an issue where the TONIC_DISABLE_IPV6 setting did not completely prevent services from binding to ipv6 addresses.

When applied to a numeric type column, the SSN generator now by default generates values without hyphens.

Amazon EMR

• Added support for the Iceberg framework in Glue. If you use the Iceberg framework, then in the Spark Configuration section of the workspace details, make sure to add the following configurations: spark.sql.catalog.glue_catalog, .warehouse, .catalog-impl, and .io-impl.

File connector

• Improved resilience to missing cloud files when configuring file groups, previewing data, and running generation jobs.

MongoDB

• Fixed an issue with creating virtual foreign keys in MongoDB workspaces.

MySQL

• Improved the performance of the destination database teardown step for databases that have many partitions.

Oracle

• You can now use the Data Pipeline V2 process to run data generation for an Oracle workspace. The Confirm Generation panel includes a toggle to enable or disable the new process. Note that you cannot use the new process when subsetting is enabled.

Snowflake

• Fixed a subsetting issue where extra downstream rows were included when no primary keys existed in the table relationship.

Helm charts for deploying Structural to Kubernetes are now published at quay.io/tonicai/structural in addition to GitHub.

From the recommended generators panel on Privacy Hub, you can now enable or disable self-consistency for all columns within a sensitivity category.

Fixed an issue in Table View that sometimes caused the column order to be incorrect.

Added an environment variable TONIC_DISABLE_IPV6 to the PyML container. When set to true, the container no longer listens on IPv6 addresses.

File connector

• Added support for Avro files.

MySQL

• Fixed an issue with validating subsetting target table WHERE clauses when the table or schema name contained special characters.

PostgreSQL

• Added limited support for ltree columns on versions older than 1.2. For tables where all columns are assigned the Passthrough generator, Structural copies the ltree data from the source database to the destination database. In tables that are de-identified, ltree columns are nullified in the destination database. If an ltree column is not nullable, then all of the columns in the table must be assigned the Passthrough generator.

Snowflake

• Added support for date and time columns that have a seconds precision of 0.
• Fixed an issue where the incorrect credentials were used when using separate source and destination buckets and credentials for Amazon S3.

SQL Server

• Fixed an issue introduced in v1178 where XML columns might be persisted as nvarchar columns in the destination database.

From the recommended generators panel on Privacy Hub, you can now enable or disable self-consistency for each column.

When you create a virtual foreign key, the nullability of the foreign key is now determined by the nullability of the foreign key column.

On Privacy Hub and the job details view, changed the label on the Download option to Reports and Logs.

Db2 for LUW

• Subsetting is now supported.

MySQL

• Fixed an issue where data generation failed when object identifiers included symbol characters.

PostgreSQL

• Workspaces with PostgreSQL 16 databases can now write output to a container repository.

Sending telemetry to Tonic.ai is now required and enforced. After 5 days of failed telemetry connections, a warning displays. If the connection issue persists for 15 days, data generation is restricted.

Structural Cloud users can now reset their passwords from the login page. To reset your password, click Forgot your password?.

You can now set the TONIC_POSTGRES_REFRESH_MATERIALIZED_VIEWS environment setting from the Environment Settings tab on Tonic Settings.

File connector

• Added support for AWS C2S Secret Regions.

SQL Server

• Improved the upsert logic to prevent jobs from hanging.

Added a new API endpoint to resolve all schema changes in a workspace. You can choose whether to resolve only conflicting changes, only notifications, or all of the schema changes.

Fixed an issue that caused the Tonic Structural PyML Service to be unreachable in IPV4-only containers.

Added a new conflicting schema change when a column that has an assigned generator becomes a foreign key. Foreign key columns must inherit the generator from the primary key.

Structural can now generate data with subsetting when a primary key table is truncated, as long as the foreign keys that reference the primary key are nullable.

Amazon Redshift

• Added support to pass through varbyte, geometry, and hllsketch types.

File connector

• For CSV, XML, and JSON files, fixed issues with the data preview in Database View and Table View. The preview no longer includes extra rows, and the preview now correctly reflects the Skip first N rows setting.
• Fixed the validation of file groups that only contained compressed files.

Snowflake

• For subsetting, added support for virtual foreign keys.

SQL Server

• Fixed an issue that caused data generations to fail for versions of SQL Server older than SQL Server 2016.

For the notifications image, replaced alpine with ubuntu.

File connector

• For a local files workspace, the job details view for a successful data generation now includes an option to download the transformed files that were produced by that job.
• For workspaces that use files from cloud storage, you can now include prefix patterns in the file group definition. You can also provide file extension filters. The file group then includes all of the files that match a prefix pattern and the file extension filters. The file group details now include the content type, file extension filter, and prefix patterns.
• For cloud storage workspaces, users who do not have permission to view all buckets can once again specify the bucket to view.
• For cloud storage workspaces, added a configuration option to only process new files during data generation. For existing file groups, the new configuration is off by default. For new file groups, the configuration is enabled by default.

MongoDB

• Fixed an issue where environment setting updates from the UI required a restart to take effect.

MySQL

• For a workspace that is configured to write output to Ephemeral, you can now provide a custom configuration file.

Snowflake

• Fixed an issue where the Subsetting tab disappeared after you edited a workspace.

SQL Server

• For versions 2017, 2019, and 2022, you can now configure workspaces to write output to a container repository.

During the free trial, Structural now displays next step highlights to indicate the next recommended action. When you hover over the recommended action, Structural displays an explanatory tooltip.

A new environment setting, TONIC_DB_MAX_POOL_SIZE, sets the connection pool size for the Structural application database. The default value is 3.

Fixed an issue where the preview data in the JSON Mask generator editor did not respect the applied table filter.

File connector

• For workspaces that connect to Amazon S3, you can now specify different credentials for the source and output locations.
• For cloud storage workspaces, fixed a regression in Tonic v1146 where the file explorer sometimes displayed incorrect results and files could not be added.
• You can now add Parquet files to a file connector file group, either from cloud storage or a local file system. You cannot select the following Parquet file types: HalfFloat, Struct, Union, Dictionary, Map, List, FixedSizeList, or arrays of any type. There also is no file preview for Parquet files.

MongoDB

• Fixed an issue that prevented the use of regex conditions in the conditional generator.
• You can now select the UUID Key generator as a sub-generator when configuring a composite generator.

PostgreSQL

• Fixed an issue where Preserve Destination tables sometimes were not preserved when data generation failed.

Fixed an issue in Table View where characters were sometimes represented inaccurately. For example, a lowercase x would become a multiplication symbol.

Fixed an issue where data generation to Tonic Ephemeral Cloud failed with the error "Ephemeral URL not found".

For a schema change that adds a new column, both the Schema Changes view and the API response now include the data type for the new column.

For the Structural free trial, Structural now displays a checklist for each workspace. There are slightly different checklists for database-based and file connector workspaces.

Added support to run Structural in dual-stack networks and IPv6-only network environments.

On the workspace details view, fixed an issue where an Ephemeral API key appeared to be populated when no value was provided.

Fixed an issue that caused data generation to Ephemeral to fail with "Ephemeral output must be configured".

Fixed an issue where data generation to Ephemeral failed after a first successful run.

Amazon Redshift

• You can now enable TLS for workspace data connections.

MySQL

• Fixed an authentication issue that prevented output from being written to an Ephemeral snapshot.

SQL Server

• You can now configure a SQL Server workspace to write output to an Ephemeral snapshot.

Writing output to a Tonic Ephemeral snapshot - For database types that Tonic Ephemeral supports (currently PostgreSQL and MySQL), you can now write the output to an Ephemeral user snapshot. This replaces the option to write the output to an Ephemeral database, except for workspaces in the Structural free trial. In Ephemeral, you can use the user snapshot to start new Ephemeral databases.

Other updates

For the UUID Key generator, added a new configuration option, Preserve Version and Variant. By default, the setting is turned off. When turned on, the version and variant bits from the source UUID are preserved in the output value. For the API, the new setting is preserveVersionAndVariant.

In the Tonic Structural free trial, the sample workspace now by default writes the output to a Tonic Ephemeral database.

Fixed an issue where vertical scrolling was sometimes blocked.

You can now configure the allowed SSL/TLS protocols and ciphers on the Tonic Web Server. To configure the protocols and ciphers, use the environment settings TONIC_WEB_SERVER_TLS_PROTOCOLS and TONIC_WEB_SERVER_TLS_CIPHERS.

File Connector

• Data generation jobs no longer fail when they encounter a configured file group file that no longer exists in the source cloud storage location.
• For local file workspaces, fixed a regression from Tonic v1136 where users could not upload additional local files into a file group.

MongoDB

• Fixed an issue that caused generations to fail when documents contained empty arrays or document fields.

MySQL

• Improved job warnings around missing tablespaces in the destination database.

The Structural API now includes endpoints to get and set the assigned table modes and table filters for a workspace.

Fixed an issue where the workspace audit trail displayed generator preset events that occurred before the workspace was created.

Fixed an issue where an error was returned when users tried to export selected files from a file group.

Improved error message when Structural cannot write output to Ephemeral because Ephemeral does not have a compatible base image for the database.

Databricks

• When a workspace is configured with an unsupported version of Databricks, the error message now suggests the supported Databricks versions.
• Updated the default spark_version to 14.3.

Google BigQuery

• Fixed an issue with applying some generators to numeric columns on large tables.

Oracle

• Fixed an issue where rows that contained NULL values in a VARCHAR2(1) column were dropped during data generation.

Output to a Tonic Ephemeral database - Tonic Ephemeral is a separate Tonic.ai product that allows you to create temporary databases. On Tonic Cloud, for data connectors that Ephemeral supports (currently PostgreSQL and MySQL), you can configure the workspace to write the destination data to an Ephemeral database. This is the default option for data connectors that Ephemeral supports.

The database belongs to your Ephemeral account. If you do not already have an Ephemeral account, then Tonic automatically creates a two-week Ephemeral free trial account for you. The Tonic data generation job details provide access to the database connection details.

Free trial checklist - During the free trial, the sample workspace now includes a checklist to help users get through the required steps to complete their first data generation.

Other updates

Free trial users can no longer use a public email address to create an account.

Fixed an issue where password reset links lead to a blank page.

Fixed an issue where pay-as-you-go users would see the countdown for a free trial.

In the sample workspace, fixed an issue where a faulty destination database template caused an error when a user tried to update it.

Google BigQuery

• Tonic can now de-identify snapshot tables.

Oracle

• For databases that use non-default database character sets, fixed an issue where rows that contained character data sometimes failed to be written to the destination database.
• For databases that use non-default database character sets, fixed an issue where passthrough CLOB data types had extra bytes inserted when they were copied to the destination database.

PostgreSQL

• If you do not use extensions, then the destination database no longer requires a super user.

Snowflake

• Subsetting is now supported on Snowflake workspaces. The table filtering option is still available.

You can now manually add selected environment settings to the Environment Settings list on Tonic Settings.

Improved the performance of data previews in the Tonic application.

For workspaces that write output data to a container repository, fixed an issue that prevented GAR credentials from being saved.

Google BigQuery

• Improved performance for data preview.

Snowflake

• The workspace configuration now includes a Trust Server Certificate option for the source and destination connections. When enabled, it indicates to bypass certificate revocation checks.
• Data generation jobs are now more resilient when they encounter views that have dependencies that Tonic does not have permissions for.
• Destination privileges can now exclude schema creation. If the user does not have schema creation permissions, then the schemas must already exist in the destination database.

SQL Server

• The upsert function now supports tables that have identity columns but do not have primary keys.

New Db2 for LUW data connector - Tonic now has a data connector for IBM Db2 for Linux, Unix, and Windows (Db2 for LUW). Tonic supports Db2 for LUW version 11.5.

Other updates

When the AI Synthesizer is used in a workspace, Tonic now verifies before data generation that the AI Synthesizer does not use more than the maximum allowed categories.

Amazon EMR

• Improved data generation performance when using Spark 2.4.0-3.2.x.Databricks
  
  
  • Fixed an issue in Databricks 11.3+ where a generator that was consistent on another column received modified input values if a generator was applied to the consistency column. Generators that are consistent on another column now correctly receive unmodified input values.
  • Improved data generation performance when using Spark 2.4.0-3.2.x.
  Google BigQuery
  
  
  • You can now de-identify views, except for views that are written in Google Legacy SQL. On the workspace settings view, to enable de-identification of views, toggle De-identify Views to the on position. You can then assign table modes and generators to the views. In the table lists, views are identified by (view) after the view name.
    
    For each view, Tonic:
    
    
    • Writes the de-identified data to a table called <view name>_tonic_table.
    • Creates a view that has the same name and metadata as the view in the source data, but is populated from the destination table.
  • Fixed an issue where Passthrough tables that had table constraints failed to copy to the destination database.
  • Fixed an issue where cloned tables did not appear in the Tonic application, but caused job failures. Cloned tables now display within Tonic and can be managed in the same way as any other table.
  Oracle
  
  
  • For Oracle 12, improved the schema remapping across multiple object types and configurations.
  • Fixed an issue where Tonic failed to copy database link objects that were configured using passwords. Tonic no longer processes database links. If required, users must manually create database links in the destination schema.
  • Fixed an issue where tables that contained computed columns dropped rows during data generation.
  Snowflake
  
  
  • You can now assign the JSON Mask generator to Snowflake variant columns.

Privacy Report PDF file

We added a new Privacy Report PDF that you can download from Privacy Hub and the job details view. The Privacy Report PDF contains a summary of the privacy ranking values, visualizations to summarize the workspace column privacy rankings based on the applied generators, and a summary table that contains the .csv Privacy Report data.

To accommodate the new file, on Privacy Hub and the job details page, the available downloads are combined into a Download menu.

Assigning recommended generators from Database View

On Database View, when an unprotected column has a recommended generator, the generator name tag now displays the type of sensitive data that was detected.

When you click the generator name tag, Tonic displays a panel that displays the sensitivity type, the recommended generator, and sample source and output data based on the recommended generator. The panel provides options to either apply or ignore the recommendation.

Other updates

Fixed an issue where changing the configuration of a generator preset did not accurately update the count of occurrences of the preset.

Oracle

• Fixed an issue with Oracle 12.1 and 12.2 where constraints that had an index creation statement failed to apply.
• Tonic can now de-identify external tables. External tables display in the Tonic application and can be assigned table modes and generators. During data generation, external tables are created as relational tables in the destination schema.

Fixed an issue with the Name generator where capitalization was not preserved if consistency was disabled.

For Table View, fixed an issue where the delete button to remove the generator assignment was sometimes hidden.

Oracle

• Tonic now supports Oracle Advanced Queues. Queue tables and queues are created in the destination database. Messages are not copied from the source database to the destination database. Queue subscribers are not currently supported. You must add them to the destination database manually.

PostgreSQL

• Tonic can now de-identify the ltree data type.

SQL Server

• Fixed an issue where indexes were not created on schema bound views.

Redesigned Database View

We redesigned Database View to improve the display and the filtering.

In the updated columns list, the Column column contains the schema, table, and column name, and the column data type. It provides access to the data preview option.

The Applied Generator column shows the applied generator. Applied Generator indicates when a column is unprotected, when the column is a primary or foreign key, and when the configuration overrides the parent workspace. If the table mode is not De-Identify, it shows the table mode. It provides access to the commenting option.

Filters other than the column name filter are moved under the Filters option. There are also new filters for the sensitivity type (the type of sensitive data that Tonic detected in the column) and whether the column has a recommended generator.

Privacy Report updates

In the Privacy Report, new column, Column Privacy Rank, indicates the privacy ranking for a column based on the assigned generator and generator configuration. The generator summary and generator reference include the possible privacy ranking values for each generator.

Added a new column, Tonic Detected Sensitivity, that indicates whether the Tonic sensitivity scan identified the column as sensitive. Renamed the Is Sensitive column to Current Sensitivity. Current Sensitivity indicates whether the column is currently marked as sensitive.

Also corrected an earlier issue with the order of the columns.

Other updates

Fixed an issue that caused all subset runs to record the percentage of rows in the subset as 100%. Subset runs that occur after updating to this version display the correct percentage.

The option to write output data to a container repository is out of beta.

Databricks

• For Databricks 13+, writing output to delta tables now uses the table’s liquid clustering configuration to cluster the insertion.
• Fixed an issue with loading tables into the Tonic application.

Google BigQuery

• Improved batch size calculations to reduce memory pressure during data generation jobs.
• Fixed an issue with applying the Timestamp Shift and Date Truncation generators on datetime or timestamp data when TONIC_GRPC_ENABLED was false.
• For the Timestamp Shift and Date Truncation generators, improved support for time values that include fractional seconds.
• Fixed an issue where the Character Substitution generator failed when assigned to a NUMERIC or BIGNUMERIC column.

Oracle

• Tonic now supports IDENTITY columns. Before this change, IDENTITYM columns caused errors during destination database creation.

For the Custom Categorical generator, you can now add a NULL value to the available custom category values. To indicate a NULL value, use the keyword {NULL}.

Made the following API updates to better accommodate users of the previous version of the API:

• jobs/{id}/workspace_snapshot now returns the WorkspaceDataModel object.
• Fixed an issue where workspace_snapshot could return empty replacements.
• Added a new endpoint, GET jobs/{id}/workspace_snapshot?api-version=v2023_07_00, that returns V17WorkspaceDataModel

Databricks

• When writing output to delta tables, the destination tables should now retain the TBLPROPERTIES from the source delta table, including 'delta.feature.allowColumnDefaults'.

Redesigned data model for generator assignments - The new version of the Tonic API includes a redesign of the data model for generator assignments. To use the previous version of the generator assignment data model, make sure that your API calls specify version 2023.07.0.

The generator assignment data model redesign includes the following changes:

• For all generators, moved the following fields to the metadata object in the link object:
  • presetId
  • generatorId
  • customValueProcessor
  • encryptionProcessor
• For the <value type>: Mask composite generators:
  • Moved pathExpression to the metadata object in the link object.
  • Removed the following fields from the link object:
    • subPresetId
    • subGeneratorId
    • customSubGeneratorValueProcessor
  • Added the following fields to the subGeneratorMetadata object under metadata:
    • presetId
    • generatorId
    • customValueProcessor
  Other updates
  
  Updated the sensitivity scan to better identify company and organization names and suggest the appropriate generator.
  
  Databricks
  
  
  • The Conditional Generator is now supported on version 11.3+.
  Google BigQuery
  
  
  • Fixed some cases where a copied View did not have its definition updated to reference resources within the destination dataset. This could result in failures when attempting to copy the View into the destination dataset.
  • Added support for Preserve Destination table mode.
  • Fixed an issue where sub-queries in table filter clauses did not work if TONIC_GRPC_ENABLED was set to true.
  MySQL
  
  
  • You can now enable diagnostic logging for a MySQL data generation job.
  PostgreSQL
  
  
  • Improved the destination database permissions check on Google Cloud SQL to handle additional superuser authentication setups.
  SQL Server
  
  
  • Data generation now preserves hidden properties of columns.

NOTE: v1074 was removed.

If your instance of Tonic is deployed on Docker, you can now use an external Kubernetes cluster to enable the option to write destination data to container artifacts.

You can now assign the Integer Key generator to a column with a decimal data type. The actual column values must still be integers.

Fixed an issue in Table View where an error displayed if you changed the selected table while the data was loading.

Databricks

• For cluster versions 11.3 and above, Tonic now displays accurate lists of available and suggested generators.

File connector

• Fixed an issue that prevented users from adding files to existing local file groups.

SQL Server

• Fixed an issue that caused errors when creating inline functions in the destination database.

The Enable Diagnostic Logging global permission is now granted to the built-in Account Admin permission set.

Databricks

• CREATE CATALOG or CREATE SCHEMA permissions are no longer required if the destination catalog or schema already exists.
• For Databricks 11.3 and higher, added support for the Random Double generator.
• For Databricks 11.3 and higher, added support for the IP Address generator.

Diagnostic logging for data generation - By default, Tonic now redacts sensitive data in data generation log files.

When users start a data generation or upsert job, if they have the new global permission Enable diagnostic logging, they can choose to enable diagnostic logging, which does not redact the logs. The Enable diagnostic logging permission is also required to download the diagnostic logs. By default, the permission is only granted to the Admin and Admin (Environment) global permission sets.

In addition to the option for individual jobs, there are environment settings that enable diagnostic logging for specific data connectors.

Other updates

In the Release Candidate version of the API, the response model for the GET /api/workspace/minimal endpoint has been updated for more straightforward de-serialization.

Fixed an issue where a non-unique composite primary key column could only be assigned unique generators.

Users can now press Enter to finish copying a workspace or a generator preset, instead of having to click Copy.

File connector

• Added support for the Conditional generator for file groups that contain CSV files.

Google BigQuery

• Post-job scripts now support DDL and EXPORT.

Oracle

• Fixed an issue with the permissions check that prevented connecting to Amazon RDS for Oracle databases.

SQL Server

• In Data Definition Language (DDL) that applies to the destination database, Tonic now strips WITH INLINE clauses from definitions of user-defined functions (UDF). Inlining does not require these clauses. WITH_INLINE clauses in UDF definitions that do not meet the requirements for inlining can prevent the UDF from being restored properly in the destination database.
• Fixed an issue where the order of XML columns was changed in the destination database.