Tonic Structural release information

Snowflake - Fixed an issue where temporary files in the cloud storage provider were not deleted when a job was canceled.

SQL Server - Fixed an issue where Structural processed SPROC object re-creation when TONIC_SQL_SERVER_SKIP_CREATE_DB=true.

Bug fixes and other internal updates.

SQL Server - Fixed an issue on older versions of SQL Server where the error UnknownPropertyException (NetName) was thrown.

For upsert, improved the error handling. Instead of failing the upsert job when a single table fails, do a try/catch of issues with warnings.

Bug fixes and other internal updates.

Fixed an issue where the Database View filters initially displayed no content.

Oracle - Fixed an issue where automatic sensitivity scans on workspaces that exceeded the job timeout were reported as failed instead of canceled.

Salesforce - Resolved FIELD_INTEGRITY_EXCEPTIONs on OpportunityLineItem creation.

The generator configuration for the HIPAA Address generator no longer suggests linking address types that cannot be linked. You can no longer link the Address and HIPAA Address generators.

PostgreSQL - Fixed an issue where upsert failed with a bad insert statement.

Bug fixes and other internal updates.

To prevent upsert failures, improved the schema check to include comparisons of primary keys, unique columns, foreign key constraints, and indexed columns in the source and destination databases.

Fixed an issue introduced in v1332 where the dropdown for selecting a user when transferring a workspace or assigning a global permission set would not close upon selection.

Added a new environment setting TONIC_ENABLE_JAVA_SDK_DOWNLOAD that controls whether to allow downloads of the Java SDK. The default value is true.

Fixed an issue where users who had permission to view destination data, but did not have permission to view source data, were able to use composite generators to view unmasked data.

Bug fixes and other internal updates.

Databricks - Fixed an issue where the Struct Mask generator failed on array columns in Databricks 14.3+.

PostgreSQL - Fixed an issue where upsert failed for generated columns.

When you output destination data to Ephemeral or to a container repository, you can now configure the workspace to override the statistics seed.

Bug fixes and other internal updates.

Bug fixes and other internal updates.

From the Java SDK, you can now enable diagnostic logging when you generate statistics as part of processing a DataFrame.

Fixed an issue that caused the preview generated values to display incorrectly in the Database View generator recommendations.

Bug fixes and other internal updates.

Fixed Jobs hang when database connections (source or destination) are not available.

SQL Server - Added support for propagating non-system SQL CLR assemblies from the source to the destination.

Bug fixes and other internal updates.

Fixed an issue that caused the preview generated values to display incorrectly in the Database View generator recommendations.

MongoDB - Fixed an issue where specifying an unscanned collection as a subset target caused data generation to fail.

Salesforce - The Date Truncation generator now works correctly.

PostgreSQL - Fixed an issue where upsert operations failed on tables that contained custom types and unique constraints.

Bug fixes and other internal updates.

Bug fixes and other internal updates.

For Data Pipeline V2, fixed a rare issue where a job was not marked as failed after a fatal error, and the worker would not process other jobs.

Sensitivity confidence information in Privacy Hub - On Privacy Hub, the column boxes in the At-Risk Columns panel now reflect the confidence level in the sensitivity detection for each column. The sensitivity confidence is also displayed for the columns on the Recommended Generators by Sensitivity Type panel.

Bug fixes and other internal updates.

Bug fixes and other internal updates.

Fixed an issue where connection type limit in the Connection Type section of workspace settings was displayed incorrectly.

Bug fixes and other internal updates.

Fixed an issue where Microsoft Entra ID SSO groups were not retrieved if the membership in them was transitive.

Bug fixes and other internal updates.

Fixed an issue where PostgreSQL container artifacts failed to generate.

You can now specify the port that dotnet-monitor listens on within a worker. To set the port, use the environment setting TONIC_WORKER_DOTNET_MONITOR_PORT. Workers that are colocated on the same instance should be assigned different port bindings.

File connector - For custom Amazon S3 URLs, you can now indicate to trust the server certificate. To configure this, use the TONIC_AWS_S3_TRUST_SERVER_CERT environment setting.

MongoDB - Fixed an issue that caused layout problems for Privacy Hub.

Snowflake - Fixed an issue where a worker would sometimes become unresponsive after cancelling a data generation job.

Google BigQuery - Improved progress tracking for tearing down and recreating entities in the destination dataset.

Databricks - For new workspaces, changed the default Databricks runtime to 14.3.x-scala2.12.

When you configure a workspace to write output to a container repository, you can now provide a custom password for the destination database superuser.

File connector - The file upload now restricts accepted files to those that have supported file extensions.

MySQL - Improved resilience to transient issues when applying table constraints.

Fixed an issue where the first recommendation in the Recommended Generators by Sensitivity Type panel was cut off.

Fixed an issue where the generator recommendation preview sometimes failed with the message No Workspaces have been created yet.

Added API endpoints to enable and disable subsetting for a workspace.

Database View redesign and sensitivity confidence - On Database View, the Applied Generator column is now split into two columns. The Status column shows the current sensitivity status (Not Sensitive, At Risk, or Protected). From the Status column, you can display information about the detected sensitivity type and configure whether the column is sensitive. For at risk columns, the Status column also indicates the confidence level in the sensitivity detection. The Applied Generator column indicates the current generator assignment and allows you update the generator configuration.

Improved the detection accuracy for values in defined categories, such as gender.

Added the option to collect performance metrics when starting a data generation job that uses the Data Pipeline V2 processing.

When you write output to a container repository, you can now use assumed roles for the AWS credentials for Amazon ECR.

Snowflake - Improved performance when loading workspaces that connect to large databases.

Bug fixes and other internal updates.

Fixed an issue that caused the job details view to crash for some jobs that had warnings.

Added a new configuration setting TONIC_SENSITIVITY_SCAN_IGNORE_PII_TYPES. Accepts a comma-separated list of sensitivity type identifiers that the sensitivity scan should ignore.

Fixed an issue where missing data could cause the application to crash.

Bug fixes and other internal updates.

On Database View, when you create a sensitivity rule from the bulk edit function, Structural now automatically populates the data type.

Bug fixes and other internal updates.

Fixed an issue where generator application errors appeared to be encrypted.

Bug fixes and other internal updates.

Fixed an issue where generator application errors appeared to be encrypted.

Fixed an issue where generator suggestions caused the sensitivity scan to fail.

Fixed an issue that caused an error to sometimes occur when configuring a workspace to write output to Tonic Ephemeral.

Fixed an issue that caused display issues for Collection View rows that had applied generators.

Fixed an issue that prevented the application from redirecting after the workspace settings were saved.

Docker Compose files that are generated for output to a container repository now no longer contain an obsolete version specification.

Removed data science mode from Structural. The option to enable data science mode no longer appears in workspace settings.

International Address generator - The new International Address Generator can generate street addresses and postal codes for Canada, and postal codes for the United Kingdom.

Snowflake - Improved performance when loading workspaces that connect to large databases.

For new workspaces that support it, Data Pipeline V2 is now the default data generation option.

Structural now provides suggested generators for additional sensitivity types such as marital status.

The Protection Audit Trail now includes sensitivity rule events, including when a sensitivity rule is created, edited, or deleted.

Bug fixes and other internal updates.

Fixed an issue that caused a horizontal scrollbar to appear beneath the Privacy Hub sensitive column cards.

Bug fixes and other internal updates.

Structural no longer requires an external mount to monitor workers.

The new environment setting TONIC_DB_SCHEMA configures the Tonic Structural application database schema. Use the new environment setting TONIC_MIGRATION_ENABLE_LOGGING to enable or disable logging when applying migrations to the Tonic Structural application database. Note that if TONIC_DB_SCHEMA is a non-public value, then you must restart the API container. If there was existing data in the Tonic Structural application database, that data is not migrated. After the API is restarted and the migrations are applied, you can import a backup of the existing data.

When you create a custom sensitivity rule from the Bulk Edit panel of Database View, you can now choose to apply the generator preset to all of the workspace columns that match the new rule. The Save and Apply option replaces the previous Save and Scan option.

Bug fixes and other internal updates.

Oracle - When TONIC_ORACLE_SKIP_CREATE_DB is true, tables in the destination database are preserved. If a table filter is provided in TONIC_TABLE_WHITELIST, then destination tables that are not in the whitelist are preserved. Before this change, those tables were truncated.

Bug fixes and other internal updates.

MongoDB - Added the environment setting TONIC_MONGO_MASK_FIELD_IDS. When set to true, Structural assigns the Mongo ObjectId Key generator to ObjectId field names. Consistency is enabled. The default value is false.

Bug fixes and other internal updates.

Improved error handling for PII detectors.

Google BigQuery - Fixed an issue where some BigQuery workspaces did not release rows from memory in a timely manner.

MySQL - The new environment setting TONIC_MYSQL_USE_NATIVE_DUMP_TOOL, when set true, prefers the mysql native tools over the existing default of mariadb.

MySQL - You can now configure the following environment settings to override the Structural default behavior when a connection opens and a session is established:

• TONIC_MYSQL_NETWORK_READ_TIMEOUT

• TONIC_MYSQL_NETWORK_WRITE_TIMEOUT

• TONIC_MYSQL_WAIT_TIMEOUT

• TONIC_MYSQL_LOCK_WAIT_TIMEOUT

• TONIC_MYSQL_INNODB_LOCK_WAIT_TIMEOUT

Azure SSO - Added support for authenticating application service principals using the EntraID client-credentials flow. Service principals can access the Structural API. For configuration requirements, refer to the Azure/EntraID SSO configuration information in the Structural User Guide.

SQL Server - Fixed the copy of stored procedures that reference full-text indexes.

Structural now supports diagnostic logging for upsert data generation.

Snowflake - Fixed an issue where, when using the connection string and key-pair authentication options, the test connection button was disabled unless a password was entered.

Updated the legal text that is displayed for new account creation on self-hosted Structural instances.

For custom sensitivity rules, column matching rules are now always case insensitive. Previously, the column matching rules were always case sensitive.

SQL Server - Added support for:

• Full-Text Search catalogs and indexes
• Change tracking for database and table scopes

‍

Bug fixes and other internal updates.

Improved the free trial onboarding flow. Added to the available data connectors. Fixed an issue with creating a Google BigQuery workspace. Fixed a duplicate display issue.

File connector - Fixed an issue that a caused authorization failures when using Assume Role to authorize access to Amazon S3 from Structural Cloud.

Fixed an issue where after an import from a JSON file, Subsetting view did not immediately reflect the state of the workspace.

Spark - Removed support for Livy on Hive.

PostgreSQL - Fixed an issue where Structural failed to process tables that contained a generated column that referenced a user-defined type.

Scheduling data generation - From the Jobs view (renamed from Job History) for a workspace, you can now configure the data generation to run automatically on a schedule. The schedule consists of one or more cron expressions, along with the time zone to use for the schedule. The Structural API includes new endpoints to manage the job schedule.

MongoDB - Fixed an issue that caused connection errors for connections to a Mongo Atlas cluster from Tonic Cloud.

Bug fixes and other internal updates.

File connector - You can now specify a separate IAM role for the output location.

Create sensitivity rule from Database View - When you select the Database View bulk edit option for columns that have the same data type, do not have an assigned generator, and do not have a recommended generator, you now have the option to create a custom sensitivity rule. You can then immediately run a new sensitivity scan to catch matching columns.

Fixed an issue with the JSON Mask generator configuration panel where the example data did not update correctly.

Structural now displays a warning when the pre-job checks determine that the source database is on a newer major version than the destination database.

Salesforce - Rewrote the connector algorithm to avoid using sentinels, and to improve subset creation.

Snowflake - Fixed a regression introduced in v1213 that limited table parallelism for all data generations that use the V2 pipeline.

Oracle - Fixed an issue where for subsetting data generation, the Maximum Character Limit was not calculated properly.

Databricks - Structural now supports writing Identity columns to tables.File connector - You can now assign the Timestamp Shift and Date Truncation generators to Parquet date fields.

PostgresSQL - Removed the option to run PostgreSQL jobs using the older flow. All jobs now run with the Data Pipeline v2.

Snowflake

• Fixed an issue where after any row failed to parse, Structural did not process the rest of an unloaded table file.
• For AWS, added retry support to address some transient issues with Amazon S3.

Bug fixes and other internal updates.

The detector for city names now ignores misleading values that are not city names.

Bug fixes and other internal updates.

Bug fixes and other internal updates.

Updated the application to reflect the rename to Tonic Structural. Includes renaming the Tonic Settings view to Structural Settings.

From the Access Management tab of Structural Settings, users with permission to manage Structural access can now restore deleted users.

For a column that is part of a unique compound index, Structural now only suggests generators that can be used for unique columns.

Structural now detects SWIFT codes based on the format of the data in addition to the column name.

Fixed an issue where all subsetting WHERE clauses failed internally.

Databricks

• On self-hosted instances, you can now configure whether Structural creates the destination database schema for Databricks tables. The environment setting TONIC_DATABRICKS_SKIP_CREATE_DB indicates whether to skip the schema creation. The default is false. The environment setting TONIC_DATABRICKS_ENABLE_WORKSPACE_SKIP_CREATE_DB indicates whether to include the option in the workspace configuration, and use TONIC_DATABRICKS_SKIP_CREATE_DB to determine the default. The default is true. You can add these settings to the Environment Settings list on Structural Settings.
• When a cluster is resizing, and the workspace is configured with Use Databricks Job Cluster turned off, Structural data generation now waits for the resizing to complete.

File connector

• Fixed an issue where a copied local files workspace that contained Avro or Parquet files could not be loaded.
• Structural now supports authentication to Amazon S3 buckets using assumed roles for cross-account access.
• Fixed an issue where file connector workspaces did not load when they used an Amazon S3 source with environment-based authentication.

Snowflake

• Fixed a rare subsetting issue where processing a large table could cause the job to hang.
• Fixed an issue where Structural did not retry transient failures when it read unloaded files from a cloud storage provider.

The scheduled sensitivity scans are now daily instead of weekly. By default, the scans run every day at midnight. Structural scans the 10 workspaces that have the most recent activity. Activity is defined as either a user-initiated workspace event that is added to the Protection Audit Trail, or a data generation job.

On the details view for custom sensitivity rules, fixed an issue where the Edit Current Preset button was always disabled.

When a generation to Ephemeral fails, Structural job logs now include the Ephemeral logs and destination database pod logs.

For users who do not have permission to manage sensitivity rules, the Sensitivity Rules option now displays in a disabled state.

When you configure a workspace to write to a self-hosted Ephemeral instance, or to write to Ephemeral Cloud from a self-hosted Structural instance, the workspace configuration now includes an option to test the Ephemeral connection.

Yugabyte data connector - Structural now allows you to connect to databases on Yugabyte version 2024.1 and above. The Yugabyte data connector is available with a Professional or Enterprise license. It only supports Yugabyte SQL (YSQL).

When you configure a custom security rule, you can now create or edit the assigned generator preset. You can also use a workspace to preview the security rule results. The preview displays the matching columns for the selected workspace.

Structural can now detect the following additional sensitivity types:

• US driver’s license number
• Passport number
• Marital status
• GPS coordinates
• Non-birthday dates: admission date, discharge date, date of death
• US license plate

MySQL

• When the SQL mode ALLOW_INVALID_DATES is set, Structural now allows Passthrough for columns that contain invalid dates.

Snowflake

• For the Data Pipeline V2 process, reduced the frequency of polling cloud platforms for new unloaded files.

Fixed an issue on the webhook configuration panel where users could not click Save when the Message Body tab contained large property values.

Fixed an issue that caused the Notifications service to stop processing webhooks.

Improved the detection of name values to identify more specific types of names.

Amazon EMR

• Fixed an issue with previewing data after applying the Timestamp Shift Generator or Date Truncation Generator to timestamp columns with timezones.