Azure SQL customers are now able to import or export data from Azure Data Lake Store (ADLS) directly into Azure SQL Data Warehouse (SQL DW). Microsoft made the announcement on its blog. The company says Imports and exports of data are carried out using External Tables.
Microsoft says that Azure Data Lake Store was purposefully created for massively parallel processing. The no-limits storage solutions has SQL DW PolyBase integrated. ADLS gives customers the ability to load data directly into SQL DW at nearly three TB per hour.
The update improves the capabilities of Azure SQL Data Warehouse. The service already supported direct data transports from Windows Azure Storage Blob. With the new support of ADLS, any type of storage service in Azure can load data to SQL DW. Microsoft says this is important for providing flexibility to developers.
Users can import data from ORC, RC, Parquet, or Delimited Text File formats straight into Data Warehouse.
Importing from Azure Data Lake Store to SQL DW
Microsoft shows one use for ADLS working with SQL DW. ADLS ingests raw data from several different sources, with ADL Analytics cleaning and processing the data into a ready format. Any data can then be imported to Azure SQL DW via PolyBase.
“ADLS has a variety of built-in security features that PolyBase uses to ensure your data remains secure, such as always-on encryption, ACL-based authorization, and Azure Active Directory (AAD) integration,” Microsoft explains in its blog post.
“To load data from ADLS via PolyBase, you need to create an AAD application. Read and write privileges are managed for the AAD application on either a per directory, subdirectory, or file basis. This allows you to provide fine-grained access control of what data can be loaded into SQL DW from ADLS resulting in an easy to manage security model.”